Unit test

Content:

  1. Introduction
  2. Example
  3. Unit test
  4. What to test
    1. Code coverage
    2. Test values
    3. Test trivial code or not
    4. Unit test vs integration test

Introduction:

Developers have written small test program since the beginning of programming. But the term unit test implies a formal unit test framework and not just some adhoc test programs.

This more formalized form for unit test have been used for 20 years by now. The first widely used unit test framework JUnit (for Java) date back to the late 1990's.

Today use of unit test frameworks is a generally accepted standard for all professional software development no matter what programming language.

Running unit tests are usually done automatically by the build scripts. The build scipts check out from source control, build the software and run the unit tests.

And if a CI (Continious Integration) tool is used then build and therefor unit tests are run all the time.

The benefits from using unit tests are huge:

Initial development
More bugs are found before developers hand over the code to QA
Maintenance
Besides the same as for initial development then a huge benefit is to quickly run all unit tests after a fix to check if the fix broke something not touched

It is so important for initial development that a development methodology TDD (Test Driven Development) has been created where the unit tests are driving/documenting the actual software design.

A key principle is to write the unit tests before the actual code. That is the best way to ensure that you test what the code should be doing and not test what the code is actual doing.

For bug fixing that means start by creating a unit test that triggers the bug and causes the unit test to fail. And first after that do the fix and verify that th eunit test now succeed.

It is important to keep that unit test around after the fix by adding it to the unit tests run regularly. Because experience show that bugs tend to be occasionally reintroduced - if the bug was easy to make once then it is rather likely that a new implementation will make the same mistake.

I strongly agree with the approach of using unit tests and writing the tests before the code to be tested.

Example:

To demonstrate unit tests I will use a little trivial integer vector class with a simple bug in.

Do not focus on whether the code make sense or not. It is just a demo.

package unittest;

public class IntVector {
    private int[] v;
    public IntVector(int n) {
        v = new int[n];
    }
    public IntVector(int[] v) {
        this.v = v;
    }
    public IntVector add(int k) {
        int[] res = new int[v.length];
        for (int i = 0; i < v.length; i++) res[i] = v[i] + k;
        return new IntVector(res);
    }
    public IntVector sub(int k) {
        int[] res = new int[v.length];
        for (int i = 0; i < v.length; i++) res[i] = v[i] - k;
        return new IntVector(res);
    }
    public IntVector mul(int k) {
        int[] res = new int[v.length];
        for (int i = 0; i < v.length; i++) res[i] = v[i] * k;
        return new IntVector(res);
    }
    public IntVector div(int k) {
        int[] res = new int[v.length];
        for (int i = 0; i < v.length; i++) res[i] = v[i] / k;
        return new IntVector(res);
    }
    public IntVector mod(int k) {
        int[] res = new int[v.length];
        for (int i = 0; i < v.length; i++) res[i] = v[i] - (v[i] / k);
        return new IntVector(res);
    }
    /*
    public MathVector mod(int k) {
        int[] res = new int[v.length];
        for (int i = 0; i < v.length; i++) res[i] = v[i] - (v[i] / k) * k;
        return new IntVector(res);
    }
    */
    public int size() {
        return v.length;
    }
    public int[] getV() {
        return v;
    }
}
using System;

namespace UnitTest
{
    public class IntVector {
        private int[] v;
        public IntVector(int n)
        {
            v = new int[n];
        }
        public IntVector(int[] v)
        {
            this.v = v;
        }
        public IntVector Add(int k)
        {
            int[] res = new int[v.Length];
            for (int i = 0; i < v.Length; i++) res[i] = v[i] + k;
            return new IntVector(res);
        }
        public IntVector Sub(int k)
        {
            int[] res = new int[v.Length];
            for (int i = 0; i < v.Length; i++) res[i] = v[i] - k;
            return new IntVector(res);
        }
        public IntVector Mul(int k)
        {
            int[] res = new int[v.Length];
            for (int i = 0; i < v.Length; i++) res[i] = v[i] * k;
            return new IntVector(res);
        }
        public IntVector Div(int k)
        {
            int[] res = new int[v.Length];
            for (int i = 0; i < v.Length; i++) res[i] = v[i] / k;
            return new IntVector(res);
        }
        public IntVector Mod(int k)
        {
            int[] res = new int[v.Length];
            for (int i = 0; i < v.Length; i++) res[i] = v[i] - (v[i] / k);
            return new IntVector(res);
        }
        /*
        public MathVector Mod(int k)
        {
            int[] res = new int[v.Length];
            for (int i = 0; i < v.Length; i++) res[i] = v[i] - (v[i] / k) * k;
            return new IntVector(res);
        }
        */
        public int Count {
            get { return v.Length; }
        }
        public int[] Raw {
            get { return v; }
        }
    }
}
Imports System

Namespace UnitTest
    Public Class IntVector
        Private v As Integer()
        Public Sub New(n As Integer)
            v = New Integer(n - 1) {}
        End Sub
        Public Sub New(v As Integer())
            Me.v = v
        End Sub
        Public Function Add(k As Integer) As IntVector
            Dim res As Integer() = New Integer(v.Length - 1) {}
            For i As Integer = 0 To v.Length - 1
                res(i) = v(i) + k
            Next
            Return New IntVector(res)
        End Function
        Public Function Subtract(k As Integer) As IntVector
            Dim res As Integer() = New Integer(v.Length - 1) {}
            For i As Integer = 0 To v.Length - 1
                res(i) = v(i) - k
            Next
            Return New IntVector(res)
        End Function
        Public Function Multiply(k As Integer) As IntVector
            Dim res As Integer() = New Integer(v.Length - 1) {}
            For i As Integer = 0 To v.Length - 1
                res(i) = v(i) * k
            Next
            Return New IntVector(res)
        End Function
        Public Function Divide(k As Integer) As IntVector
            Dim res As Integer() = New Integer(v.Length - 1) {}
            For i As Integer = 0 To v.Length - 1
                res(i) = v(i) \ k
            Next
            Return New IntVector(res)
        End Function
        Public Function Modulus(k As Integer) As IntVector
            Dim res As Integer() = New Integer(v.Length - 1) {}
            For i As Integer = 0 To v.Length - 1
                res(i) = v(i) - (v(i) \ k)
            Next
            Return New IntVector(res)
        End Function
        'Public Function Modulus(k As Integer) As IntVector
        '   Dim res As Integer() = New Integer(v.Length - 1) {}
        '   For i As Integer = 0 To v.Length - 1
        '       res(i) = v(i) - (v(i) \ k) * k
        '   Next
        '   Return New IntVector(res)
        'End Function
        Public ReadOnly Property Count() As Integer
            Get
                Return v.Length
            End Get
        End Property
        Public ReadOnly Property Raw() As Integer()
            Get
                Return v
            End Get
        End Property
    End Class
End Namespace
<?php
namespace UnitTest;

class IntVector {
    private $v;
    public function __construct($arg) {
        if(is_int($arg)) {
            $this->v = array_fill(0, $arg, 0);
        } else if(is_array($arg) && is_int($arg[0])) {
            $this->v = $arg;
        } else {
            throw new \Exception('Bad argument to IntVector ctor');
        }
    }
    public function add($k) {
        $res = array();
        for ($i = 0; $i < count($this->v); $i++) $res[$i] = $this->v[$i] + $k;
        return new IntVector($res);
    }
    public function sub($k) {
        $res = array();
        for ($i = 0; $i < count($this->v); $i++) $res[$i] = $this->v[$i] - $k;
        return new IntVector($res);
    }
    public function mul($k) {
        $res = array();
        for ($i = 0; $i < count($this->v); $i++) $res[$i] = $this->v[$i] * $k;
        return new IntVector($res);
    }
    public function div($k) {
        $res = array();
        for ($i = 0; $i < count($this->v); $i++) $res[$i] = (int)($this->v[$i] / $k);
        return new IntVector($res);
    }
    public function mod($k) {
        $res = array();
        for ($i = 0; $i < count($this->v); $i++) $res[$i] = $this->v[$i] - (int)($this->v[$i] / $k);
        return new IntVector($res);
    }
    /*
    public function mod($k) {
        $res = array();
        for ($i = 0; $i < count($this->v); $i++) $res[$i] = $this->v[$i] - (int)($this->v[$i] / $k) * $k;
        return new IntVector($res);
    }
    */
    public function size() {
        return count($this->v);
    }
    public function getV() {
        return $this->v;
    }
}

?>

(the code may look a bit more Java'ish than PHP'ish but ignore that)

class IntVector(object):
    def __init__(self, arg):
        if isinstance(arg,list):
            self.v = arg
        else:
            self.v = [0 for _ in range(arg)]
    def add(self, k):
        return IntVector(list(map(lambda i: i + k, self.v)))
    def sub(self, k):
        return IntVector(list(map(lambda i: i - k, self.v)))
    def mul(self, k):
        return IntVector(list(map(lambda i: i * k, self.v)))
    def div(self, k):
        return IntVector(list(map(lambda i: i / k, self.v)))
    def mod(self, k):
        return IntVector(list(map(lambda i: i - (i / k), self.v)))
#    def mod(self, k):
#        return IntVector(list(map(lambda i: i - (i / k) * k, self.v)))

For Python 3.x change / to //.

unit IntVectorUnit;

interface

uses
  Classes;

type
  ArrayOfInteger = array of integer;
  IntVector = class(TObject)
    constructor Create(n : integer);
    constructor Create(v : ArrayOfInteger);
    function OpAdd(k : integer) : IntVector;
    function OpSub(k : integer) : IntVector;
    function OpMul(k : integer) : IntVector;
    function OpDiv(k : integer) : IntVector;
    function OpMod(k : integer) : IntVector;
    function Size : integer;
    function GetV : ArrayOfInteger;
  private
    _v : array of integer;
  end;

implementation

constructor IntVector.Create(n : integer);

begin
  SetLength(_v, n);
end;

constructor IntVector.Create(v : ArrayOfInteger);

var
  i : integer;

begin
  SetLength(_v, Length(v));
  for i := Low(_v) to High(_v) do begin
    _v[i] := v[i];
  end;
end;

function IntVector.OpAdd(k : integer) : IntVector;

var
  res : ArrayOfInteger;
  i : integer;

begin
  SetLength(res, Length(_v));
  for i := Low(_v) to High(_v) do begin
    res[i] := _v[i] + k;
  end;
  OpAdd := IntVector.Create(res);
end;

function IntVector.OpSub(k : integer) : IntVector;

var
  res : ArrayOfInteger;
  i : integer;

begin
  SetLength(res, Length(_v));
  for i := Low(_v) to High(_v) do begin
    res[i] := _v[i] - k;
  end;
  OpSub := IntVector.Create(res);
end;

function IntVector.OpMul(k : integer) : IntVector;

var
  res : ArrayOfInteger;
  i : integer;

begin
  SetLength(res, Length(_v));
  for i := Low(_v) to High(_v) do begin
    res[i] := _v[i] * k;
  end;
  OpMul := IntVector.Create(res);
end;

function IntVector.OpDiv(k : integer) : IntVector;

var
  res : ArrayOfInteger;
  i : integer;

begin
  SetLength(res, Length(_v));
  for i := Low(_v) to High(_v) do begin
    res[i] := _v[i] div k;
  end;
  OpDiv := IntVector.Create(res);
end;

function IntVector.OpMod(k : integer) : IntVector;

var
  res : ArrayOfInteger;
  i : integer;

begin
  SetLength(res, Length(_v));
  for i := Low(_v) to High(_v) do begin
    res[i] := _v[i] - (_v[i] div k);
  end;
  OpMod := IntVector.Create(res);
end;

(*
function IntVector.OpMod(k : integer) : IntVector;

var
  res : ArrayOfInteger;
  i : integer;

begin
  SetLength(res, Length(_v));
  for i := Low(_v) to High(_v) do begin
    res[i] := _v[i] - (_v[i] div k) * k;
  end;
  OpMod := IntVector.Create(res);
end;
*)

function IntVector.Size : integer;

begin
  Size := Length(_v);
end;

function IntVector.GetV : ArrayOfInteger;

begin
  GetV := _v;
end;

end.

Operator methods are prefixed with "Op" because div and mod are keywords in Pascal.

Unit test:

Now we will look at testing this code using some unit testing frameworks.

These unit testing frameworks are often collectively known as xUnit.

Unit test:

package unittest.junit3;

import junit.framework.TestCase;

import unittest.IntVector;

public class TestIntVector extends TestCase {
    public TestIntVector(String s) {
        super(s);
    }
    protected void setUp() {
    }
    protected void tearDown() {
    }
    public void testAdd() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.add(777);
        assertEquals("add size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("add " + i, iv.getV()[i] + 777, iv2.getV()[i]);
        }
    }
    public void testSub() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.sub(777);
        assertEquals("sub size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("sub " + i, iv.getV()[i] - 777, iv2.getV()[i]);
        }
    }
    public void testMul() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.mul(777);
        assertEquals("mul size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("mul " + i, iv.getV()[i] * 777, iv2.getV()[i]);
        }
    }
    public void testDiv() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.div(777);
        assertEquals("div size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("div " + i, iv.getV()[i] / 777, iv2.getV()[i]);
        }
    }
    public void testMod() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.mod(777);
        assertEquals("mod size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("mod " + i, iv.getV()[i] % 777, iv2.getV()[i]);
        }
    }
}

Suite of unit tests:

package unittest.junit3;

import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;

public class AllTests extends TestCase {
    public AllTests(String name) {
        super(name);
    }
    public static Test suite() {
        TestSuite test = new TestSuite();
        test.addTestSuite(TestIntVector.class);
        /*
        test.addTestSuite(Abc.class);
        test.addTestSuite(Xyz.class);
        */
        return test;
    }
}

The rules are:

The test/test suite can be run directly from the IDE (Eclipse/NetBeans/IntelliJ IDEA).

It can be run from the command line:

java -cp junit.jar;myprojectbindir junit.textui.TestRunner unittest.junit3.AllTests

Output:

.....F
Time: 0.002
There was 1 failure:
1) testMod(unittest.junit3.TestIntVector)junit.framework.AssertionFailedError: mod 0 expected:<-1> but was:<-998713>
        at unittest.junit3.TestIntVector.testMod(TestIntVector.java:57)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

FAILURES!!!
Tests run: 5,  Failures: 1,  Errors: 0

It can be run via ant. build.xml snippet:

<junit fork="yes">
    <formatter type="plain" usefile="false"/>
    <classpath path="junit.jar;myprojectbindir"/>
    <test name="unittest.junit3.AllTests"/>
</junit>

Unit test:

package unittest.junit4;

import static org.junit.Assert.assertEquals;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import unittest.IntVector;

public class TestIntVector {
    @Before
    public void setUp() {
    }
    @After
    public void tearDown() {
    }
    @Test
    public void testAdd() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.add(777);
        assertEquals("add size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("add " + i, iv.getV()[i] + 777, iv2.getV()[i]);
        }
    }
    @Test
    public void testSub() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.sub(777);
        assertEquals("sub size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("sub " + i, iv.getV()[i] - 777, iv2.getV()[i]);
        }
    }
    @Test
    public void testMul() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.mul(777);
        assertEquals("mul size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("mul " + i, iv.getV()[i] * 777, iv2.getV()[i]);
        }
    }
    @Test
    public void testDiv() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.div(777);
        assertEquals("div size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("div " + i, iv.getV()[i] / 777, iv2.getV()[i]);
        }
    }
    @Test
    public void testMod() {
        int[] v = { -1000000, -1, 0, 1, 1000000 };
        IntVector iv = new IntVector(v);
        IntVector iv2 = iv.mod(777);
        assertEquals("mod size", iv.size(), iv2.size());
        for(int i = 0; i < iv2.size(); i++) {
            assertEquals("mod " + i, iv.getV()[i] % 777, iv2.getV()[i]);
        }
    }
}

Suite of unit tests:

package unittest.junit4;

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;

@RunWith(Suite.class)
@SuiteClasses({TestIntVector.class/*,Abc.class,Xyz.class*/})
public class AllTests {
}

The rules are:

The test/test suite can be run directly from the IDE (Eclipse/NetBeans/IntelliJ IDEA).

It can be run from the command line:

java -cp junit.jar;myprojectbindir org.junit.runner.JUnitCore unittest.junit4.AllTests

Output:

.....E
Time: 0.008
There was 1 failure:
1) testMod(unittest.junit4.TestIntVector)
java.lang.AssertionError: mod 0 expected:<-1> but was:<-998713>
        at org.junit.Assert.fail(Assert.java:91)
        at org.junit.Assert.failNotEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:126)
        at org.junit.Assert.assertEquals(Assert.java:470)
        at unittest.junit4.TestIntVector.testMod(TestIntVector.java:65)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        ...

It can be run via ant. build.xml snippet:

<junit fork="yes">
    <formatter type="plain" usefile="false"/>
    <classpath path="junit.jar;myprojectbindir"/>
    <test name="unittest.junit4.AllTests"/>
</junit>
using System;

using Microsoft.VisualStudio.TestTools.UnitTesting;

using UnitTest;

namespace UnitTest.MS
{
    [TestClass]
    public class TestIntVector
    {
        [ClassInitialize]
        public static void SetUp(TestContext context)
        {
        }
        [ClassCleanup]
        public static void TearDown()
        {
        }
        [TestMethod]
        public void TestAdd()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Add(777);
            Assert.AreEqual(iv.Count, iv2.Count, "add size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] + 777, iv2.Raw[i], "add " + i);
            }
        }
        [TestMethod]
        public void TestSub()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Sub(777);
            Assert.AreEqual(iv.Count, iv2.Count, "sub size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] - 777, iv2.Raw[i], "sub " + i);
            }
        }
        [TestMethod]
        public void TestMul()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Mul(777);
            Assert.AreEqual(iv.Count, iv2.Count, "mul size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] * 777, iv2.Raw[i], "mul " + i);
            }
        }
        [TestMethod]
        public void TestDiv()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Div(777);
            Assert.AreEqual(iv.Count, iv2.Count, "div size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] / 777, iv2.Raw[i], "div " + i);
            }
        }
        [TestMethod]
        public void TestMod()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Mod(777);
            Assert.AreEqual(iv.Count, iv2.Count, "mod size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] % 777, iv2.Raw[i], "mod " + i);
            }
        }
    }
}

The rules are:

The tests can be run directly from the IDE (Visual Studio).

It can be run from the command line:

mstest /testcontainer:UnitTest.MS.dll /resultsfile:z.z

Output:

Loading UnitTest.MS.dll
...
Starting execution...

Results               Top Level Tests
-------               ---------------
Passed                UnitTest.MS.TestIntVector.TestAdd
Passed                UnitTest.MS.TestIntVector.TestDiv
Failed                UnitTest.MS.TestIntVector.TestMod
Passed                UnitTest.MS.TestIntVector.TestMul
Passed                UnitTest.MS.TestIntVector.TestSub
4/5 test(s) Passed, 1 Failed

Summary
-------
Test Run Failed.
  Passed  4
  Failed  1
  ---------
  Total   5
Results file:  z.z
Test Settings: Default Test Settings
using System;

using NUnit.Framework;

using UnitTest;

namespace UnitTest.NUnit
{
    [TestFixture]
    public class TestIntVector
    {
        [SetUp]
        public static void SetUp()
        {
        }
        [TearDown]
        public static void TearDown()
        {
        }
        [Test]
        public void TestAdd()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Add(777);
            Assert.AreEqual(iv.Count, iv2.Count, "add size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] + 777, iv2.Raw[i], "add " + i);
            }
        }
        [Test]
        public void TestSub()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Sub(777);
            Assert.AreEqual(iv.Count, iv2.Count, "sub size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] - 777, iv2.Raw[i], "sub " + i);
            }
        }
        [Test]
        public void TestMul()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Mul(777);
            Assert.AreEqual(iv.Count, iv2.Count, "mul size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] * 777, iv2.Raw[i], "mul " + i);
            }
        }
        [Test]
        public void TestDiv()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Div(777);
            Assert.AreEqual(iv.Count, iv2.Count, "div size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] / 777, iv2.Raw[i], "div " + i);
            }
        }
        [Test]
        public void TestMod()
        {
            int[] v = { -1000000, -1, 0, 1, 1000000 };
            IntVector iv = new IntVector(v);
            IntVector iv2 = iv.Mod(777);
            Assert.AreEqual(iv.Count, iv2.Count, "mod size");
            for(int i = 0; i < iv2.Count; i++)
            {
                Assert.AreEqual(iv.Raw[i] % 777, iv2.Raw[i], "mod " + i);
            }
        }
    }
}

The rules are:

It can be run from the command line:

nunit-console-x86 UnitTest.NUnit.dll

Output:

ProcessModel: Default    DomainUsage: Single
Execution Runtime: Default
...F..
Tests run: 5, Errors: 0, Failures: 1, Inconclusive: 0, Time: 0.0690039 seconds
  Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

Errors and Failures:
1) Test Failure : UnitTest.NUnit.TestIntVector.TestMod
     mod 0
  Expected: -1
  But was:  -998713

at UnitTest.NUnit.TestIntVector.TestMod() in \UnitTest\UnitTest.NUnit\UnitTestNUnit.cs:line 77

It can be run via NAnt. Build file snippet:

<nunit2>
    <formatter type="Plain"/>
    <test>
        <assemblies>
            <include name="UnitTest.NUnit.dll"/>
        </assemblies>
    </test>
</nunit2>
Imports System

Imports Microsoft.VisualStudio.TestTools.UnitTesting

Imports UnitTest

Namespace UnitTest.MS
    <TestClass> _
    Public Class TestIntVector
        <ClassInitialize> _
        Public Shared Sub SetUp(context As TestContext)
        End Sub
        <ClassCleanup> _
        Public Shared Sub TearDown()
        End Sub
        <TestMethod> _
        Public Sub TestAdd()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Add(777)
            Assert.AreEqual(iv.Count, iv2.Count, "add size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) + 777, iv2.Raw(i), "add " & i)
            Next
        End Sub
        <TestMethod> _
        Public Sub TestSub()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Subtract(777)
            Assert.AreEqual(iv.Count, iv2.Count, "sub size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) - 777, iv2.Raw(i), "sub " & i)
            Next
        End Sub
        <TestMethod> _
        Public Sub TestMul()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Multiply(777)
            Assert.AreEqual(iv.Count, iv2.Count, "mul size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) * 777, iv2.Raw(i), "mul " & i)
            Next
        End Sub
        <TestMethod> _
        Public Sub TestDiv()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Divide(777)
            Assert.AreEqual(iv.Count, iv2.Count, "div size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) \ 777, iv2.Raw(i), "div " & i)
            Next
        End Sub
        <TestMethod> _
        Public Sub TestModulus()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Modulus(777)
            Assert.AreEqual(iv.Count, iv2.Count, "mod size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) Mod 777, iv2.Raw(i), "mod " & i)
            Next
        End Sub
    End Class
End Namespace

The rules are:

The tests can be run directly from the IDE (Visual Studio).

It can be run from the command line:

mstest /testcontainer:UnitTest.MS.dll /resultsfile:z.z

Output:

Loading UnitTest.MS.dll
...
Starting execution...

Results               Top Level Tests
-------               ---------------
Passed                UnitTest.MS.TestIntVector.TestAdd
Passed                UnitTest.MS.TestIntVector.TestDivide
Failed                UnitTest.MS.TestIntVector.TestModulus
Passed                UnitTest.MS.TestIntVector.TestMultiply
Passed                UnitTest.MS.TestIntVector.TestSubtract
4/5 test(s) Passed, 1 Failed

Summary
-------
Test Run Failed.
  Passed  4
  Failed  1
  ---------
  Total   5
Results file:  z.z
Test Settings: Default Test Settings
Imports System

Imports NUnit.Framework

Imports UnitTest

Namespace UnitTest.NUnit
    <TestFixture> _
    Public Class TestIntVector
        <SetUp> _
        Public Shared Sub SetUp()
        End Sub
        <TearDown> _
        Public Shared Sub TearDown()
        End Sub
        <Test> _
        Public Sub TestAdd()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Add(777)
            Assert.AreEqual(iv.Count, iv2.Count, "add size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) + 777, iv2.Raw(i), "add " & i)
            Next
        End Sub
        <Test> _
        Public Sub TestSub()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Subtract(777)
            Assert.AreEqual(iv.Count, iv2.Count, "sub size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) - 777, iv2.Raw(i), "sub " & i)
            Next
        End Sub
        <Test> _
        Public Sub TestMul()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Multiply(777)
            Assert.AreEqual(iv.Count, iv2.Count, "mul size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) * 777, iv2.Raw(i), "mul " & i)
            Next
        End Sub
        <Test> _
        Public Sub TestDiv()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Divide(777)
            Assert.AreEqual(iv.Count, iv2.Count, "div size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) \ 777, iv2.Raw(i), "div " & i)
            Next
        End Sub
        <Test> _
        Public Sub TestModulus()
            Dim v As Integer() = {-1000000, -1, 0, 1, 1000000}
            Dim iv As New IntVector(v)
            Dim iv2 As IntVector = iv.Modulus(777)
            Assert.AreEqual(iv.Count, iv2.Count, "mod size")
            For i As Integer = 0 To iv2.Count - 1
                Assert.AreEqual(iv.Raw(i) Mod 777, iv2.Raw(i), "mod " & i)
            Next
        End Sub
    End Class
End Namespace

The rules are:

It can be run from the command line:

nunit-console-x86 UnitTest.NUnit.dll

Output:

ProcessModel: Default    DomainUsage: Single
Execution Runtime: Default
...F..
Tests run: 5, Errors: 0, Failures: 1, Inconclusive: 0, Time: 0.0690039 seconds
  Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

Errors and Failures:
1) Test Failure : UnitTest.NUnit.TestIntVector.TestModulus
     mod 0
  Expected: -1
  But was:  -998713

at UnitTest.NUnit.TestIntVector.TestModulus() in \UnitTest\UnitTest.NUnit\UnitTestNUnit.vb:line 63

It can be run via NAnt. Build file snippet:

<nunit2>
    <formatter type="Plain"/>
    <test>
        <assemblies>
            <include name="UnitTest.NUnit.dll"/>
        </assemblies>
    </test>
</nunit2>
<?php
use PHPUnit\Framework\TestCase;

require 'IntVector.php';

use UnitTest\IntVector;

class IntVectorTest extends TestCase {
    protected function setUp() {
    }
    protected function tearDown() {
    }
    public function testAdd() {
        $v = array( -1000000, -1, 0, 1, 1000000 );
        $iv = new IntVector($v);
        $iv2 = $iv->add(777);
        $this->assertEquals($iv->size(), $iv2->size(), "add size");
        for($i = 0; $i < $iv2->size(); $i++) {
            $this->assertEquals($iv->getV()[$i] + 777, $iv2->getV()[$i], "add " + $i);
        }
    }
    public function testSub() {
        $v = array( -1000000, -1, 0, 1, 1000000 );
        $iv = new IntVector($v);
        $iv2 = $iv->sub(777);
        $this->assertEquals($iv->size(), $iv2->size(), "sub size");
        for($i = 0; $i < $iv2->size(); $i++) {
            $this->assertEquals($iv->getV()[$i] - 777, $iv2->getV()[$i], "sub " + $i);
        }
    }
    public function testMul() {
        $v = array( -1000000, -1, 0, 1, 1000000 );
        $iv = new IntVector($v);
        $iv2 = $iv->mul(777);
        $this->assertEquals($iv->size(), $iv2->size(), "mul size");
        for($i = 0; $i < $iv2->size(); $i++) {
            $this->assertEquals($iv->getV()[$i] * 777, $iv2->getV()[$i], "mul " + $i);
        }
    }
    public function testDiv() {
        $v = array( -1000000, -1, 0, 1, 1000000 );
        $iv = new IntVector($v);
        $iv2 = $iv->div(777);
        $this->assertEquals($iv->size(), $iv2->size(), "div size");
        for($i = 0; $i < $iv2->size(); $i++) {
            $this->assertEquals((int)($iv->getV()[$i] / 777), $iv2->getV()[$i], "div " + $i);
        }
    }
    public function testMod() {
        $v = array( -1000000, -1, 0, 1, 1000000 );
        $iv = new IntVector($v);
        $iv2 = $iv->mod(777);
        $this->assertEquals($iv->size(), $iv2->size(), "mod size");
        for($i = 0; $i < $iv2->size(); $i++) {
            $this->assertEquals($iv->getV()[$i] % 777, $iv2->getV()[$i], "mod " + $i);
        }
    }
}

?>

The rules are:

It can be run from the command line:

phpunit .

Output:

....F                                                              5 / 5 (100%)

Time: 324 ms, Memory: 8.50MB

There was 1 failure:

1) IntVectorTest::testMod
0
Failed asserting that -998713 matches expected -1.

IntVectorTest.php:55

FAILURES!
Tests: 5, Assertions: 26, Failures: 1.

Unit test:

import unittest

from IntVector import IntVector

class TestIntVector(unittest.TestCase):
    def setUp(self):
        pass
    def tearDown(self):
        pass
    def test_add(self):
        v = [ -1000000, -1, 0, 1, 1000000 ]
        iv = IntVector(v)
        iv2 = iv.add(777)
        self.assertEqual(len(iv.v), len(iv2.v), 'add size')
        for i in range(len(iv.v)):
            self.assertEqual(iv.v[i] + 777, iv2.v[i], 'add ' + str(i))
    def test_sub(self):
        v = [ -1000000, -1, 0, 1, 1000000 ]
        iv = IntVector(v)
        iv2 = iv.sub(777)
        self.assertEqual(len(iv.v), len(iv2.v), 'add size')
        for i in range(len(iv.v)):
            self.assertEqual(iv.v[i] - 777, iv2.v[i], 'sub ' + str(i))
    def test_mul(self):
        v = [ -1000000, -1, 0, 1, 1000000 ]
        iv = IntVector(v)
        iv2 = iv.mul(777)
        self.assertEqual(len(iv.v), len(iv2.v), 'add size')
        for i in range(len(iv.v)):
            self.assertEqual(iv.v[i] * 777, iv2.v[i], 'mul ' + str(i))
    def test_div(self):
        v = [ -1000000, -1, 0, 1, 1000000 ]
        iv = IntVector(v)
        iv2 = iv.div(777)
        self.assertEqual(len(iv.v), len(iv2.v), 'add size')
        for i in range(len(iv.v)):
            self.assertEqual(iv.v[i] / 777, iv2.v[i], 'div ' + str(i))
    def test_mod(self):
        v = [ -1000000, -1, 0, 1, 1000000 ]
        iv = IntVector(v)
        iv2 = iv.mod(777)
        self.assertEqual(len(iv.v), len(iv2.v), 'add size')
        for i in range(len(iv.v)):
            self.assertEqual(iv.v[i] % 777, iv2.v[i], 'mod ' + str(i))

unittest.main(verbosity=2)

For Python 3.x change / to //.

The rules are:

The test is a standalone program that can be run as any other Python program.

Output:

test_add (__main__.TestIntVector) ... ok
test_div (__main__.TestIntVector) ... ok
test_mod (__main__.TestIntVector) ... FAIL
test_mul (__main__.TestIntVector) ... ok
test_sub (__main__.TestIntVector) ... ok

======================================================================
FAIL: test_mod (__main__.TestIntVector)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "TestIntVector.py", line 44, in test_mod
    self.assertEqual(iv.v[i] % 777, iv2.v[i], 'mod ' + str(i))
AssertionError: mod 0

----------------------------------------------------------------------
Ran 5 tests in 0.000s

FAILED (failures=1)

Delphi DUnit2 is ported to Lazarus as FPTest. I have only tested with Lazarus and FPTest.

Unit test:

program TestDemo;

uses
  Classes,
  SysUtils,
  IntVectorUnit,
  TestFramework,
  TextTestRunner;

type
    TestIntVector = class(TTestCase)
      procedure TestAdd;
      procedure TestSub;
      procedure TestMul;
      procedure TestDiv;
      procedure TestMod;
    protected
      procedure SetUp; override;
      procedure Teardown; override;
    end;

procedure TestIntVector.SetUp;

begin
end;

procedure TestIntVector.Teardown;

begin
end;

procedure TestIntVector.TestAdd;

var
  iv, iv2 : IntVector;
  v, v2 : ArrayOfInteger;
  i : integer;

begin
  SetLength(v, 5);
  v[0] := -1000000;
  v[1] := -1;
  v[2] := 0;
  v[3] := 1;
  v[4] := 1000000;
  iv := IntVector.Create(v);
  iv2 := iv.OpAdd(777);
  CheckEquals(iv.Size, iv2.Size, 'add size');
  v2 := iv2.GetV;
  for i := 0 to 4 do begin
    CheckEquals(v[i] + 777, v2[i], 'add ' + IntToStr(v[i]));
  end;
end;

procedure TestIntVector.TestSub;

var
  iv, iv2 : IntVector;
  v, v2 : ArrayOfInteger;
  i : integer;

begin
  SetLength(v, 5);
  v[0] := -1000000;
  v[1] := -1;
  v[2] := 0;
  v[3] := 1;
  v[4] := 1000000;
  iv := IntVector.Create(v);
  iv2 := iv.OpSub(777);
  CheckEquals(iv.Size, iv2.Size, 'sub size');
  v2 := iv2.GetV;
  for i := 0 to 4 do begin
    CheckEquals(v[i] - 777, v2[i], 'sub ' + IntToStr(v[i]));
  end;
end;

procedure TestIntVector.TestMul;

var
  iv, iv2 : IntVector;
  v, v2 : ArrayOfInteger;
  i : integer;

begin
  SetLength(v, 5);
  v[0] := -1000000;
  v[1] := -1;
  v[2] := 0;
  v[3] := 1;
  v[4] := 1000000;
  iv := IntVector.Create(v);
  iv2 := iv.OpMul(777);
  CheckEquals(iv.Size, iv2.Size, 'mul size');
  v2 := iv2.GetV;
  for i := 0 to 4 do begin
    CheckEquals(v[i] * 777, v2[i], 'mul ' + IntToStr(v[i]));
  end;
end;

procedure TestIntVector.TestDiv;

var
  iv, iv2 : IntVector;
  v, v2 : ArrayOfInteger;
  i : integer;

begin
  SetLength(v, 5);
  v[0] := -1000000;
  v[1] := -1;
  v[2] := 0;
  v[3] := 1;
  v[4] := 1000000;
  iv := IntVector.Create(v);
  iv2 := iv.OpDiv(777);
  CheckEquals(iv.Size, iv2.Size, 'div size');
  v2 := iv2.GetV;
  for i := 0 to 4 do begin
    CheckEquals(v[i] div 777, v2[i], 'div ' + IntToStr(v[i]));
  end;
end;

procedure TestIntVector.TestMod;

var
  iv, iv2 : IntVector;
  v, v2 : ArrayOfInteger;
  i : integer;

begin
  SetLength(v, 5);
  v[0] := -1000000;
  v[1] := -1;
  v[2] := 0;
  v[3] := 1;
  v[4] := 1000000;
  iv := IntVector.Create(v);
  iv2 := iv.OpMod(777);
  CheckEquals(iv.Size, iv2.Size, 'mod size');
  v2 := iv2.GetV;
  for i := 0 to 4 do begin
    CheckEquals(v[i] mod 777, v2[i], 'mod ' + IntToStr(v[i]));
  end;
end;

begin
  TestFramework.RegisterTest(TestIntVector.Suite);
  RunRegisteredTests;
end.

Output:

FPTest / Testing
.....F
Time: 00:00:00.000


Test Results:
Run:             5
Errors:          0
Failures:        1
Warnings:        0

There was 1 failure:

  1) TestDemo.exe.TestIntVector.TestMod: ETestFailure
     at   $0000000100002017 line 143 of TestDemo.pas
      "mod -1000000"
Expected:
  "-1"
But was:
  "-998713"

What to test:

Code coverage

The code coverage percentage is the percentage of code being executed when all unit tests are run.

It is usually recommended to achieve 100% code coverage.

I fully agree with that recommendation.

Note that 100% code coverage does not guarantee that "everything" is tested. A method with 2 if statemments has 4 different flows through it, so to just test all possible flows then 4 unit tests would be required. Also some bugs will only be triggered by some test values that may not have been tests by unit tests.

Test values

It is beneficial to test methods with multiple values. Preferably including potential critical values. Critical values can be zero, very large value, very small value, very long string, empty string, null (if allowed).

Test trivial code or not

There are two schools of thougth:

I strongly believe in the second approach. Because:

  1. Even trivial code can have bugs (maybe from copy paste).
  2. Sometime a method with trivial code get enhanced with non-trivial code later and then when the build test run the old unit tests then it is still not tested
  3. When writing the unit tests before the actuall code (which is recommended) then sometimes one does not even know if the implementation will be trivial or non-trivial.

Unit test vs integration test

There are also two schools of thougth for unit testing multi layered applications.

The "purists" want to unit test only one layer at a time and do that by mocking lower layers. They consider a unit test that hits more than one layer to be an automated integration test and not a unit test.

Visual:

purists view

The "pragmatists" are OK with hitting more than one layer in a unit test as long as the lower levels has already passed their own unit tests.

Visual:

pragmatists view

I am a pragmatist. In theory I agree with the purist view, but in practice I think the risk of the mockup having different behavior than what it is mocking is way more important.

Article history:

Version Date Description
1.0 October 9th 2016 Initial version based on multiple old articles on Eksperten.dk
1.1 November 4th 2018 Add Python builtin
1.2 April 21st 2019 Add Delphi/Lazarus DUnit2/FPTest

Other articles:

See list of all articles here

Comments:

Please send comments to Arne Vajhøj