For years JUNit was the dominant player in the unit testing world, it could be described as the father of Unit testing frameworks as other frameworks like NUnit and PHPUnit were based on it. Most of us cut our testing teeth on it but is there a better alternative available today? We believe there is.

TestNG (the NG stands for Next Generation) offers the same features as JUnit but also offers some significant enhancements that are lacking in JUnit. A detailed comparison of the two frameworks can be found in this excellent blog post. In our experience the most usable features of TestNG are test groups, parametrized testing and thread support.

Test Groups

Take this scenario … we have a traditional multi tiered web application with anaemic domain classes and stateless services. Having changed a service we want to run our unit tests but we don’t necessarily want to run all our domain tests because the domain classes should not be affected by a change in the service tier. We could create two test suites, one for the service tests and one for the domain tests but TestNG offers us test groups, basically tags that we can assign to individual tests (not just test classes).

@Test(groups = "service")
public void testMyCode() { ... }

We can then tell maven to execute tests with a particular tag …

 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-surefire-plugin</artifactId>
 <configuration>
 <groups>${groups}</groups>
 </configuration>
 </plugin>

$mvn -Dgroups=”service”

The TestNG ecliupse plugin also offers a simple way of executing tests with a particular tag:

Paramaterized testing

Writing good unit tests is only half the battle, maintaining them is equally challenging. One way to reduce the amount of boilerplate code is to write paramaterized tests. JUnit offers paramaterized support but it’s not clean. We need to create a static method which returns a collection of paramaters, create instance variables to hold the current paramaters, then implement a constructor which ties the static method and the instance variables together:

@RunWith(value = Parameterized.class)
public class JunitTest6 {

 private int number;

 public JunitTest6(int number) {
 this.number = number;
 }

 @Parameters
 public static Collection<Object[]> data() {
 Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
 return Arrays.asList(data);
 }

 @Test
 public void pushTest() {
 System.out.println("Parameterized Number is : " + number);
 }
}

In contrast TestNG’s approach is much cleaner:

public class TestNGTest {

 @DataProvider("numbers")
 public Object[][] getNumbers() {
 Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
 return data;
 }

 @Test(dataProvider="numbers")
 public void pushTest(int number) {
 System.out.println("Parameterized Number is : " + number);
 }
}

The distinction between JUnit and TestNG becomes even more apparent when we need to support multiple groups of paramaters:

public class TestNGTest {

 @DataProvider("numbers")
 public Object[][] getNumbers() {
 Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
 return data;
 }

 @DataProvider("names")
 public Object[][] getNames(){
 Object[][] data = new Object[][] { { "fred" }, { "harry" }, { "bob" }, { "alice" } };
 return data;
 }

 @Test(dataProvider="numbers")
 public void pushNumers(int number) {
 System.out.println("Parameterized Number is : " + number);
 }

 @Test(dataProvider="names")
 public void pushStrings(String name) {
 System.out.println("Parameterized name is : " + name);
 }

}

JUnit’s big limitation is that it only support a single dataprovider method which must set up all parameters for all tests.

XML Driven paramaters

TestNG also allows us to specify our parameters in an XML file which is often more readable:

<!DOCTYPE suite SYSTEM "<a href="http://beust.com/testng/testng-1.0.dtd" target="_blank">http://beust.com/testng/testng-1.0.dtd</a>" >
 <suite name="a">
 <test name="testing">

 <parameter name="numbers" value="1"/>
 <parameter name="numbers" value="2"/>
 <parameter name="numbers" value="3"/>

 <classes>
 <class name="com.fsecure.demo.testng.TestNGTest6_0" />
 </classes>

 </test>
 </suite>

 @Test(dataProvider="numbers")
 public void pushTest(int number) {
 System.out.println("Parameterized Number is : " + number);
 }

Testing threads

Anyone who has written a high performance application will have dealt with threads and thread pools. Some libraries (e.g. Spring Security) use threads behind the scenes. Our code may well run under a single thread of execution but what happens when it’s being hit by 10 concurrent threads?

Let’s take some naive code:

package com.test;

public class NaiveClass {

	private int value = 0;

	public void increment(int amount) {
		value += amount;
	}

	public int getValue() {
		return value;
	}

	public void setValue(int value) {
		this.value = value;
	}

}

And our JUnit test:

package com.test;

import static org.junit.Assert.*;
import java.util.Random;

import org.junit.Before;
import org.junit.Test;

public class TestNaiveClass {

	private NaiveClass classUnderTest;
	private Random generator;

	@Before
	public void setup() {
		classUnderTest = new NaiveClass();
		generator = new Random();
	}

	@Test
	public void testNaiveClass() {
		int startValue = generator.nextInt(100);
		int incrementValue = generator.nextInt(100);
		classUnderTest.setValue(startValue);
		classUnderTest.increment(incrementValue);
		int expected = startValue + incrementValue;
		assertEquals(expected, classUnderTest.getValue());
	}
}

and the result …

We know our code is flaky but no matter how many times we run the test it will always pass … not good!

Of course our test runs in a single thread so we can’t test thread safety. We need to fire off multiple threads:

package com.test;

import static org.junit.Assert.assertEquals;

import java.util.Random;

import org.junit.Test;

public class TestNaiveClassThreaded {

	private final NaiveClass classUnderTest = new NaiveClass();
	private final Random generator = new Random();

	@Test
	public void testNaiveClass() {
		for (int i=0; i<100; i++) {
			ThreadedTest threadTest = new ThreadedTest();
			Thread t = new Thread(threadTest);
			t.start();
		}
	}

	class ThreadedTest implements Runnable {

		@Override
		public void run() {
			try {
				int startValue = generator.nextInt(100);
				int incrementValue = generator.nextInt(100);
				classUnderTest.setValue(startValue);
				Thread.sleep(100);
				classUnderTest.increment(incrementValue);
				int expected = startValue + incrementValue;
				assertEquals(expected, classUnderTest.getValue());
			} catch (InterruptedException e) {
				throw new RuntimeException(e);
			}
		}

	}
}

This should fail …

Our test still passes! Something strange is happening here … the threads should sleep for 10 seconds but our test passes immediately. The problem lies with JUnit’s TestRunner which doesn’t support threads … it fires off threads then returns immediately. We can easily prove this:

package com.test;

import static org.junit.Assert.fail;

import org.junit.Test;

public class TestFail {

	@Test
	public void testNaiveClass() {
		for (int i=0; i<100; i++) {
			ThreadedTest threadTest = new ThreadedTest();
			Thread t = new Thread(threadTest);
			t.start();
		}
	}

	class ThreadedTest implements Runnable {

		@Override
		public void run() {
			fail();
		}

	}
}

This test always passes!

TestNG supports threads

In contrast TestNG offers explicit support for threads. We can easily verify that our NaiveClass isn’t thread safe by writing a simple test:

package com.test;

import static org.testng.Assert.assertEquals;

import java.util.Random;

import org.testng.annotations.Test;

public class TestNaiveClassTestNG {

	private final NaiveClass classUnderTest = new NaiveClass();
	private final Random generator = new Random();

	@Test(threadPoolSize=10, invocationCount=100)
	public void testNaiveClass() throws Exception {
		int startValue = generator.nextInt(100);
		int incrementValue = generator.nextInt(100);
		classUnderTest.setValue(startValue);
		Thread.sleep(10);
		classUnderTest.increment(incrementValue);
		int expected = startValue + incrementValue;
		assertEquals(classUnderTest.getValue(), expected);
	}
}

TestNG will create 10 threads and execute the test 100 times. As we can see the test fails because multiple threads are modifying the instance variable …

Conclusion

We believe TestNG offers some significant enhancements over JUnit and from what we have seen there are no drawbacks … it really is the Next Generation testing framework!