Parameterised Testing with JUnit 4

Anybody who's ever written software more complex than the typical Hello, World! examples knows that software without bugs doesn't exist and that it needs to be tested. Most people who have ever done testing on programs written in the Java language have come across JUnit. Now, JUnit is a very small package. In fact, when you look at the number of classes provided you'd think it doesn't do anything useful. But don't be fooled, in this small amount of code lies extreme power and flexibility. Even more so since JUnit 4 that makes full use of the power of annotations, introduced in Java 5.0.

Typically, when using JUnit 3, you'd create one test case class per class you want to test and one test method per method you want to test. The problem is, as soon as your method takes a parameter, it is likely that you will want to test your method with a variety of values for this parameter: normal and invalid values as well as border conditions. In the old days, you had two ways to do it:

• Test the same method with several different parameters in a single test: it works but all your tests with different values are bundled into a single test when it comes to output. And if you have 100 error conditions and it fails on the second one, you don't know whether any of the remaining 98 work.
• Create one method per parameter variation you want to try: you can then report each test condition individually but it can be difficult to tell when they are related to the same method. And it requires a stupid amount of extra code.

There must be a better way to do this. And JUnit 4 has the answer: parameterised tests. So how does it work? Let's take a real life example. I've been playing with the new script package in Java 6.0 to create a basic Forth language interpreter in Java. So far, I can do the four basic arithmetic operations and print stuff off the stack. That sort of code is the ideal candidate for parameterised testing as I'd want to test my eval method with a number of different scripts without having to write one single method for each test script. So, the JUnit test class I started from looked something like this:

<some more imports go here...>

import static org.junit.Assert.*;
import org.junit.Test;

public class JForthScriptEngineTest {

 @Before
 public void setUp() throws Exception {
  jforthEngineFactory = new JForthScriptEngineFactory();
  jforthEngine = jforthEngineFactory.getScriptEngine();
 }

 @After
 public void tearDown() throws Exception {
  jforthEngine = null;
  jforthEngineFactory = null;
 }

 @Test
 public void testEvalReaderScriptContext() throws ScriptException {
  // context
  String script = "2 3 + .";
  String expectedResult = "5 ";
  ScriptContext context = new SimpleScriptContext();
  StringWriter out = new StringWriter();
  context.setWriter(out);
  // script
  StringReader in = new StringReader(script);
  Object r = jforthEngine.eval(in, context);
  assertEquals("Unexpected script output", expectedResult,
   out.toString());
 }

 @Test
 public void testGetFactory() {
  ScriptEngine engine = new JForthScriptEngine();
  ScriptEngineFactory factory = engine.getFactory();
  assertEquals(JForthScriptEngineFactory.class, factory.getClass());
 }

 private ScriptEngine jforthEngine;
 
 private ScriptEngineFactory jforthEngineFactory;
}

My testEvalReaderScriptContext method was exactly the sort of things that should be parameterised. The solution was to extract that method from this test class and create a new parameterised test class:

<some more imports go here...>

import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class ParameterizedJForthScriptEngineTest {
 @Parameters
 public static Collection<String[]> data() {
  return Arrays.asList(new String[][] {
    { "2 3 + .", "5 " },
    { "2 3 * .", "6 " }
  }
  );
 }
 
 public ParameterizedJForthScriptEngineTest(
   String script, String expectedResult) {
  this.script = script;
  this.expectedResult = expectedResult;
 }

 @Before
 public void setUp() throws Exception {
  jforthEngineFactory = new JForthScriptEngineFactory();
  jforthEngine = jforthEngineFactory.getScriptEngine();
 }

 @After
 public void tearDown() throws Exception {
  jforthEngine = null;
  jforthEngineFactory = null;
 }

 @Test
 public void testEvalReaderScriptContext() throws ScriptException {
  // context
  ScriptContext context = new SimpleScriptContext();
  StringWriter out = new StringWriter();
  context.setWriter(out);
  // script
  StringReader in = new StringReader(script);
  Object r = jforthEngine.eval(in, context);
  assertEquals("Unexpected script output", expectedResult,
   out.toString());
 }

 private String script;
 
 private String expectedResult;

 private ScriptEngine jforthEngine;
 
 private ScriptEngineFactory jforthEngineFactory;
}

The annotation before the class declaration tells it to run with a custom runner, in this case, the Parameterized one. Then the @Parameters annotation tells the runner what method will return a collection of parameters that can then be passed to the constructor in sequence. In my case, each entry in the collection is an array with two String values which is what the constructor takes. If I want to add more test conditions, I can just add more values in that collection. Of course, you could also write the data() method so that it reads from a set of files rather than hard code the test conditions and then you will reach testing nirvana: complete separation between tested code, testing code and test data!

So there you have it again: size doesn't matter, it's what you put in your code that does. JUnit is a very small package that packs enormous power. Use it early, use it often. It even makes testing fun! Did I just say that?

Update

I added the ability to produce compiled scripts in my Forth engine today, in addition to straight evaluation. I added a new test method in ParameterizedJForthScriptEngineTest and all parameterised tests are now run through both methods without having to do anything. What's better, I can immediately see what test data work in one case but not the other.