JUnit: A Little Beyond @Test, @Before, @After

Most people I see using JUnit are limiting themselves to the very basic features, that is, indicating tests with @Test and maybe @Before and @After. Hey, I myself didn't do much more than that up to a couple of weeks ago. This is just a syntactical update to Java 5, but later releases of JUnit can do much more.

Consider my jrawio project: it's a decoder for different image file formats, so the integration tests match the same scheme: get a file, read and extract some information from it, assert expected results (as a marginal note for understanding the following examples, to assert that I'm reading the expected raster, which is made by many millions of pixels, I'm computing its MD5 and checking against it).

This is a sketch from a decoder test as it was at the beginning of this month:

public class NEFImageReaderTest   
  {
    @Test(timeout=60000)
    public void testJRW146()
      throws Exception
      {
        final String path = "https://imaging.dev.java.net/nonav/TestSets/fabriziogiudici/Nikon/D100/NEF/NikonCaptureEditor/ccw90.nef";
        final ImageReader ir = getImageReader(path);
        assertEquals(1, ir.getNumImages(false));
        assertEquals(1, ir.getNumThumbnails(0));
        assertImage(ir, 3034, 2024);
        assertThumbnail(ir, 0, 120, 160);
        final BufferedImage image = assertLoadImage(ir, 3034, 2024, 3, 16);
        assertLoadThumbnail(ir, 0, 120, 160);
        close(ir);

        assertRaster(image, path, "3659664029723dc8ea29b09a923fca7d");
      }

    @Test(timeout=60000)
    public void testJRW148()
      throws Exception
      {
        final String path = "https://imaging.dev.java.net/nonav/TestSets/fabriziogiudici/Nikon/D100/TIFF/TIFF-Large.TIF";
        final ImageReader ir = getImageReader(path);
        assertEquals(1, ir.getNumImages(false));
        assertEquals(1, ir.getNumThumbnails(0));
        assertImage(ir, 3008, 2000);
        assertThumbnail(ir, 0, 160, 120);
        final BufferedImage image = assertLoadImage(ir, 3008, 2000, 3, 16);
        assertLoadThumbnail(ir, 0, 160, 120);
        close(ir);

        assertRaster(image, path, "03383e837402452f7dc553422299f057");
      }
      
    ...
  }

Pretty much copy & paste, isn't it? Figure out that I've got dozens of test files, more will be added, and a lot of metadata items must be tested too. This isn't going to scale a lot.

Now, recent JUnit versions come with a handy way for running the same test multiple times, with variations. What we need is the pair of annotations @RunWith and @Parameters:

import javax.annotation.Nonnull;
import java.util.Collection;
import it.tidalwave.imageio.ExpectedResults;
import it.tidalwave.imageio.NewImageReaderTestSupport;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(value=Parameterized.class)
public class NEFImageReaderImageTest extends ImageReaderTestSupport
  {
    public NEFImageReaderImageTest (final @Nonnull ExpectedResults expectedResults)
      {
        super(expectedResults);
      }

    @Parameters
    public static Collection<Object[]> expectedResults()
      {
        // discuss about this later
      }
  }

@RunWith is a powerful extension point with JUnit, as it lets you change the default test runner. Parameterized is, as the name says, a parameterized test runner, which runs the same test multiple times. It searches for a static method, annotated with @Parameters, that must provide a Collection of Object arrays. Each item in the Collection matches a test run; i.e. if the Collection contains 10 items, the test will be run 10 times. Each element in the Collection is a set of parameters; Parameterized now will search for a constructor of the test class that accepts arguments, and passes to it the objects in the array. Thus, if the arrays in the Collection are made of three items, the test constructor must accept three objects. In my case, things are simpler since I have a single parameter which is an instance of the ExpectedResults class: it holds all the expected results that will be verified.

The actual body of the test is implemented once and for all in the base class ImageReaderTestSupport:

public class ImageReaderTestSupport
  {
    private final ExpectedResults expectedResults;
  	
    public ImageReaderTestSupport (final @Nonnull ExpectedResults expectedResults)
      {
      	this.expectedResults = expectedResults;
      }
      
    @Test
    public void testImage()
      {
        // do al the stuff, assert against attributes of expectedResults
      }
  }
  

This is a sketch of code that creates a collection of ExpectedResults:

@Parameters
public static Collection<Object[]> expectedResults()
  {
    return fixed
      (
        // D1x
        ExpectedResults.create("http://www.rawsamples.ch/raws/nikon/d1x/RAW_NIKON_D1X.NEF").
                        image(4028, 1324, 3, 16, "d3d3b27908bc6f9ed97d1f68c9d7a4af").
                        thumbnail(160, 120),

        // D1
        ExpectedResults.create("http://www.rawsamples.ch/raws/nikon/d1/RAW_NIKON_D1.NEF").
                        image(2012, 1324, 3, 16, "69c3916e9a583f7e48ca3918d31db135").
                        thumbnail(160, 120),
        // D2X v1.0.1
        ExpectedResults.create("http://s179771984.onlinehome.us/RAWpository/images/nikon/D2X/1.01/_DSC0733.NEF").
                        image(4320, 2868, 3, 16, "0cb29a0834bb2293ee4bf0c09b201631").
                        thumbnail(160, 120).
                        thumbnail(4288, 2848),
        // D2Xs v1.0.0
        ExpectedResults.create("http://s179771984.onlinehome.us/RAWpository/images/nikon/D2Xs/1.00/DSC_1234.nef").
                        image(4320, 2868, 3, 16, "37d5aa7aab4e2d4fd667efb674f558ed").
                        thumbnail(160, 120).
                        thumbnail(4288, 2848),
        // D3
        ExpectedResults.create("http://www.rawsamples.ch/raws/nikon/d3/RAW_NIKON_D3.NEF").
                        image(4288, 2844, 3, 16, "fadead8af5aefe88b4ca8730cfb7392c").
                        thumbnail(160, 120).
                        thumbnail(4256, 2832),
        // D3x
        ExpectedResults.create("http://www.rawsamples.ch/raws/nikon/d3x/RAW_NIKON_D3X.NEF").
                        image(6080, 4044, 3, 16, "2a0cfc36cea7c3346b8d39355bf786e6").
                        thumbnail(160, 120).
                        thumbnail(6048, 4032).
                        issues("JRW-221"),
      );
  }
}

As you can see, I'm using the fluent interface pattern to have readable code, and each instance specifies the URL of the test file, the size of the image, the number of colors and bits per sample, the MD5 of the raster, the size of the thumbnail(s), the Jira code of the issues covered by this test, etc. I'm also working on some more methods that, using reflection, are able to inspect metadata and assert single item values.

The static method fixed() is a very simple facility that converts the inlined array of ExpectedResults into the Collection of Object arrays that Parameters wants:

@Nonnull
protected static Collection<Object[]> fixed (final @Nonnull ExpectedResults ... er)
  {
    final List<Object[]> result = new ArrayList<Object[]>();

    for (final ExpectedResults e : er)
      {
        result.add(new Object[]{ e });
      }

    return result;
  }

This approach drastically reduced the number of LOC in my tests and allowed me to add more files and checks. I'm even evaluating to create the ExpectedResults instances with a Groovy script, that would basically make it possible to use a configuration file for declaring test files and expected results.

A minor annoyance is that JUnit doesn't provide a readable output when running parameterized tests. In a report they are displayed as:

testImage[0] PASSED
testImage[1] PASSED
testImage[2] PASSED
testImage[3] PASSED
testImage[4] PASSED
testImage[5] FAILED
testImage[6] PASSED

This happens everywhere, both with JUnit run from a shell and with an IDE such as NetBeans. It isn't helpful to see that testImage[5] failed, as you have to manually look up the source code, or look at the test log files, to find out which test file triggered the failure.

Inspecting JUnit sources, I've found that there are two specific methods in Parameterized that returns the display name of the test. So I copied the source into my own MyParameterized that has been patched as follows:

public class MyParameterized extends Suite
  {
    private class TestClassRunnerForParameters extends BlockJUnit4ClassRunner
      {
      	...
      	
        @Override
        protected String getName()
          {
            return String.format("[%s]", Arrays.toString(fParameterList.get(fParameterSetNumber)));
          }

        @Override
        protected String testName (final FrameworkMethod method)
          {
            return String.format("%s[%s]", method.getName(), Arrays.toString(fParameterList.get(fParameterSetNumber)));
          }

	...
      }
  }
  

In this way ExpectedResults.toString() is used as the test display name and I see speaking reports even in my NetBeans IDE:

testImage["http://www.rawsamples.ch/raws/nikon/d1x/RAW_NIKON_D1X.NEF"] PASSED 
testImage["http://www.rawsamples.ch/raws/nikon/d1/RAW_NIKON_D1.NEF"] PASSED
testImage["http://s179771984.onlinehome.us/RAWpository/images/nikon/D2X/1.01/_DSC0733.NEF"] PASSED
testImage["http://s179771984.onlinehome.us/RAWpository/images/nikon/D2Xs/1.00/DSC_1234.nef"] PASSED
testImage["http://www.rawsamples.ch/raws/nikon/d3/RAW_NIKON_D3.NEF"] PASSED
testImage["http://www.rawsamples.ch/raws/nikon/d3x/RAW_NIKON_D3X.NEF"] PASSED

I've not submitted a patch to JUnit yet since I'm studing this stuff to enhance it even more; for instance, to run tests in parallel. I've just rented a 8-core server for my Continuous Integration and I hope to make my tests run faster at least of a 4x/6x factor. But that's a topic for another article.