Detecting Multiple Errors in a Single Test with JUnit

Now, it's the turn of another little useful thing. Recalling my test examples, and taking one with more details (i.e. testing some metadata values):

package it.tidalwave.imageio.raf;

import javax.annotation.Nonnull;
import java.util.Collection;
import it.tidalwave.imageio.ExpectedResults;
import it.tidalwave.imageio.NewImageReaderTestSupport;
import org.junit.runners.Parameterized.Parameters;

public class RAFImageReaderImageTest extends NewImageReaderTestSupport
  {
    public RAFImageReaderImageTest (final @Nonnull ExpectedResults expectedResults)
      {
        super(expectedResults);
      }

    @Nonnull
    @Parameters
    public static Collection<Object[]> expectedResults()
      {
        return fixed
          (
            // S9500
            ExpectedResults.create("https://imaging.dev.java.net/nonav/TestSets/peterbecker/Fujifilm/FinePixS9500/RAF/DSCF3756.RAF").
                            image(4292, 4291, 3, 16, "f53e19fbd1f512ddf052e13097735383").
                            thumbnail(160, 120).
                            thumbnail(1600, 1200).
                            issues("JRW-252").
                            metadata("metadata.fujiRawData.header", "FUJIFILMCCD-RAW 0201FF393101FinePix S9500  \u0000\u0000\u0000\u0000\u0000").
                            metadata("metadata.fujiRawData.version", "\u0000\u0000\u0000\u0000").
                            metadata("metadata.fujiRawData.b1", new byte[]{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 48, 50, 54, 57}).
                            metadata("metadata.fujiRawData.b2", new byte[]{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}).
                            metadata("metadata.fujiRawData.JPEGImageOffset", 148).
                            metadata("metadata.fujiRawData.JPEGImageLength", 611759).
                            metadata("metadata.fujiRawData.table1Offset", 611914).
                            metadata("metadata.fujiRawData.table1Length", 3958).
                            metadata("metadata.fujiRawData.CFAOffset", 615872).
                            metadata("metadata.fujiRawData.CFALength", 18528512).
                            metadata("metadata.fujiRawData.unused1", 0).
                            metadata("metadata.fujiRawData.unused2", 18528512).
                            metadata("metadata.fujiRawData.unused3", 0).
                            metadata("metadata.fujiRawData.unused4", 0).
                            metadata("metadata.fujiRawData.unused5", 0).
                            metadata("metadata.fujiRawData.unused6", 0).
                            metadata("metadata.fujiRawData.unused7", 0).
                            metadata("metadata.fujiRawData.unused8", 0).
                            metadata("metadata.fujiRawData.unused9", 0).
                            metadata("metadata.fujiRawData.unused10", 0).
                            metadata("metadata.fujiRawData.fujiTable1.width", 2448).
                            metadata("metadata.fujiRawData.fujiTable1.height", 3688).
                            metadata("metadata.fujiRawData.fujiTable1.rawWidth", 2512).
                            metadata("metadata.fujiRawData.fujiTable1.rawHeight", 3688).
                            metadata("metadata.fujiRawData.fujiTable1.fujiLayout", true).
                            metadata("metadata.fujiRawData.fujiTable1.coefficients", new short[]{320, 502, 320, 422})
          );
      }
  }

,,,you can see that there are a lot of things that can fail: image size, raster, thumbnail size, any metadata item; there can be multiple failures at the same time, but with the usualy style of handling test errors (that is with plain asserting) you can only see the first. You fix it, just to discover another, etc. For instance, if the image size is different than the expected, the whole section of assertions for metadata wont' be executed.

In my experience with jrawio, many errors with a new sample file could be fixed at the same time; wouldn't be much better if you could see *all* the failures immediately?

You can say "thanks!" to Rules. Rules are objects with a specific preset behaviour, annotated with @Rule. For instance, ErrorCollector is a container of multiple errors that can occur in a single test. It must be declared as a test class field as follows:

public class MyTest 
  {
    @Nonnull
    private final ExpectedResults expectedResults;

    // test follows as usual
    @Test
    public void testImage() { ... } 
  }

Now you can accumulate errors with code such as the following:

for (int t = 0; t < thumbnailCount; t++)
  {
    try
      {
        final ExpectedResults.Image expectedThumbnail = expectedResults.getThumbnail(t);
        final Dimension size = expectedThumbnail.getSize();
        assertLoadThumbnail(ir, t, size.width, size.height);
      }
    catch (Throwable e)
      {
        errors.addError(e);
      }
  }

...that is, you try/catch Throwables around assertions and pass them to ErrorCollector.addError(). You could directly add a throwable with a if, but it's much more convenient to stay with the Assert framework, so you enjoy some self-describing error messages.

If ErrorCollector is not empty at the end of the test, JUnit will consider the test as failed. JUnit will report details about *all* the errors that occurred, as you can see in the following excerpt from a Maven Surefire report. Note that you could do a similar thing with standard stuff, accumulating exceptions in a Collection and asserting at the end that is empty - I've tried that - but it would be harder to correctly get all the exception dumps in the report.

-------------------------------------------------------------------------------
Test set: it.tidalwave.imageio.raf.RAFImageReaderImageTest
-------------------------------------------------------------------------------
Tests run: 16, Failures: 15, Errors: 0, Skipped: 0, Time elapsed: 60.184 sec <<< FAILURE!
testImage[[https://imaging.dev.java.net/nonav/TestSets/peterbecker/Fujifilm/FinePixS9500/RAF/DSCF3756.RAF]](it.tidalwave.imageio.raf.RAFImageReaderImageTest)  Time elapsed: 34.917 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[8c256e68fe9897a4fac12a06f1a07fb4]> but was:<[f53e19fbd1f512ddf052e13097735383]>
	at org.junit.Assert.assertEquals(Assert.java:123)
	at org.junit.Assert.assertEquals(Assert.java:145)
	at it.tidalwave.imageio.ImageReaderTestSupport.assertRaster(ImageReaderTestSupport.java:282)
	at it.tidalwave.imageio.NewImageReaderTestSupport.testImage(NewImageReaderTestSupport.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)

testImage[[https://imaging.dev.java.net/nonav/TestSets/peterbecker/Fujifilm/FinePixS9500/RAF/DSCF3756.RAF]](it.tidalwave.imageio.raf.RAFImageReaderImageTest)  Time elapsed: 34.919 sec  <<< FAILURE!
java.lang.AssertionError: metadata.fujiRawData.table1Offset expected:<650246> but was:<611914>
	at org.junit.Assert.fail(Assert.java:91)
	at org.junit.Assert.failNotEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:126)
	at it.tidalwave.imageio.NewImageReaderTestSupport.testImage(NewImageReaderTestSupport.java:231)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)

testImage[[https://imaging.dev.java.net/nonav/TestSets/peterbecker/Fujifilm/FinePixS9500/RAF/DSCF3756.RAF]](it.tidalwave.imageio.raf.RAFImageReaderImageTest)  Time elapsed: 34.919 sec  <<< FAILURE!
java.lang.AssertionError: metadata.fujiRawData.fujiTable1.fujiLayout expected:<false> but was:<true>
	at org.junit.Assert.fail(Assert.java:91)
	at org.junit.Assert.failNotEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:126)
	at it.tidalwave.imageio.NewImageReaderTestSupport.testImage(NewImageReaderTestSupport.java:231)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)

testImage[[https://imaging.dev.java.net/nonav/TestSets/peterbecker/Fujifilm/FinePixS9500/RAF/DSCF3756.RAF]](it.tidalwave.imageio.raf.RAFImageReaderImageTest)  Time elapsed: 34.931 sec  <<< FAILURE!
metadata.fujiRawData.fujiTable1.coefficients: arrays first differed at element [0]; expected:<336> but was:<320>
	at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:54)
	at org.junit.Assert.internalArrayEquals(Assert.java:414)
	at org.junit.Assert.assertArrayEquals(Assert.java:260)
	at it.tidalwave.imageio.NewImageReaderTestSupport.testImage(NewImageReaderTestSupport.java:203)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)