Unfortunately handling exceptions thrown by methods under test has always been cumbersome. There has been some relief in the last few months, as both JUnit and TestNG made an effort to improve the handling of error conditions:

• JUnit 4.7 added rules with ExpectedException as one implementation

• TestNG 5.10 added the attribute expectedExceptionsMessageRegExp to the @Test annotation

However despite these efforts, the overhead for handling exceptions cases is still quite high.

The visual approach introduced by MagicTest on the other hand makes handling exceptions a breeze.

This article will present the features implemented by JUnit and TestNG and compare them to the new visual approach featured by MagicTest.

Example

To compare the different approaches, we will look at an example method which returns a configuration value. The method under test must implement the following requirements:

• If namespace and key are valid, the configuration value must be returned

• If the namespace is valid, but the key is not known, an error must be thrown containing the invalid key

• If the namespace is not valid, an error must be thrown containing the invalid namespace

So a simple implementation could look like this:

public class Config {
	public static Object getValue(String namespace, String key) {
		if (!namespace.equals("database")) {
			throw new IllegalArgumentException("Unknown namespace: " + namespace);
		}
		if (!key.equals("name")) {
			throw new IllegalArgumentException("Unknown key: " + key);
		}
		return "db1";
	}
}

Our test method should simply test the three requirements. So basically we would just like to have to code the following:

// Successful tests 
Config.getValue("database", "name");	// result: "db1"

// Tests with errors
Config.getValue("database", "UNKNOWN");	// error: "Unknown key: UNKNOWN"
Config.getValue("UNKNOWN", "UNKNOWN");	// error: "Unknown namespace: UNKNOWN"

As we will see, MagicTest allows us to use exactly these three lines of code to cover all testing needs.

But how would we implement testing these requirements using JUnit and TestNG?

Traditional approach

Before TestNG entered the stage, you probably used JUnit 3 for testing – which offered no support at all for testing error cases. So you had to catch the exceptions manually and ended up with code like this:

public class ConfigTest_Traditional {
	@Test
	public void testGetValue() {
		assertEquals(Config.getValue("database", "name"), "db1");

		try {
			Config.getValue("database", "UNKNOWN");
			fail("Exception must be thrown");
		} catch (IllegalArgumentException e) {
			// expected exception
		}

		try {
			Config.getValue("UNKNOWN", "UNKNOWN");
			fail("Exception must be thrown");
		} catch (IllegalArgumentException e) {
			assertEquals(e.getMessage(), "Unknown namespace: UNKNOWN");
		}
	}
}

While this approach forced you to type a lot, you had the possibilty to check the exception messages for correctness and also check several exceptions in one test method.

TestNG

TestNG added the annotation expectedExceptions which allowed you to easily check whether an exception of the given type was thrown. This could save you quite some lines of code - if you had only to check a single expection case in your test method and did not have to check the exception message.

TestNG 5.10 then added the attribute expectedExceptionsMessageRegExp, which allows checking the content of the exception message.

public class ConfigTest_TestNG510 {
	@Test
	public void testConcat() {
		assertEquals(Config.getValue("database", "name"), "db1");
	}
	
	@Test(expectedExceptions = IllegalArgumentException.class)
	public void testConcatErr1() {
		Config.getValue("database", "UNKNOWN");
	}

	@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = ".*null.*")
	public void testConcatErr2() {
		Config.getValue("UNKNOWN", "UNKNOWN");
	}
}

JUnit

With version 4, JUnit adapted the annotation based configuration approach of TestNG and the ability to easily check for an expected exception. As the approach is the same, so are the drawbacks: no checking of exception message and only one exceptionper test method can be handled.

JUnit 4.7 added a concept called rules. One implementation of a rule is called ExpectedException and allows you to specify the expected exception with its message programmatically.

public class ConfigTest_Junit47 {
	@Test
	public void testGetValue() {
		assertEquals(Config.getValue("database", "name"), "db1");
	}
	
	@Test(expected = IllegalArgumentException.class)
	public void testGetValueErrJUnit40() {
		Config.getValue("database", "UNKNOWN");
	}

	@Rule
	public ExpectedException errJUnit47 = ExpectedException.none();

	@Test
	public void testGetValueErrJUnit47() {
		errJUnit47.expect(IllegalArgumentException.class);
		errJUnit47.expectMessage("Unknown namespace: UNKNOWN");
		Config.getValue("UNKNOWN", "UNKNOWN");
	}
}

Using ExpectedException it is possible to check the message of the exception, but if we compare the lines of code used for this approach with the traditional try-catch approach, we can conclude that both need five lines, so the benefit of this feature seems to remain quite low.

MagicTest

The visual approach introduced by MagicTest makes testing error cases as easy as sucessful ones. So we really just need to code three lines to test our three requirements:

public class ConfigTest {
	@Trace
	public void testGetValue() {
		// Successful tests 
		Config.getValue("database", "name");	// result: "db1"
		
		// Tests with errors
		Config.getValue("database", "UNKNOWN");	// error: "Unknown key: UNKNOWN"
		Config.getValue("UNKNOWN", "UNKNOWN");	//  error: "Unknown namespace: UNKNOWN"
	}
}

As MagicTest automatically catches and reports exceptions but then normally continues processing, we can have several exception cases in a row – something which is not possible using JUnit or TestNG without having to explicitly use try-catch clauses.

If we run this test, the test report shows us the result of the three method calls. As the test is run the first time, there is no reference result yet and the steps are therefore considered as failed.

If we then confirm the actual result to be correct by clicking on the Save link, it is saved as new reference result and therefore the test is successful.

Behind the scenes

How does this work? The magic lies in the @Trace annotation. It tells MagicTest to trace all calls to the method getValue and reports parameters and result.

To generate the needed output without having to manually code these statements, the byte code is instrumented before the test method is executed: So for each call to the getValue() method parameters, return value, and thrown exception are automatically traced.

So a call to the method under test will roughly look like this:

try {
   printParameters("database", "name");
   String result = Config.getValue("database", "name");
   printResult(result);
} catch (Throwable t) {
   printError(t);
}

The data traced out with the print-methods is then collected and visualized as HTML report.

Conclusion

We can try to collect the bare facts of the different approaches in a table:

Obviously the approach featured by MagicTest looks very promising.

Additionally the possibility to check successful and error cases in a single test method can greatly reduce the number of test methods needed – which may be easier to handle even if theory says that one test should just test a single thing.

Have a look at http://www.magicwerk.org/magictest to find out more about MagicTest.