Over a million developers have joined DZone.

The Magic Testing Challenge

DZone's Guide to

The Magic Testing Challenge

· DevOps Zone ·
Free Resource

Learn how integrating security into DevOps to deliver "DevSecOps" requires changing mindsets, processes and technology.

Writing tests belongs to a developer's daily life - but nobody likes it. Testing is often repetive and annoying and it takes just too much of your precious time. So the perfect test framework must make writing and maintaining tests as efficient as possible.

Enter MagicTest (http://www.magicwerk.org/magictest), a new test framework for Java. It features a a visual approach which makes testing a breeze. To show the advantages of MagicTest, I invite you to join a testing challenge.

The Task

We must test a class Tags with the following features (see listing 1 for implementation):

  • it maintains a list of tags

  • a tag is composed of the tag name (not null) and a count (positive integer)

  • the list is always ordered by descending tag count

How many lines of code will you need to test all these requirements? Will it be half the number of the productive code, about the same or may be even more?

The Magic Test

What if I tell you that MagicTest just needs 10 statements to get 100% test coverage? Here they are:

      Tags tags = new Tags();
      tags.add("x", 0);
      tags.add("x", -1);
      tags.add("a", 1);
      tags.add("b", 2);
      tags.add("a", 2);
      tags.add("a", 0);
      tags.add("a", -3);
      tags.add("b", -3);
      tags.add(null, 1);

You can immediately notice major differences compared to traditional unit tests written for TestNG or JUnit:

  • No assert statements needed to check the expected results

  • No exception catching necessary needed for negative tests

They are not needed because MagicTest automatically traces calls to the method under test with all relevant information including parameters and result (if successful) or exception (on failure). MagicTest collects this test output and compares it with a stored reference output: if they are equal, the test is considered successful.

If we run our test with MagicTest, we get the following comprehensible test report:

Test Report

How does this work?

The Test in Details

The whole test class looks like this:

public class TagsTest {
      parameters = Trace.THIS|Trace.ALL_PARAMS, 
      result = Trace.THIS|Trace.RESULT,
      title = "Test all methods of Tags")
   public void testTags() {
      setDescription("Create empty tags list");
      Tags tags = new Tags();
      setDescription("Tag with count 0 is ignored on add");
      tags.add("x", 0);
      setDescription("Adding a tag with negative count is rejected ");
      tags.add("x", -1);
      setDescription("Maintain tags order");
      tags.add("a", 1);
      tags.add("b", 2);
      tags.add("a", 2);
      setDescription("Adding 0 to an existing tag is ignored");
      tags.add("a", 0);
      setDescription("Tag 'a' with new count 0 is removed");
      tags.add("a", -3);
      setDescription("Changing a tag count to a negative value is rejected");
      tags.add("b", -3);
      setDescription("Tag name must not be null");
      tags.add(null, 1);

If we step through the code line by line, we see the following:

  • Naming convention: as the class under test is named Tags, we name our test class TagsTest to let MagicTest detect the relationship

  • @Trace annotation: we use traceMethod with the regular expression ".*" to specify that we want to trace calls to all methods of the class under test

  • Report.setDescription(): a textual description for test steps can be provided

It may look unfamiliar to test all requirements in a single test method. The fact however that test excecution continues even in case of an exception, makes this possible and even reasonable as much less code is needed because you can benefit from state already built up. So you get 100% test coverage with these few lines of code.

I think the puristical approach to test just one thing in a test is only widely accepted because it is backed up by a very practical reason: if you write complex test methods and one of your assertion fails in the middle of the execution, testing just ends without any information about the upcoming test steps. So you have to fix the issue and run the test again - probably just to see the next error... Obviously this is not a problem with MagicTest.

Let the Challenge Begin

To run the test yourself, you will need to download the MagicTest Eclipse plug-in. Copy it into the Eclipse dropins folder and restart Eclipse. Then download the attached Eclipse project and import it into your workspace. Run the test class TagsTest by executing Run As / MagicTest.

After the first run, the test report will show up and all test steps will be red. This is the MagicTest way of telling you that a step has failed. In our case, the steps just fail because MagicTest simply does not know anything about the expected result. So we carefully check the output and confirm its correctness by clicking on the save button. Now all steps are green - and the test is successful.

You have now seen how efficiently this test can be realized using MagicTest - it even looked like fun. Does your test tool accept the challenge? How many minutes and lines does it take you to write the test? I'm looking forward to your contributions!

Learn how enterprises are using tools to automate security in their DevOps toolchain with these DevSecOps Reference Architectures.


Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}