Introducing a Parameterized Test Suite for JUnit 4

Parameterization of test cases is a well-known feature of JUnit 4. While some use cases also promote the parameterization of test suites, this feature is not available in the test framework by default. This article introduces an extension for JUnit 4 that provides a new Runner class for this purpose. Applicability in the context of automated GUI tests is shown.

Why Would You Parameterize a Test Suite?

Most of us Java developers are familiar with JUnit and its popular features. We typically define test classes and use test suites to aggregate these tests and give them a hierarchy. Test rules like @BeforeClass or @After are also widely used to set up a test's context. For my part, I knew that tests could also be parameterized and I had seen the Fibonacci example. But I think parameterization is a feature one doesn't have to use often.

Parameterized tests became valuable for me when I started automating GUI tests for web applications with Selenium and WebDriver: Although GUI tests are no unit tests for sure the JUnit framework is very handy for test definition, execution and result aggregation. Integration with CI is also available off the shelf. Once I had reached a good coverage of use cases in a single browser, I wanted to port these test cases to other browsers as well. Thus, I parameterized my tests with the browser being the parameter and ran these tests on Chrome, Firefox and Internet Explorer. Since all my tests should be executed on all browsers, the next logical step was to define these parameters at the test suite once instead of defining them repeatedly for every test class.

Then I stumbled upon a severe limitation of JUnit 4: You can parameterize your tests, but you can't parameterize the test suite that combines all your tests!?

Introducing ParameterizedSuite Runner

In JUnit 4 the execution of a test is done by the Runner and configured by the corresponding annotation  @RunWith. BlockJUnit4ClassRunner is the standard Runnerfor tests and chosen if you don't define any. For defining a test suite you configure @RunWith(Suite.class), similar to a parameterized test: @RunWith(Parameterized.class). So that's mutually exclusive.

But how can you parameterize a test suite if there's no @RunWith(ParameterizedSuite.class)? Well, now you can.

Wanting to define parameters on suite level and having the use case of test automation in mind, I implemented and published an open source library as an extension of JUnit 4. Integration is easy and uses the same patterns as in parameterization of single test classes.

Exemplary Integration

Start by defining another dependency in addition to JUnit 4.
For Maven:

<dependency>
    <!-- https://github.com/PeterWippermann/parameterized-suite -->
    <groupId>com.github.peterwippermann.junit4</groupId>
    <artifactId>parameterized-suite</artifactId>
    <version>1.1.0</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>

Make your test suite a ParameterizedSuite and declare a method to produce parameters - just as in a parameterized test class.

import com.github.peterwippermann.junit4.parameterizedsuite.ParameterizedSuite;

@RunWith(ParameterizedSuite.class)
@SuiteClasses({
    OneTest.class,
    TwoTest.class
})
public class MyParameterizedTestSuite {

    @Parameters(name = "Parameters are {0} and {1}")
    public static Object[] params() {
        return new Object[][] {
            {
                'A',
                1
            }, {
                'B',
                2
            }, {
                'C',
                3
            }
        };
    }

    /**
     * Always provide a target for the defined parameters - even if you only want to access them in the suite's child classes.
     * Instead of fields you can also use a constructor with parameters.
     */
    @Parameter(0)
    public char myCharParameter;

    @Parameter(1)
    public int myIntParameter;
}

In JUnit 4 a test suite has no access to the instances of the test classes. This is by the framework’s design. Thus, the parameters have to be transferred via a static context, which is the ParameterContext singleton. Instead of instantiating their own parameters, parameterized test classes refer to the ParameterContext.

@RunWith(Parameterized.class)
public class MyParameterizedTestCase {

    @Parameters(name = "Parameters are {0} and {1}")
    public static Iterable < Object[] > params() {
        if (ParameterContext.isParameterSet()) {
            return Collections.singletonList(ParameterContext.getParameter(Object[].class));
        } else {
            // if the test case is not executed as part of a ParameterizedSuite, you can define fallback parameters
        }
    }

    private char myCharParameter;

    private int myIntParameter;

    public MyParameterizedTestCase(char c, int i) {
        super();
        this.myCharParameter = c;
        this.myIntParameter = i;
    }

    @Test
    public void testWithParameters() {
        // ...
    }
}

Execution Order

The set of test classes in a test suite and the set of parameters the suite defines lead to two alternative execution strategies:

1. Execute all tests per parameter

2. Execute each test with all parameters in a row

The current implementation of ParameterizedSuite implements the first strategy, i.e. for each parameter all tests are run before going over to the next parameter. That way instances of parameter objects are handed over from test to test and allow to share state between tests.

Sharing state between tests became especially important in my case for GUI tests: A browser could be set up once and be reused in all tests. That way the session, cookies, current URL and history persisted.

Due to the execution order, only a single parameter instance may be accessed from the ParameterContext at once. During the test suite's execution the test classes will be instantiated multiple times and, consequently, the method annotated with @Parameters will also be called as often.

Enabling Test Rules for Test Suites

 TestRules  encapsulate the execution of a test case and may take additional actions before or after a test is run. Popular examples are @Before, @AfterClass, and @ExternalResource. These test rules are not applicable to default test suites since a test suite is never instantiated. It rather acts as a declaration of the grouped test classes. Opposed to this, a parameterized test suite will also be instantiated and respects any definitions of test rules.

Notes to State-Sharing and Thread-Safety

Parameters of a test suite are only instantiated once and reused in all test classes, which implies these objects are shared state between the test classes! This behavior is actually identical to the parameterisation of single classes using @Parameterized. Nevertheless, state sharing has to be used with care and considered for race conditions. In the context of GUI testing, the sharing of the browser instances is explicitly wanted to preserve the web application’s state.

Regarding race conditions: The current implementation of the ParameterContextis not thread-safe. A concurrent implementation could be based on ThreadLocal. Feature requests or even pull requests are welcome at the project’s site.

Conclusion

The parameterization of test suites is required by few JUnit 4 users, but many of those seem to come from a GUI test automation background. Supporting this feature in the JUnit 4 framework has been discussed but not been carried out. The library parameterized-suite that has been introduced in this article is intended to be an extension of the framework and fills the described feature gap.