Parameterized Tests and Theories

In this series on unit testing with JUnit, we learned several unit testing aspects and how to implement them with JUnit. We can summarize the series till now as:

• Part 1: Creating a basic unit test both using Maven and IntelliJ
• Part 2: Using assertions and annotations
• Part 3: Using assertThat with Hamcrest matchers

In this post, we will learn about parameterized tests and theories.

JUnit Parameterized Tests

While testing, it’s common to execute a series of tests which differ only by input values and expected results. As an example, if you are testing a method that validates email IDs, you should test it with different email ID formats to check whether the validations are getting correctly done. But testing each email ID format separately, will result in duplicate or boilerplate code. It is better to abstract the email ID test into a single test method and provide it a list of all input values and expected results. JUnit supports this functionality through parameterized tests.

To see how parameterized test works, we’ll start with a class with two methods which we will put under test.

EmailIdUtility.java

package guru.springframework.unittest.parameterized;

import java.util.regex.Matcher;
import java.util.regex.Pattern;


public class EmailIdUtility {
    public static String createEmailID(String firstPart,String secondPart){
        String generatedId = firstPart+"."+secondPart+"@testdomain.com";
        return generatedId;
    }
    public static boolean isValid(String email){
        String regex = "^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|(([a-zA-Z\\-0-9]+\\.)+[a-zA-Z]{2,}))$";
        Pattern pattern = Pattern.compile(regex);
        Matcher m = pattern.matcher(email);
        return m.matches();
    }
}

The EmailIdUtility class above has two utility methods. The createEmailID() method accepts two String parameters and generates an email ID in a specific format. The format is simple – If you pass mark and doe as parameters to this method, it returns mark.doe@testdomain.com. The second isValid() method accepts an email ID as a String, uses regular expression to validate it’s format, and returns the validation result.

We will first test the isValid() method with a parameterized test. JUnit runs a parameterized test with a special runner, Parameterized and we need to declare it with the @RuntWith annotation. In a parameterized test class, we declare instance variables corresponding to the number of inputs to the test and the output. As the isValid() method under test takes a single String parameter and returns a boolean, we declare two corresponding variables. For a parameterized test, we need to provide a constructor, which will initialize the variables. 

EmailIdValidatorTest.class

. . .
@RunWith(value = Parameterized.class)
public class EmailIdValidatorTest {

    private String emailId;
    private boolean expected;

    public EmailIdValidatorTest(String emailId, boolean expected) {
        this.emailId = emailId;
        this.expected = expected;
    }
. . .

We also need to provide a public static method annotated with @Parameters annotation. This method will be used by the test runner to feed data into our tests.

. . .
@Parameterized.Parameters(name= "{index}: isValid({0})={1}")
public static Iterable<Object[]> data() {
    return Arrays.asList(new Object[][]{
                    {"mary@testdomain.com", true},
                    {"mary.smith@testdomain.com", true},
                    {"mary_smith123@testdomain.com", true},
                    {"mary@testdomaindotcom", false},
                    {"mary-smith@testdomain", false},
                    {"testdomain.com", false}
            }
    );
}
. . . 

The @Parametersannotated method above returns a collection of test data elements (which in turn are stored in an array). Test data elements are the different variations of the data, including the input as well as expected output needed by the test. The number of test data elements in each array must be the same with the number of parameters we declared in the constructor.

When the test runs, the runner instantiates the test class once for each set of parameters, passing the parameters to the constructor that we wrote. The constructor then initializes the instance variables we declared.

Notice the optional name attribute we wrote in the @Parameters annotation to identify the parameters being used in the test run. This attribute contains placeholders that are replaced at run time.

• {index}: The current parameter index, starting from 0.
• {0}, {1}, …: The first, second, and so on, parameter value. As an example, for the parameter{“mary@testdomain.com”, true}, then {0} = mary@testdomain.com and {1} = true.

Finally, we write the test method annotated with @Test. The complete code of the parameterized test is this.

EmailIdValidatorTest.java

package guru.springframework.unittest.parameterized;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import static org.hamcrest.CoreMatchers.*;

import java.util.Arrays;

import static org.junit.Assert.*;
@RunWith(value = Parameterized.class)
public class EmailIdValidatorTest {

    private String emailId;
    private boolean expected;

    public EmailIdValidatorTest(String emailId, boolean expected) {
        this.emailId = emailId;
        this.expected = expected;
    }
    @Parameterized.Parameters(name= "{index}: isValid({0})={1}")
    public static Iterable<Object[]> data() {
        return Arrays.asList(new Object[][]{
                        {"mary@testdomain.com", true},
                        {"mary.smith@testdomain.com", true},
                        {"mary_smith123@testdomain.com", true},
                        {"mary@testdomaindotcom", false},
                        {"mary-smith@testdomain", false},
                        {"testdomain.com", false}
                }
        );
    }
    @Test
    public void testIsValidEmailId() throws Exception {
        boolean actual= EmailIdUtility.isValid(emailId);
        assertThat(actual, is(equalTo(expected)));
    }
}

The output on running the parameterized test in IntelliJ is this.

JUnit Theories

In a parameterized test, the test data elements are statically defined and you as the programmer are responsible for figuring out what data is needed for a particular range of tests. At times, you will likely want to make tests more generalized. Say, instead of testing for specific values, you might require to test for some wider range of acceptable input values. For this scenario, JUnit provides theories.

A theory is a special test method that a special JUnit runner (Theories) executes. To use the runner, annotate your test class with the @RunWith(Theories.class) annotation. The Theories runner executes a theory against several data inputs called data points. A theory is annotated with @Theory, but unlike normal @Test methods, a @Theory method has parameters. In order to fill these parameters with values, the Theories runner uses values of the data points having the same type.

There are two types of data points. You use them through the following two annotations:

• @DataPoint: Annotates a field or method as a single data point. The value of the field or that the method returns will be used as a potential parameter for theories having the same type.
• @DataPoints: Annotates an array or iterable-type field or method as a full array of data points. The values in the array or iterable will be used as potential parameters for theories having the same type. Use this annotation to avoid single data point fields cluttering your code.

Note: All data point fields and methods must be declared as public and static.

. . .
@DataPoint
public static String name="mary";

@DataPoints
public static String[] names() {
    return new String[]{"first","second","abc","123"};
}
. . .

In the code example above, we annotated a String field with the @DataPoint annotation and a names() method that returns a String[] with the @DataPoints annotation.

Creating a JUnit Threory

Recall the createEmailID() method that we wrote earlier on this post – “The createEmailID() method accepts two String parameters and generates an email ID in a specific format.” A test theory that we can establish is “Provided stringA and stringB passed to createEmailID() are non-null, it will return an email ID containing both stringA and stringB ”. This is how we can represent the theory.

. . .
@Theory
public void testCreateEmailID(String firstPart, String secondPart) throws Exception {
 String actual= EmailIdUtility.createEmailID(firstPart,secondPart);
 assertThat(actual, is(allOf(containsString(firstPart), containsString(secondPart))));
}
. . .

The testCreateEmailID() theory we wrote accepts two String parameters. At run time, theTheories runner will call testCreateEmailID() passing every possible combination of the data points we defined of type String. For example (mary,mary), (mary,first), (mary,second), and so on.

Assumptions

It’s very common for theories NOT to be valid for certain cases. You can exclude these from a test using assumptions, which basically means “don’t run this test if these conditions don’t apply“. In our theory, an assumption is that the parameters passed to the createEmailID() method under test arenon-null values.

If an assumption fails, the data point is silently ignored. Programmatically, we add assumptions to theories through one of the many methods of the Assume class.
Here is our modified theory with assumptions.

. . .
@Theory
public void testCreateEmailID(String firstPart, String secondPart) throws Exception {
    assumeNotNull(firstPart, secondPart);
    assumeThat(firstPart, notNullValue());
    assumeThat(secondPart, notNullValue());
    String actual= EmailIdUtility.createEmailID(firstPart,secondPart);
     assertThat(actual, is(allOf(containsString(firstPart),  containsString(secondPart))));
}
. . .

In the code above, we used assumeNotNull because we assume that the parameters passed to createEmailID() are non-null values. Therefore, even if a null data point exists and the test runner passes it to our theory, the assumption will fail and the data point will be ignored.
The two assumeThat we wrote together performs exactly the same function as assumeNotNull. I have included them only for demonstrating the usage of assumeThat, which you can see is very similar to assertThat we covered in the earlier post.

The following is the complete code using a theory to test the createEmailID() method.

EmailIDCreatorTest.java

package guru.springframework.unittest.parameterized;

import org.junit.Test;
import org.junit.experimental.theories.DataPoint;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import java.util.Arrays;

import static org.hamcrest.CoreMatchers.*;
import static org.hamcrest.CoreMatchers.containsString;
import static org.junit.Assert.*;
import static org.junit.Assume.assumeNotNull;
import static org.junit.Assume.assumeThat;

@RunWith(Theories.class)
public class EmailIDCreatorTest {

    @DataPoints
    public static String[] names() {
        return new String[]{"first","second","abc","123",null};
    }
    @DataPoint
    public static String name="mary";
    /*Generated Email ID returned by EmailIdUtility.createEmailID must contain first part and second part passed to it*/
    @Theory
    public void testCreateEmailID(String firstPart, String secondPart) throws Exception {
        System.out.println(String.format("Testing with %s and %s", firstPart, secondPart));
        assumeNotNull(firstPart, secondPart);
        /*Same assumptions as assumeNotNull(). Added only  to demonstrate usage of assertThat*/
        assumeThat(firstPart, notNullValue());
        assumeThat(secondPart, notNullValue());
        String actual= EmailIdUtility.createEmailID(firstPart,secondPart);
        System.out.println(String.format("Actual: %s \n", actual));
        assertThat(actual, is(allOf(containsString(firstPart), containsString(secondPart))));
    }
}

In the test class above, I have included null as a data point in the return statement of Line 23 for our assumptions and couple of System.out.println() statements to trace how parameters are passed to theories at run time.

Here is the output of the test in IntelliJ: 

Also, here is the output I got while running the test with Maven for your review:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running guru.springframework.unittest.parameterized.EmailIDCreatorTest
Testing with mary and mary
Actual: mary.mary@testdomain.com 

Testing with mary and first
Actual: mary.first@testdomain.com 

Testing with mary and second
Actual: mary.second@testdomain.com 

Testing with mary and abc
Actual: mary.abc@testdomain.com 

Testing with mary and 123
Actual: mary.123@testdomain.com 

Testing with mary and null
Testing with first and mary
Actual: first.mary@testdomain.com 

Testing with first and first
Actual: first.first@testdomain.com 

Testing with first and second
Actual: first.second@testdomain.com 

Testing with first and abc
Actual: first.abc@testdomain.com 

Testing with first and 123
Actual: first.123@testdomain.com 

Testing with first and null
Testing with second and mary
Actual: second.mary@testdomain.com 

Testing with second and first
Actual: second.first@testdomain.com 

Testing with second and second
Actual: second.second@testdomain.com 

Testing with second and abc
Actual: second.abc@testdomain.com 

Testing with second and 123
Actual: second.123@testdomain.com 

Testing with second and null
Testing with abc and mary
Actual: abc.mary@testdomain.com 

Testing with abc and first
Actual: abc.first@testdomain.com 

Testing with abc and second
Actual: abc.second@testdomain.com 

Testing with abc and abc
Actual: abc.abc@testdomain.com 

Testing with abc and 123
Actual: abc.123@testdomain.com 

Testing with abc and null
Testing with 123 and mary
Actual: 123.mary@testdomain.com 

Testing with 123 and first
Actual: 123.first@testdomain.com 

Testing with 123 and second
Actual: 123.second@testdomain.com 

Testing with 123 and abc
Actual: 123.abc@testdomain.com 

Testing with 123 and 123
Actual: 123.123@testdomain.com 

Testing with 123 and null
Testing with null and mary
Testing with null and first
Testing with null and second
Testing with null and abc
Testing with null and 123
Testing with null and null
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.076 sec

n the output above, notice that whenever a null value is being passed to the theory, the remaining part of the theory after assumeNotNull does not execute.

Summary

Parameterized tests in JUnit helps remove boiler plate test code and that saves time while writing test code. This is particularly useful during Enterprise Application Development with the Spring Framework. However, a common complaint is that when a parameterized test fails it’s very hard to see the parameters which caused it to fail. By properly naming the @Parameters annotation and great unit testing support that modern IDEs provide, such complains are quickly failing to hold grounds. Although theories are less commonly used, they are powerful instruments in any programmers test toolkit. Theories not only makes your tests more expressive but you will see how your test data becomes more independent of the code you’re testing. This will improve the quality of your code, since you’re more likely to hit edge cases, which you may have previously overlooked.