Blow Up Your JUnit Tests With Permutations

Blow Up Your JUnit Tests With Permutations

Writing JUnit tests can be a tedious and boring process. Learn how you can improve your tests classes using permutations in combination with TestFactory methods and DynamicTest objects with a minimum of coding effort.

In this article, I will use the Java stream ORM Speedment because it includes a ready-made Permutation class and thereby helps me save development time. Speedment otherwise allows database tables to be connected to standard Java streams. Speedment is an open-source tool and is also available in a free version for commercial databases.

Testing a Stream

Consider the following JUnit5 test:

@Test
void test() {

    List<String> actual = Stream.of("CCC", "A", "BB", "BB")
        .filter(string -> string.length() > 1)
        .sorted()
        .distinct()
        .collect(toList());

    List<String> expected = Arrays.asList("BB", "CCC");

    assertEquals(actual, expected);
}

As can be seen, this test creates a Stream with the elements "CCC", "A", "BB' and "BB" and then applies a filter that will remove the "A" element (because its length is not greater than 1). After that, the elements are sorted, so that we have the elements "BB", "BB" and "CCC" in the stream. Then, a distinct operation is applied, removing all duplicates in the stream, leaving the elements "BB" and "CCC" before the final terminating operator is invoked whereby these remaining elements are collected to a List.

After some consideration, it can be understood that the order in which the intermediate operations filter(), sorted() and distinct() are applied is irrelevant. Thus, regardless of the order of operator application, we expect the same result.

But, how can we wite a JUnit5 test that proves that the order is irelevant for all permutations without writing individual test cases for all six permutations manually?

Using a TestFactory

Instead of writing individual tests, we can use a TestFactory to produce any number of DynamicTest objects. Here is a short example demonstrating the concept:

@TestFactory
Stream<DynamicTest> testDynamicTestStream() {
    return Stream.of(
        DynamicTest.dynamicTest("A", () -> assertEquals("A", "A")),
        DynamicTest.dynamicTest("B", () -> assertEquals("B", "B"))
    );
}

This will produce two, arguably meaningless, tests named "A" and "B". Note how we conveniently can return a Stream of DynamicTest objects without first having to collect them into a Collection such as a List.

Using Permutations

The Permutation class can be used to create all possible combinations of items of any type T. Here is a simple example with the type String:

Permutation.of("A", "B", "C")
            .map(
                is -> is.collect(toList())
            )
            .forEach(System.out::println);

Because Permutation creates a Stream of a Stream of type T, we have added an intermediary map operation where we collect the inner Stream to a List. The code above will produce the following output:

[A, B, C]
[A, C, B] 
[B, A, C] 
[B, C, A] 
[C, A, B] 
[C, B, A]

It is easy to prove that this is all the ways one can combine "A", "B" and "C" whereby each element shall occur exactly one time.

Creating the Operators

In this article, I have opted to create Java objects for the intermediate operations instead of using lambdas because I want to override the toString() method and use that for method identification. Under other circumstances, it would have sufficed to use lambdas or method references directly:

UnaryOperator<Stream<String>> FILTER_OP = new UnaryOperator<Stream<String>>() {
    @Override
    public Stream<String> apply(Stream<String> s) {
        return s.filter(string -> string.length() > 1);
    }

    @Override
    public String toString() {
        return "filter";
    }
 };


UnaryOperator<Stream<String>> DISTINCT_OP = new UnaryOperator<Stream<String>>() {
    @Override
    public Stream<String> apply(Stream<String> s) {
        return s.distinct();
    }

    @Override
    public String toString() {
        return "distinct";
    }
};

UnaryOperator<Stream<String>> SORTED_OP = new UnaryOperator<Stream<String>>() {
    @Override
    public Stream<String> apply(Stream<String> s) {
        return s.sorted();
    }

    @Override
    public String toString() {
        return "sorted";
    }
};

Testing the Permutations

We can now easily test the workings of Permutations on our Operators:

void printAllPermutations() {

     Permutation.of(
        FILTER_OP,
        DISTINCT_OP,
        SORTED_OP
    )
    .map(
        is -> is.collect(toList())
    )
    .forEach(System.out::println);
}

This will produce the following output:

[filter, distinct, sorted]
[filter, sorted, distinct]
[distinct, filter, sorted]
[distinct, sorted, filter]
[sorted, filter, distinct]
[sorted, distinct, filter]

As can be seen, these are all permutation of the intermediate operations we want to test.

Stitching It Up

By combining the learnings above, we can create our TestFactory that will test all permutations of the intermediate operations applied to the initial stream:

@TestFactory
Stream<DynamicTest> testAllPermutations() {

    List<String> expected = Arrays.asList("BB", "CCC");

    return Permutation.of(
        FILTER_OP,
        DISTINCT_OP,
        SORTED_OP
    )
        .map(is -> is.collect(toList()))
        .map(l -> DynamicTest.dynamicTest(
            l.toString(),
            () -> {
                List<String> actual = l.stream()
                    .reduce(
                        Stream.of("CCC", "A", "BB", "BB"),
                        (s, oper) -> oper.apply(s),
                        (a, b) -> a
                    ).collect(toList());

                assertEquals(expected, actual);
            }
            )
        );
}

Note how we are using the Stream::reduce method to progressively apply the intermediate operations on the initial Stream.of("CCC", "A", "BB", "BB"). The combiner lambda (a, b) -> a is just a dummy, only to be used for combining parallel streams (which are not used here).

Blow-Up Warning

A final warning on the inherent mathematical complexity of permutation is in its place. The complexity of permutation is, by definition, O(n!) meaning, for example, adding just one element to a permutation of an existing eight element will increase the number of permutations from 40,320 to 362,880.

This is a double-edged sword. We get a lot of tests almost for free but we have to pay the price of executing each of the tests on each build.

Code

The source code for the tests can be found here.

Speedment ORM can be downloaded here

Conclusions

The Permutation, DynamicTest and TestFactory classes are excellent building blocks for creating programmatic JUnit5 tests.

Take care not to use too many elements in your permutations. "Blow up" can mean two different things ...