Strengthening Testing through Mutation: A DevOps Engineer’s Experience

I work as a DevOps engineer for a large public-facing application, which has around 90+ microservices (Java-based). Consistently, we are hit by scenarios that we’d discover in the field, which were not caught in any of our testing. Despite improving our test strategies and code coverage assessments, we were unable to assess the "strength" of our test cases.

We looked around for options and solutions that could help us to be more sure about our test cases and the code we develop. As a DevOps engineer, my responsibility was to identify options and trial them to fix this problem and include it as part of our automated build process.

We came across the mutation testing approach, which was offering answers to the questions/concerns we had around our test cases. I was inclined to try this approach to help developers write more concrete test cases and once proven, I could use it as part of the build process to make it completely automated.

You may also enjoy: Mutation Testing: Covering Your Code With the Right Test Cases

Traditional test coverage approaches have a drawback, as they only check the code executed by the test cases and show code that is not covered. They do not help us evaluate or identify hidden issues in the code. Mutation enables developers to write strong cases (not just indicate if the code was executed while running the test cases) and inherently makes sure the code is tested thoroughly.

Given that ours is a Java tech stack, we had to look at relevant options, which would help us do mutation testing and analysis in our ecosystem. PIT turned out to be the most viable option based on our primary needs.

I’ve attempted to walk through the concepts I’ve understood and the key aspects I discovered in this journey. Some of the biggest tasks were to evangelize this to our developers, enable PIT Mutation Testing successfully on our ecosystem, and ingrain it as part of our Build process. I am summarizing my learnings and challenges we faced through this journey: I hope to benefit my fellow engineers.

I am starting with some fundamentals about mutation testing; you may find these details publically available across sites/portals.

What Is Mutation Testing?

In mutation testing, we modify our source code and run the test cases against our code. These modifications of source code are called mutants. Suppose if you have written (x > y) in your code, a mutant of this can be the code with (x < y).

Once the mutant code runs against your test cases, if it survives (passes the test cases), it becomes part of survived mutants. If the mutant fails while running against your test cases, it becomes part of the killed mutants. The higher the number of killed mutants, the stronger your test cases are.

The reason for running mutations is to make sure to write concrete test cases that should kill a mutant.

Types of Mutation Testing

Mutation testing can be divided into three categories: value mutation, decision mutation, and statement mutation.

Value Mutation

It changes a value in the source code to detect errors.

Example: if(x=) change x to something other than 20.

Decision Mutations

These change the decisions/conditions to test your cases for design errors.

Example: if ( newResponse   instanceof   ErrorResponse), change the  newResponse object to Null

Statement Mutations
This changes a statement by removing or adding the same line to test whether the developer has copied and pasted the code.

Mutation Score

The mutation score is the unit that defines how strong/effective your mutation analysis is. It is defined as:

Mutation Score = (Killed Mutants / Total number of Mutants) * 100

Enabling PIT With Your Java Source Code

There are multiple solutions that can enable mutation analysis in your Java code. However, in this article, will go through the PIT tool to enable mutation testing. PIT is faster and easier to use compared to other tools. Also, it’s actively getting developed and supported.

Refer to the Github page for PIT.

Mutators in PIT

PIT currently provides some built-in mutators, of which, most are activated by default. The default set can be overridden, and different operators can be selected, bypassing the names of the required operators to the mutators parameter.

Default Mutators

Conditionals Boundary Mutator — replaces the relational operators <, <=, >, >=

Increments Mutator — replace increments with decrements and vice versa.

Invert Negatives Mutator — inverts negation of integer and floating-point number.

Math Mutator — replaces binary arithmetic operations for either integer or floating-point arithmetic operations with other operations.

Negate Conditionals Mutator — negate the conditional checks.

Return Values Mutator — mutates the return values of the method call depending on the return type of the method. For Object return type, mutates to null.

Void Method Calls Mutator — removes method calls to void methods.

Non-Default Mutators

Constructor Calls Mutator — replaces constructor calls with null.

Inline Constant Mutator — mutates inline constants and replaces default values based on its data type.

Non Void Method Calls Mutator — removes method calls to non-void methods, and their return value is replaced by the Java Default Value for that specific type

Remove Conditionals Mutator — remove all conditionals statements, such that, guarded statements always execute.

Member Variable Mutator (Experimental) — removing assignments to member variables and also final members.

Switch Mutator (Experimental) — mutates the switch statement by replacing the default label.

Setting Up PIT

Configure PIT Plugin in Your IDE

You can go to the Eclipse market place, download the PIT plugin, and run the mutation tests.

Enabling Pit for Your Java code

1. Add a Dependency

You eed to add a dependency for pitest in pox.xml, so the pitest jar should get downloaded.

xxxxxxxxxx

<dependency>

<groupId>org.pitest</groupId>

<artifactId>pitest</artifactId>

<version>1.4.0</version>

</dependency>

2. Configure PIT Plugin

Once the dependency is added, we need to set up the pitest plugin configuration. The following snippet is from the pom.xml file and is an example of pitest plugin setup:

xxxxxxxxxx

<plugin>

    <groupId>org.pitest</groupId>

    <artifactId>pitest-maven</artifactId>

    <version>1.4.0</version>

    <configuration>

        <outputFormats>

            <outputFormat>XML</outputFormat>

            <outputFormat>HTML</outputFormat>

        </outputFormats>

        <avoidCallsTo>

            <avoidCallsTo>org.apache.log4j.*</avoidCallsTo>

            <avoidCallsTo>org.slf4j.*</avoidCallsTo>

        </avoidCallsTo>

        <targetClasses>

            <param>com.project.*</param>

            <param>com.project.*</param>

        </targetClasses>

        <targetTests>

            <param>com.projecttest.*</param>

        </targetTests>

        <threads>4</threads>

        <historyInputLocation>target/pitHistory.txt</historyInputLocation>

        <historyOutputLocation>target/pitHistory.txt</historyOutputLocation>

        <timestampedReports>false</timestampedReports>

    </configuration>

</plugin>

NOTE: Please add the plugin configuration below the PLUGIN MANAGEMENT tag; adding it elsewhere will cause errors.

3. Plugin Config in Detail:

Update the following in the configuration based in your project package information.

Note: Only packages where you want to enable mutation analysis are needed inside Target Classes. Update with a list of packages and classes that you want to be considered for the scope of mutation.

xxxxxxxxxx

<targetClasses>

            <param>com.project.*</param>

            <param>com.project.*</param>

</targetClasses>

Note: Only test cases under which you want to run mutation testing are needed inside Target Tests. The list of globs of Test Classes and/or packages will be used by Pitest for checking the mutation.

xxxxxxxxxx

<targetTests>

            <param>com.projecttest.*</param>

</targetTests>

Avoid Calls to:

List of packages and classes that are considered outside the scope of mutation. Any lines of code containing calls to these classes will not be mutated.

xxxxxxxxxx

       <avoidCallsTo>

         <avoidCallsTo>org.apache.log4j.*</avoidCallsTo>

            <avoidCallsTo>org.slf4j.*</avoidCallsTo>

        </avoidCallsTo>

4. Additional Plugin Config in Detail

Output Formats:

A list of formats to write the mutation results once the mutations are analyzed. HTML and XML reports are needed for SonarQube to generate Mutation details. (If you are planning to send the test results to Sonarqube, as a quality gate process.)

xxxxxxxxxx

        <outputFormats>

            <outputFormat>XML</outputFormat>

            <outputFormat>HTML</outputFormat>

        </outputFormats>

History:

The ath to a file containing history information for incremental analysis to speed up the Mutation testing.

xxxxxxxxxx

<historyInputLocation>target/pitHistory.txt</historyInputLocation>

<historyOutputLocation>target/pitHistory.txt</historyOutputLocation>

Mutators:

List of mutation operators to apply for Mutation Testing. If not given, the mutation test will proceed with default Mutators.

xxxxxxxxxx

<mutators>

<mutator>CONSTRUCTOR_CALLS</mutator>

<mutator>NON_VOID_METHOD_CALLS</mutator>

</mutators>

Threads

The number of threads to use when mutation testing. By default, a single thread will be used.

xxxxxxxxxx

<threads>5</threads>

Running Mutation Analysis:

Run the following command to generate the mutation analysis reports:

xxxxxxxxxx

clean test verify org.pitest:pitest-maven:mutationCoverage

This command will create a pit-reports folder inside the target. Go inside the pit-reports folder and open the index.html file. This will give you details of mutation analysis.

• The reports produced by PIT are in an easy to read format combining line coverage and mutation coverage information. For a good mutation coverage result, it is recommended to run a project that has at least 80-90% line coverage.

Click on the main class to check the reports — light green shows line coverage, and dark green shows mutation coverage. You can also see what mutations run on the particular block of code.

The details of the mutation killed/survived are shown pertaining to the respective number.

The example snippet is taken from the coverage report of a test project.

In the report, light pink will show a lack of line coverage, and dark pink will show a lack of mutation coverage.

Sending Reports to Sonar

Sonarqube also offers multiple plugins to enable pit analysis in its quality gate. With these, you can set up rules for your quality gate on the basis of the number of mutants, killed mutants, survived mutants, and mutation coverage.

Refer to the following links for Pit plugins in Sonar:

• https://github.com/VinodAnandan/sonar-pitest.
• http://www.sonarplugins.com/pitest.

Integration With SONAR and Jenkins for Quality Gate

For integrating mutation analysis with Sonar, you can download the following plugins in your Sonar and add the rules in your quality gate.

Once downloaded the plugin, you need to add the rules in your default quality profile. Once the rules are in the default quality profile, you can set up the quality gate and enable the quality gate in your build pipeline, so if the quality gate status is an error, the build should fail.

Conclusion

Using Pit resolved lots of issues in our code. Our test cases became more concrete, and we started focusing more on the test case strength as well as the test coverage. Our test cases are now more focused on testing the logic of the code instead of just the test coverage. This helped us fix bugs and improved the build time itself.

However, there were some challenges. For example, setting up PIT and modifying our test cases was a little time-consuming. Also, the build job now takes a little extra time when compared to our previous process. We improved this over the time by using the incremental analysis.

From the developers’ side, they had to focus on which class they wanted to run the mutation using which test cases. Also, they had to select the type of mutator for the basis of the core functionality.

Further Reading

Code Smells: Mutation.

Unit Testing Best Practices: How to Get the Most Out of Your Test Automation.

Mutation Testing: Covering Your Code With the Right Test Cases (Part 2).