{{announcement.body}}
{{announcement.title}}

What Is Your Test Quality?

DZone 's Guide to

What Is Your Test Quality?

In this post, we will take a look at mutation testing which will test the quality of your unit tests.

· Performance Zone ·
Free Resource

You have consistently written unit tests and you have a line coverage of, let us say, 80% and all of your tests pass. Pretty good, isn’t it? But then you change your code and still all of your tests pass although you have changed code which is covered by your unit tests. In this post, we will take a look at mutation testing which will test the quality of your unit tests.

Introduction

We value our code quality very much. We execute static code analysis, we write unit tests, integration tests, etc. We can set minimum threshold values for certain metrics: we do not allow critical, major analysis issues, all tests must pass, etc. These are all good and valuable items to check in the pipeline to ensure a certain quality level. But what is the value of the metric itself? 

When your unit tests do not assert anything or assert the wrong items, the metric of passed unit tests does not give any indication about the quality of your unit tests. Of course, our static code analysis will raise an issue when we do not assert anything in our unit test but it will not alert us for wrong assertions. Besides that, we can take a look at the Line Coverage of our unit tests. This tells us something about how much of our code is covered by our unit tests. But even that can be deceiving. What if we do not assert the correct items? 

What if we change our code and the test still passes? In other words, we need something which will give us an indication of the quality of our tests. That is where mutation testing is for. With mutation testing, faults (or mutants) are introduced in your code and then your tests are run again. If your test fails, the mutant is killed. If your test passes, the mutant is lived. We now have a new metric Mutation Coverage which tells us how to interpret the Line Coverage metric more accurately.

PIT Mutation Testing

How to get this new Mutation Coverage metric? For Java-based applications, we can make use of the PIT Mutation Testing Maven plugin. Traditionally, we use the JaCoCo Maven plugin, but JaCoCo can be replaced with the PIT Mutation Testing plugin because it provides us both metrics we want: the Line Coverage metric and the Mutation Coverage metric. A standard set of mutators is being used, for a complete list, see the PIT website.

In Practice

The proof of the pudding is in the eating. Let’s create a basic Spring Boot application and some unit tests to achieve 100% line coverage. Next, we will run the PIT Mutation Testing Maven plugin and verify whether our unit tests survive the mutants or not. The source code can be found at GitHub.

Basic Spring Boot Application

We are going to create two URLs that will perform a basic operation on a parameter and then return a result. We make use of Spring Web MVC and also add the Spring Boot test dependency. Our pom is the following:

XML
 




x
11


1
<dependencies>
2
    <dependency>
3
        <groupId>org.springframework.boot</groupId>
4
        <artifactId>spring-boot-starter-web</artifactId>
5
    </dependency>
6
    <dependency>
7
        <groupId>org.springframework.boot</groupId>
8
        <artifactId>spring-boot-starter-test</artifactId>
9
        <scope>test</scope>
10
    </dependency>
11
</dependencies>



We add a MutationController. The first method compareToFifty compares whether the input value is greater than 50 or smaller than or equal to 50 and returns a corresponding text message. The second method increment increments the input value with one and returns the incremented value.

Java
 




xxxxxxxxxx
1
22


 
1
@RestController
2
public class MutationController {
3
 
4
    @GetMapping("/compareToFifty/{value}")
5
    public String compareToFifty(@PathVariable int value) {
6
        String message = "Could not determine comparison";
7
        if (value > 50) {
8
            message = "Greater than 50";
9
        } else {
10
            message = "Smaller than or equal to 50";
11
        }
12
 
13
        return message;
14
    }
15
 
16
    @GetMapping("/increment/{value}")
17
    public int increment(@PathVariable int value) {
18
        value++;
19
        return value;
20
    }
21
 
22
}



Run the application:

Shell
 




xxxxxxxxxx
1


 
1
$ mvn spring-boot:run



Verify whether the URL’s are accessible:

Shell
 




xxxxxxxxxx
1


 
1
$ curl http://localhost:8080/compareToFifty/40
2
Smaller than or equal to 50
3
$ curl http://localhost:8080/compareToFifty/60
4
Greater than 50
5
$ curl http://localhost:8080/increment/5
6
6



We add three unit tests for the above:

  • A test smallerThanOrEqualToFiftyMessage to verify the response when the value is smaller than or equal to 50. We deliberately test with value 49 which is a poor boundary test because we should use 50 as a value to test.
  • A test greaterThanFiftyMessage to verify the response when the value is greater than 50.
  • A test increment to verify whether the value is incremented. We deliberately only test whether a successful response is received and we do not test the return value.
Java
 




xxxxxxxxxx
1
25


 
1
@SpringBootTest
2
@AutoConfigureMockMvc
3
public class MutationTest {
4
 
5
    @Autowired
6
    private MockMvc mockMvc;
7
 
8
    @Test
9
    public void smallerThanOrEqualToFiftyMessage() throws Exception {
10
        this.mockMvc.perform(get("/compareToFifty/49")).andDo(print()).andExpect(status().isOk())
11
                .andExpect(content().string("Smaller than or equal to 50"));
12
    }
13
 
14
    @Test
15
    public void greaterThanFiftyMessage() throws Exception {
16
        this.mockMvc.perform(get("/compareToFifty/51")).andDo(print()).andExpect(status().isOk())
17
                .andExpect(content().string("Greater than 50"));
18
    }
19
 
20
    @Test
21
    public void increment() throws Exception {
22
        this.mockMvc.perform(get("/increment/5")).andDo(print()).andExpect(status().isOk());
23
    }
24
 
25
}



Line Coverage

First, we verify what our line coverage is when using the JaCoCo Maven Plugin. Add the plugin to the pom:

XML
 




xxxxxxxxxx
1
24


 
1
<build>
2
    <plugins>
3
        ...
4
        <plugin>
5
            <groupId>org.jacoco</groupId>
6
            <artifactId>jacoco-maven-plugin</artifactId>
7
            <version>0.8.5</version>
8
            <executions>
9
                <execution>
10
                    <goals>
11
                        <goal>prepare-agent</goal>
12
                    </goals>
13
                </execution>
14
                <execution>
15
                    <id>report</id>
16
                    <phase>prepare-package</phase>
17
                    <goals>
18
                       <goal>report</goal>
19
                    </goals>
20
                </execution>
21
            </executions>
22
        </plugin>
23
    </plugins>
24
</build>



Run the build:

Shell
 




xxxxxxxxxx
1


 
1
$ mvn clean install



When finished, navigate to the directory target\site\jacoco\ and open the index.html file which shows the results in your browser.

The report shows us that the MutationController has 100% line coverage. The total line coverage also equals 100% because the line coverage MyMutationTestingPlanetApplication is not applicable (The PIT Mutation report will not list the not applicable results for MyMutationTestingPlanetApplication).
The configuration with JaCoCo is present in the branch feature/jacoco.

Mutation Coverage

Remove the JaCoCo plugin and add the PIT Mutation Test Plugin.

XML
 




xxxxxxxxxx
1
17


 
1
<build>
2
    <plugins>
3
        ....
4
        <plugin>
5
            <groupId>org.pitest</groupId>
6
            <artifactId>pitest-maven</artifactId>
7
            <version>1.5.0</version>
8
            <dependencies>
9
                <dependency>
10
                    <groupId>org.pitest</groupId>
11
                    <artifactId>pitest-junit5-plugin</artifactId>
12
                    <version>0.12</version>
13
                </dependency>
14
            </dependencies>
15
        </plugin>
16
    </plugins>
17
</build>



We also needed to add the dependency pitest-junit5-plugin because we are using JUnit 5. If you do not add this dependency, your unit tests will not be found. This information was not explicitly mentioned in the PIT documentation. It is, however, documented in the Maven quickstart section when the option testPlugin is described: Support for other test frameworks such as junit5 can be added via plugins.

Run the following Maven goal:

Shell
 




xxxxxxxxxx
1


 
1
$ mvn org.pitest:pitest-maven:mutationCoverage



When finished, navigate to the directory target\pit-reports\ and open the index.html file which shows the results in your browser.

Again, we notice a 100% line coverage for our MutationController, but now it also indicates a 40% Mutation Coverage. Click down to the MutationController details, it shows us the following:

The report shows us exactly which mutants survived the test. The tests for compareToFifty survived the boundary test because the greater-than sign (>) has been replaced by the mutant with greater than and equal to (>=). We used 49 as input value in our test which is not a good boundary test. The unit test for the increment method does not assert the returned value. Changes made to the code for returning the value will not cause our unit test to fail.

Fix the Unit Tests

We fix the unit tests in branch feature/solutions.

The solution for the smallerThanOrEqualToFiftyMessage the test is to test the value 50 instead of 49.

Java
 




xxxxxxxxxx
1


1
@Test
2
public void smallerThanOrEqualToFiftyMessage() throws Exception {
3
    this.mockMvc.perform(get("/compareToFifty/50")).andDo(print()).andExpect(status().isOk())
4
            .andExpect(content().string("Smaller than or equal to 50"));
5
}



The solution for the increment test is to check the returned value besides the successful response.

Java
 




xxxxxxxxxx
1


1
@Test
2
public void increment() throws Exception {
3
    this.mockMvc.perform(get("/increment/5")).andDo(print()).andExpect(status().isOk())
4
            .andExpect(content().string("6"));
5
}



If you run the mutation coverage Maven goal again, you will notice that your report has not been changed. There will still be a 40% mutation coverage. PIT analyzes the byte code and is not automatically going to compile your test classes again. Therefore, the report looks the same because the byte code has not been changed. First, run your tests again and then generate the Mutation Coverage report.

The report shows us besides the 100% line coverage, also a 100% mutation coverage.

Conclusion

The PIT Mutation Maven plugin is a great tool for testing the quality of your tests. It is easy to use and can be added instantly to your CI/CD pipeline to generate useful results about the quality of your unit tests.

Topics:
mutation testing

Published at DZone with permission of Gunter Rotsaert , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}