Mutation Testing: Covering Your Code With the Right Test Cases (Part 2)

In part one, we achieved 100 percent coverage. In this installment, we mutate our code and check how good our tests are.

There are several mutation testing frameworks. In this case, we will use PITEST.

What mutation might be applied to the code? Here is a list of pitest mutations here:

Let me introduce our mutants (just few of them). First is the line of origin, and the following are mutants (modified code is highlighed by red):

Now, let's execute the pitest Maven plugin and check the mutation testing result:

mvn clean test

mvn org.pitest:pitest-maven:mutationCoverage

Open report located in target/pit-reports/201902032155/index.html

So, the report says:

1. Line coverage 100 percent, but we knew it from the JoCoCo report earlier
2. There are two mutation that survived and 23 were killed.
3. Mutant 1 was created by changing conditional boundary at line 14 and the second by replacing return with true.

Let's clarify what mutants looks like:

So, what are the missing test cases that might eliminate our mutants?

1. For our first mutant, we create a boundary case and assert that the client is created as far as zero passport id is valid
2. For the second mutant, we check that the service doesn't have an unexistent client.

Results give us 100 percent killed mutants, but can we rely on it?

Unfortunately not. Mutation testing gives us just some hints —what cases we might forget (especially for boundary conditions).If we count all possible application set of states, we found out there are more "missing" cases.

Criticism of Mutation Testing

• Time-consuming. Due big number of execution: number of mutations * number of tests
• Creates impression of fully tested app if no mutations been found 
• Don't provide clear metrics of test quality 

Conclusion

• Mutation testing helps us to find possible missing test cases
• Mutation testing can't find all missing test

Lastly, I didn't describe a case when mutation can be equivalent to the initial code, but it does not happen that often.

Code used in this article is available on GitHub.