Mutation Testing: Covering Your Code With the Right Test Cases (Part 1)

Mutation Testing Basics

The concept of mutation testing is to modify code in a small way and verify that tests detect that modification. Undetected modification gives a hint to what test is likely missing.

Mutation Procedure Next Steps:

1. Modify code in small way (creating mutant)
2. Execute existing test suites (killing mutant)
3. Verify that at least one existent test failed (mutant killed)
4. If all test succeded (mutant survived), there is likely a missing test case

But why do we have to consider testing the result of the modified code? Let's take a look at an example:

Initially, the system was fully covered by two test cases. First, two mutations have been detected by failed tests. But what about the third one? The last modification wasn't detected for two reasons:

1. There is a missing test case.
2. Last modification made the equivalent version of the same code

Covering Code by Tests Without Mutations: Registration API Example

To demonstrate the power of mutation testing, we create tests based on simple line coverage (old classical way). To count the line coverage, we use the JoCoCo plugin.

Assume that we have client and registration service with two functions:

1. Register new client
2. Check that client is already registred

The old good way to check your test coverage is to verify that every line was executed during the test suite (at least once). Lets see default test scenario:

Now, it seems that all cases are counted, right? To double check, execute JoCoCo and check code coverage rate using Maven:

mvn clean test

mvn jacoco:report

Let's see generated JoCoCo report located in target/site/jacoco/index.html file

Hmm. It seems that the coverage rate is 100 percent. But can we rely on it?

No. No, we cannot. Stay tuned for the next installment where we explore why!