Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Faster Mutation Testing

DZone's Guide to

Faster Mutation Testing

Is Mutation Testing too slow to be of any real use? Is there a way to speed it up? Or is the argument even valid? Read on to find out more.

· Java Zone
Free Resource

Learn how to troubleshoot and diagnose some of the most common performance issues in Java today. Brought to you in partnership with AppDynamics.

As an ardent promoter of Mutation Testing, I sometimes get comments that it’s too slow to be of real use. This is always very funny as it also applies to Integration Testing, or GUI. Yet, this argument is only used against Mutation Testing, though it cost nothing to setup, as opposed to the former. This will be the subject of another post. In this one, I will provide proposals on how to speed up mutation testing, or more precisely PIT, the Java Mutation Testing reference.

Setting the Bar

A project reference for this article is required. Let’s use the codec submodule of Netty 4.1.

At first, let’s compile the project to measure only PIT-related time. The following should be launched at the root of the Maven project hierarchy:

mvn -pl codec clean test-compile

The -pl option let the command be applied to only specified sub-projects - the codec sub-project in this case. Now just run the tests:

mvn -pl codec surefire:test
...
Results :

Tests run: 340, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.954 s
[INFO] Finished at: 2016-06-13T21:41:10+02:00
[INFO] Final Memory: 12M/309M
[INFO] ------------------------------------------------------------------------

For all it’s worth, 9 seconds will be the baseline. This is not really precise measurement, but good enough for the scope of this article.

Now let’s run PIT:

mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
 org.pitest:pitest-maven:mutationCoverage

Note: PIT flags the above class as failing, even though Surefire has no problem about that

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:14 min
[INFO] Finished at: 2016-06-13T22:02:48+02:00
[INFO] Final Memory: 14M/329M

A whopping 14 minutes! Let’s try to reduce the time taken.

Speed vs. Relevancy Trade-Off

There is no such thing as a free lunch.

PIT offers a lot of tweaks to improve testing speed. However, most of them imply a huge drop in relevancy: faster means less feedback on code quality less than complete. As it’s the ultimate goal of Mutation Testing, it’s IMHO meaningless.

Those configuration options are:

  • Set a limited a set of mutators
  • Limit scope of target classes
  • Limit number of tests
  • Limit dependency distance

All benefits those flags bring are negated by less information.

Running on Multiple Cores

By default, PIT uses a single core even though most personal computers, not to mention servers have many more available.

The easiest way to run faster is to use more cores. If it takes X minutes to run PIT on a single core, and if Y additional cores are 100% available, then it should only take X / (Y + 1) minutes to run on the first and additional cores.

The number of cores is governed by the thread Java system property when launching Maven.

mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
 -Dthreads=4 org.pitest:pitest-maven:mutationCoverage

This yields the following results, which probably means that those additional cores were already performing some tasks during the run.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:30 min
[INFO] Finished at: 2016-06-13T22:25:38+02:00
[INFO] Final Memory: 15M/318M

Still, 50% is good enough for such a small configuration effort.

Incremental Analysis

The rationale between incremental analysis is that unchanged tests running unchanged classes will produce the same outcome as the last one.

Hence, PIT will create a hash for every test and production class. If they are similar, the test won’t run and PIT will reuse the same result. This just requires configuring where to store those hashes.

Let’s run PIT with incremental analysis. The historyInputFile system property is the file where PIT will read hashes, historyOutputFile the file where it will write them. Obviously, they should point to the same file; anyone care to enlighten me as why they can be different? Anyway:

mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
 -DhistoryInputFile=~/.fastermutationtesting -DhistoryOutputFile=~/.fastermutationtesting \
 -Dthreads=4 org.pitest:pitest-maven:mutationCoverage

That produces:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:05 min
[INFO] Finished at: 2016-06-13T23:04:59+02:00
[INFO] Final Memory: 15M/325M

That’s about the time of the previous run… It didn’t run any faster! What could have happened? Well, that’s the first run so that hashes were not created. If PIT is run again a second time with the exact same command-line, the output is now:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.806 s
[INFO] Finished at: 2016-06-13T23:12:43+02:00
[INFO] Final Memory: 20M/575M

That’s way better! Just the time to start up the engine, create the hashes and compare them.

Using SCM

An alternative to the above incremental analysis is to delegate change checks to the VCS. However, that requires the scm section to be adequately configured in the POM and the usage of the scmMutationCoverage goal in place of the mutationCoverage one.

Note this is the reason why Maven should be launched at the root of the project hierarchy, to benefit from the SCM configuration in the root POM.

mvn -pl codec -DexcludedClasses=io.netty.handler.codec.compression.LzfDecoderTest \
 -Dthreads=4 org.pitest:pitest-maven:scmMutationCoverage

The output is the following:

[INFO] No locally modified files found - nothing to mutation test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.587 s
[INFO] Finished at: 2016-06-14T22:11:29+02:00
[INFO] Final Memory: 14M/211M

This is even faster than incremental analysis, probably since there’s no hash comparison involved.

Changing files without committing will correctly send the related mutants to be tested. Hence, this goal should be used only be developers on their local repository to check their changes before commit.

Conclusion

Now that an authoritative figure has positively written about Mutation Testing, more and more people will be willing to use it. In that case, the right configuration might make a difference between wide adoption and complete rejection.

Understand the needs and benefits around implementing the right monitoring solution for a growing containerized market. Brought to you in partnership with AppDynamics.

Topics:
reference ,maven ,configuration ,testing ,mutation testing ,quality assurance ,java ,integration testing

Published at DZone with permission of Nicolas Frankel, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}