{{announcement.body}}
{{announcement.title}}

Mutation Testing With SonarQube

DZone 's Guide to

Mutation Testing With SonarQube

We will take a look at how we can integrate the results with SonarQube, my favorite software analysis tool.

· Performance Zone ·
Free Resource

In a previous post, we explored the PIT Mutation Testing Maven plugin. This time, we will take a look at how we can integrate the results with SonarQube, our favorite software analysis tool.

Introduction

Before reading this post, it is advised to revise our previous post about Mutation Testing. In short, with Mutation Testing faults (or mutants) are introduced into your code and consequently, your tests are run again. When tests fail, the mutants are killed. When tests survive, the mutants live

This will provide us a new metric, the Mutation Coverage, which will give us insight into the quality of our tests. Since we are talking about quality here, it is a good idea to check how we can integrate this in a software analysis tool like SonarQube.

In the following paragraphs, we will set up a local instance of SonarQube and take the necessary steps for enabling the Mutation Test results into SonarQube. The sources can be found at GitHub in branches feature/master-sonar-analysis and feature/solutions-sonar-analysis.

Installation of SonarQube

We will go for quick local installation of SonarQube. The instructions below are intended when you want to use SonarQube for evaluation purposes. It is not intended to run SonarQube in a production environment with this setup. E.g. we will be using an embedded H2 database which will not be migrated during upgrades.

Our starting point is the Docker project for SonarQube. We want to get started quickly, so we will go for the Get Started in Two Minutes Guide. Prerequisite for this setup is the installation of Docker. We will be using the SonarQube Community Edition.

Open your terminal and start the Docker image:

Shell
 




x


1
$ docker run -d --name sonarqube -p 9000:9000 sonarqube:lts



After downloading and starting the Docker image, navigate to the browser to URL http://localhost:9000. Log in with the default System Administrator credentials (login=admin, password=admin).

That’s about it, we are all set and ready to go.

Configuration of SonarQube

In this section, we will configure SonarQube in order to be able to view the Mutation Testing results.

Installation of Plugin

We need to install the Mutation Analysis SonarQube plugin in order for SonarQube to be able to interpret the Mutation Analysis results.

Navigate to Administration – Marketplace.

administration

Search in the search box for pit and install the plugin.

plugins

After a while, the status of the plugin will show Install Pending and on top of the screen, a message is shown to restart the server. After restarting the server, the plugin is successfully installed.

Local Configuration

We will run our Maven build from our local machine. It is necessary to configure Maven in such a way that it knows where to publish the Sonar results to. Therefore, add the following to your local settings.xml file (by default, the settings.xml file is located in your home directory in the .m2 directory):

XML
 




xxxxxxxxxx
1
28


 
1
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
2
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
4
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
5
      <localRepository/>
6
      <interactiveMode/>
7
      <offline/>
8
      <pluginGroups>
9
        <pluginGroup>org.sonarsource.scanner.maven</pluginGroup>
10
      </pluginGroups>
11
      <servers/>
12
      <mirrors/>
13
      <proxies/>
14
      <profiles>
15
        <profile>
16
            <id>sonar</id>
17
            <activation>
18
                <activeByDefault>true</activeByDefault>
19
            </activation>
20
            <properties>
21
                <sonar.host.url>
22
                  http://localhost:9000
23
                </sonar.host.url>
24
            </properties>
25
        </profile>
26
      </profiles>
27
      <activeProfiles/>
28
</settings>



Check out the feature/master-sonar-analysis branch and run the Maven build with Sonar analysis:

Shell
 




xxxxxxxxxx
1


1
$ mvn clean verify sonar:sonar



The Sonar results are visible after the build has finished.

MyMutationTestingPlanet


As can be seen, there is a 0.0% Coverage. This means that we are not yet finished configuring.

Enable the Mutation Analysis Rules

Go to Quality Profiles in SonarQube. There are two profiles available in the Java section. One profile Sonar Way, which is the default profile and one profile Mutation Analysis. In the Used column, we notice that profile Sonar way has been used in our initial analysis.

Java, 2 profiles

We do not want to set the Mutation Analysis profile as default, because we would lose all the Java rules from the Sonar way profile and that is not what we want. We want to add the Mutation Analysis rules to be added to the Sonar way profile.

Click the Sonar way profile and open the Settings menu.

changelog

Choose Extend and give the extended profile a new name. It is not possible to change the default Sonar profiles. When you want to change something to the default profiles, you will need to create a new profile. Extending an existing profile is a good practice for this purpose.

extend profile

After creating the new profile, we want to enable the Mutation Analysis rules in this new profile. We, therefore, click the Activate More button (The screen below can be shown via Quality Profiles – Java and then select the Sonar + Mutation profile)

sonar + mutation

Click the MutationAnalysis repository and the Ready status. We do not want to include Beta or Deprecated rules.

filters

The search results screen shows us the selected rules. At the top of the screen, click the Bulk Change button and choose Activate in Sonar + Mutation.

bulk change

Navigate again to Quality Profiles, go to the Java section and set the Sonar + Mutation profile as default.

Java 3 profiles

The last thing to do is to change the output report format in our Maven build from the default html to xml. SonarQube needs the report in XML format.

XML
 




xxxxxxxxxx
1
17


1
<plugin>
2
    <groupId>org.pitest</groupId>
3
    <artifactId>pitest-maven</artifactId>
4
    <version>1.5.0</version>
5
    <configuration>
6
        <outputFormats>
7
            <outputFormat>xml</outputFormat>
8
        </outputFormats>
9
    </configuration>
10
    <dependencies>
11
        <dependency>
12
             <groupId>org.pitest</groupId>
13
             <artifactId>pitest-junit5-plugin</artifactId>
14
             <version>0.12</version>
15
        </dependency>
16
    </dependencies>
17
</plugin>



Run the Maven build including Mutation Analysis and Sonar Analysis.

Shell
 




xxxxxxxxxx
1


1
$ mvn clean verify org.pitest:pitest-maven:mutationCoverage sonar:sonar



Go to SonarQube and view the results. Our Coverage metric shows 50.0% which is already a better result than the earlier 0.0%.

MyMutationTestingPlanet

Click the Sonar project MyMutationTestingPlanet and select the Measures tab:

MyMutationTestingPlanet

We can see a Mutation Analysis section in the Measures. When we click the Mutation: Coverage, we can drill down to the MutationController. The results of the Mutation Analysis are now available in SonarQube.

SonarQube

SonarQube

Solve Mutation Issues

Let’s solve the mutation issues in our unit tests. We do so in branch feature/solutions-sonar-analysis. In order to create a new Sonar analysis project, we will run the Sonar analysis with a different Project Name and Project Key. 

The Developer Edition of SonarQube provides support for branches, but this is out of scope for this blog. You can read more about it in the official SonarQube documentation.

Shell
 




xxxxxxxxxx
1


 
1
$ mvn clean verify org.pitest:pitest-maven:mutationCoverage sonar:sonar -Dsonar.projectName="MutationTesting-solutions" -Dsonar.projectKey="feature/solutions-sonar-analysis"



A new Sonar analysis project has been created with identical results after a successful build.

MutationTesting-solutions

We solve the Mutation issues just like we did in our previous post. Run the Sonar analysis again. As expected, our Mutation Coverage reaches 100%.

Mutation analysis

Drilling down to the MutationController, shows us that all the Mutation issues have disappeared.

MutationController

Unresolved Warning

During the Mutation Analysis, we encountered the following warning in our build log:

Shell
 







This is strange because our Sonar profile contains active Mutation Analysis rules. We have not discovered why this warning occurs. If you have a clue, please let us know.

Conclusion

In this post, we looked at how we can integrate the Mutation Analysis with SonarQube. The integration with SonarQube can be achieved with little effort when you know how to configure SonarQube.

Topics:
docker ,mutation testing ,performance ,sonarqube ,tutorial

Published at DZone with permission of Gunter Rotsaert , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}