Mutation Testing With SonarQube

In a previous post, we explored the PIT Mutation Testing Maven plugin. This time, we will take a look at how we can integrate the results with SonarQube, our favorite software analysis tool.

Introduction

Before reading this post, it is advised to revise our previous post about Mutation Testing. In short, with Mutation Testing faults (or mutants) are introduced into your code and consequently, your tests are run again. When tests fail, the mutants are killed. When tests survive, the mutants live. 

This will provide us a new metric, the Mutation Coverage, which will give us insight into the quality of our tests. Since we are talking about quality here, it is a good idea to check how we can integrate this in a software analysis tool like SonarQube.

In the following paragraphs, we will set up a local instance of SonarQube and take the necessary steps for enabling the Mutation Test results into SonarQube. The sources can be found at GitHub in branches feature/master-sonar-analysis and feature/solutions-sonar-analysis.

Installation of SonarQube

We will go for quick local installation of SonarQube. The instructions below are intended when you want to use SonarQube for evaluation purposes. It is not intended to run SonarQube in a production environment with this setup. E.g. we will be using an embedded H2 database which will not be migrated during upgrades.

Our starting point is the Docker project for SonarQube. We want to get started quickly, so we will go for the Get Started in Two Minutes Guide. Prerequisite for this setup is the installation of Docker. We will be using the SonarQube Community Edition.

Open your terminal and start the Docker image:

$ docker run -d --name sonarqube -p 9000:9000 sonarqube:lts

After downloading and starting the Docker image, navigate to the browser to URL http://localhost:9000. Log in with the default System Administrator credentials (login=admin, password=admin).

That’s about it, we are all set and ready to go.

Configuration of SonarQube

In this section, we will configure SonarQube in order to be able to view the Mutation Testing results.

Installation of Plugin

We need to install the Mutation Analysis SonarQube plugin in order for SonarQube to be able to interpret the Mutation Analysis results.

Navigate to Administration – Marketplace.

Search in the search box for pit and install the plugin.

After a while, the status of the plugin will show Install Pending and on top of the screen, a message is shown to restart the server. After restarting the server, the plugin is successfully installed.

Local Configuration

We will run our Maven build from our local machine. It is necessary to configure Maven in such a way that it knows where to publish the Sonar results to. Therefore, add the following to your local settings.xml file (by default, the settings.xml file is located in your home directory in the .m2 directory):

xxxxxxxxxx

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0

                          https://maven.apache.org/xsd/settings-1.0.0.xsd">

      <localRepository/>

      <interactiveMode/>

      <offline/>

      <pluginGroups>

        <pluginGroup>org.sonarsource.scanner.maven</pluginGroup>

      </pluginGroups>

      <servers/>

      <mirrors/>

      <proxies/>

      <profiles>

        <profile>

            <id>sonar</id>

            <activation>

                <activeByDefault>true</activeByDefault>

            </activation>

            <properties>

                <sonar.host.url>

                  http://localhost:9000

                </sonar.host.url>

            </properties>

        </profile>

      </profiles>

      <activeProfiles/>

</settings>

Check out the feature/master-sonar-analysis branch and run the Maven build with Sonar analysis:

xxxxxxxxxx

$ mvn clean verify sonar:sonar

The Sonar results are visible after the build has finished.

As can be seen, there is a 0.0% Coverage. This means that we are not yet finished configuring.

Enable the Mutation Analysis Rules

Go to Quality Profiles in SonarQube. There are two profiles available in the Java section. One profile Sonar Way, which is the default profile and one profile Mutation Analysis. In the Used column, we notice that profile Sonar way has been used in our initial analysis.

We do not want to set the Mutation Analysis profile as default, because we would lose all the Java rules from the Sonar way profile and that is not what we want. We want to add the Mutation Analysis rules to be added to the Sonar way profile.

Click the Sonar way profile and open the Settings menu.

Choose Extend and give the extended profile a new name. It is not possible to change the default Sonar profiles. When you want to change something to the default profiles, you will need to create a new profile. Extending an existing profile is a good practice for this purpose.

After creating the new profile, we want to enable the Mutation Analysis rules in this new profile. We, therefore, click the Activate More button (The screen below can be shown via Quality Profiles – Java and then select the Sonar + Mutation profile)

Click the MutationAnalysis repository and the Ready status. We do not want to include Beta or Deprecated rules.

The search results screen shows us the selected rules. At the top of the screen, click the Bulk Change button and choose Activate in Sonar + Mutation.

Navigate again to Quality Profiles, go to the Java section and set the Sonar + Mutation profile as default.

The last thing to do is to change the output report format in our Maven build from the default html to xml. SonarQube needs the report in XML format.

xxxxxxxxxx

<plugin>

    <groupId>org.pitest</groupId>

    <artifactId>pitest-maven</artifactId>

    <version>1.5.0</version>

    <configuration>

        <outputFormats>

            <outputFormat>xml</outputFormat>

        </outputFormats>

    </configuration>

    <dependencies>

        <dependency>

             <groupId>org.pitest</groupId>

             <artifactId>pitest-junit5-plugin</artifactId>

             <version>0.12</version>

        </dependency>

    </dependencies>

</plugin>

Run the Maven build including Mutation Analysis and Sonar Analysis.

xxxxxxxxxx

$ mvn clean verify org.pitest:pitest-maven:mutationCoverage sonar:sonar

Go to SonarQube and view the results. Our Coverage metric shows 50.0% which is already a better result than the earlier 0.0%.

Click the Sonar project MyMutationTestingPlanet and select the Measures tab:

We can see a Mutation Analysis section in the Measures. When we click the Mutation: Coverage, we can drill down to the MutationController. The results of the Mutation Analysis are now available in SonarQube.

Solve Mutation Issues

Let’s solve the mutation issues in our unit tests. We do so in branch feature/solutions-sonar-analysis. In order to create a new Sonar analysis project, we will run the Sonar analysis with a different Project Name and Project Key. 

The Developer Edition of SonarQube provides support for branches, but this is out of scope for this blog. You can read more about it in the official SonarQube documentation.

xxxxxxxxxx

$ mvn clean verify org.pitest:pitest-maven:mutationCoverage sonar:sonar -Dsonar.projectName="MutationTesting-solutions" -Dsonar.projectKey="feature/solutions-sonar-analysis"

A new Sonar analysis project has been created with identical results after a successful build.

We solve the Mutation issues just like we did in our previous post. Run the Sonar analysis again. As expected, our Mutation Coverage reaches 100%.

Drilling down to the MutationController, shows us that all the Mutation issues have disappeared.

Unresolved Warning

During the Mutation Analysis, we encountered the following warning in our build log:

[WARNING] /!\ At least one Mutation Analysis rule needs to be activated the current profile.

/!\ At least one Mutation Analysis rule needs to be activated the current profile.

This is strange because our Sonar profile contains active Mutation Analysis rules. We have not discovered why this warning occurs. If you have a clue, please let us know.

Conclusion

In this post, we looked at how we can integrate the Mutation Analysis with SonarQube. The integration with SonarQube can be achieved with little effort when you know how to configure SonarQube.