Reporting Code Coverage Using Maven and JaCoCo Plugin

It’s known that having automated tests as part of your build process improves the software quality and reduces the number of bugs.

Do you know if you need more unit tests? Or if your tests cover all possible branches of an if or switch statements? Or if your code coverage is decreasing over time? Especially after you join a team to work on an on-going project.

Code coverage helps to answer these questions. This post covers reporting code coverage using Maven’s jacoco-maven-plugin, a library that adds minimal overhead with a normal build.

Requirements

• Java 7+
• Maven 3.2+
• Overview of my previous post, Splitting Unit and Integration Tests using Maven and Surefire plugin because this post uses the same source code.

Sample Application

The example application has two unit test classes, DemoControllerTest andDefaultSomeBusinessServiceTest, and two integration tests classes, DemoControllerIT, ApplicationTests; this is similar to those discussed in Splitting Unit and Integration Tests Using Maven and Surefire plugin section.

Configuring jacoco-maven-plugin and Coverage Threshold

Let’s configure jacoco-maven-plugin in pom.xml:

...
<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version>0.8.3</version>
  <executions>
    <execution>
      <id>coverage-initialize</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
    </execution>
    <execution>
      <id>coverage-report</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>report</goal>
      </goals>
    </execution>
    <!-- Threshold -->
    <execution>
      <id>coverage-check</id>
      <goals>
        <goal>check</goal>
      </goals>
      <configuration>
        <rules>
          <rule>
            <element>CLASS</element>
            <excludes>
              <exclude>com.asimio.demo.Application</exclude>
            </excludes>
            <limits>
              <limit>
                <counter>LINE</counter>
                <value>COVEREDRATIO</value>
                <minimum>80%</minimum>
              </limit>
            </limits>
          </rule>
        </rules>
      </configuration>
    </execution>
  </executions>
</plugin>
...

The prepare-agent goal sets up the property argLine (for most packaging types), pointing to the JaCoCo runtime agent. You can also pass argLine as a VM argument. Maven-surefire-plugin uses argLine to set the JVM options to run the tests.

If you are explicitly setting argLine, make sure it allows late replacements like:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <argLine>@{argLine} -more -arguments</argLine>
...
  </configuration>
</plugin>

This is so that the maven-surefire-plugin picks up changes made by other Maven plugins such as jacoco-maven-plugin.

The JaCoCo Java agent will collect coverage information when maven-surefire-plugin runs the tests. It will write it to destFile property value, if set, or target/jacoco.exec by default. Read more at https://www.eclemma.org/jacoco/trunk/doc/prepare-agent-mojo.html.

The report goal creates code coverage reports for tests in HTML, XML, CSV formats. Stay tuned, I’ll cover uploading code coverage reports to SonarQube in another post. This goal reads the dataFile property value, if set, or target/jacoco.exec. And, it writes the resulting reports to outputDirectory property value or target/site/jacoco. Read more at https://www.eclemma.org/jacoco/trunk/doc/report-mojo.html.

The check goal validates that the coverage rules (discussed later) are met. In case they are not, it interrupts and fails the build unless the haltOnFailure property is set to false. Read more at https://www.eclemma.org/jacoco/trunk/doc/check-mojo.html.

Running the Tests and Creating the Coverage Reports

Let’s build the application and analyze the Maven command output:

mvn clean verify
...
[INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (coverage-initialize) @ unit-integration-tests-jacoco-coverage ---
[INFO] argLine set to -javaagent:/Users/ootero/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/Users/ootero/Projects/bitbucket.org/unit-integration-tests-jacoco-coverage/target/jacoco.exec
...
[INFO] --- maven-surefire-plugin:2.22.1:test (unit-tests) @ unit-integration-tests-jacoco-coverage ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
...
[INFO] --- maven-surefire-plugin:2.22.1:test (integration-tests) @ springboot2-split-unit-integration-tests ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
...
[INFO] --- jacoco-maven-plugin:0.8.3:report (coverage-report) @ unit-integration-tests-jacoco-coverage ---
[INFO] Loading execution data file /Users/ootero/Projects/bitbucket.org/unit-integration-tests-jacoco-coverage/target/jacoco.exec
[INFO] Analyzed bundle 'unit-integration-tests-jacoco-coverage' with 3 classes
...
[INFO] --- jacoco-maven-plugin:0.8.3:check (coverage-check) @ unit-integration-tests-jacoco-coverage ---
[INFO] Loading execution data file /Users/ootero/Projects/bitbucket.org/unit-integration-tests-jacoco-coverage/target/jacoco.exec
[INFO] Analyzed bundle 'unit-integration-tests-jacoco-coverage' with 3 classes
[INFO] All coverage checks have been met.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...

Right after the clean phase completes, jacoco-maven-plugin’s prepare-agent goal (bound to the Maven’s Build Default Lifecycle’s initialize phase) sets the argLine property pointing to the JaCoCo Java agent.

Unit and Integration tests ran separately as covered in a previous post.

Next, not included in this log output, the Maven artifact is built and repackaged.

After that, jacoco-maven-plugin’s coverage-report goal (bound to the Maven’s Build Default Lifecycle’s post-integration-test phase) generates HTML, XML and CSV reports.

$ ls -1a target/site/jacoco/
.
..
com.asimio.demo
com.asimio.demo.rest
com.asimio.demo.service
index.html
jacoco.csv
jacoco.xml
jacoco-resources
jacoco-sessions.html

Opening the HTML report at target/site/jacoco/index.html results in:

Code coverage report for a successful build

Lastly, jacoco-maven-plugin’s check goal (bound to the Maven’s Build Default Lifecycle’s verify phase) checks the code coverage metrics are met.

JaCoCo Rules

Let’s take a closer look at the jacoco-maven-plugin’s coverage-checkrules configuration in pom.xml:

<rules>
  <rule>
    <element>CLASS</element>
    <excludes>
      <exclude>com.asimio.demo.Application</exclude>
    </excludes>
    <limits>
      <limit>
        <counter>LINE</counter>
        <value>COVEREDRATIO</value>
        <minimum>80%</minimum>
      </limit>
    </limits>
  </rule>
</rules>

Setting the rule element to CLASS means every Java class from the application would need to meet each counter limit for the build to pass.

In this example, there is only one limit, a LINE counter that needs coverage of at least 80 percent. I’ll cover JaCoCo Counters later.

Other JaCoCo rules you could use are:

Notice that as you move down in the JaCoCo rules table, the check goal becomes more constraining.

As an example, if you remove com.asimio.demo.Application from the excludes sections, the build fails because the LINE counter doesn’t reach 80 percent for said class:

-    <excludes>
-      <exclude>com.asimio.demo.Application</exclude>
-    </excludes>

mvn clean verify
...
[WARNING] Rule violated for class com.asimio.demo.Application: lines covered ratio is 0.33, but expected minimum is 0.80
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...

JaCoCo Counters

Even though I only used the LINE counter when covering JaCoCo Rules, there are a handful of other counters you could include in the limits set in the jacoco-maven-plugin configuration.

Notice that as you move down in the JaCoCo counters table, the check goal becomes less constraining.

Examples of associating a counter to a rule are:

<limit>
  <counter>LINE</counter>
  <value>COVEREDRATIO</value>
  <minimum>80%</minimum>
</limit>
<limit>
  <counter>CLASS</counter>
  <value>MISSEDCOUNT</value>
  <maximum>0</maximum>
</limit>

A counter value is one of:

and a numeric minimum or maximum.

Conclusion

Although not a silver bullet, code coverage helps to measure what percentage of code is executed when running the test suites. And thus, it helps to reduce the number of bugs and improve the software release quality.

Keeping a certain threshold might get difficult over time as a development team adds edge cases or implement defensive programming.

JaCoCo adds minimal overhead to the build process. Jacoco-maven-plugin’s prepare-agentgoal, bound to the initialize phase, sets the agent responsible for instrumenting the Java code before maven-surefire-plugin runs. Coverage-report goal is bound to the post-integration-test phase. And coverage-report goal is bound to the verify phase. Read more at Maven’s Build Default Lifecycle. This means, unlike other libraries, JaCoCo doesn’t need to run the tests twice.

Thanks for reading and sharing. If you found this post helpful and would like to receive updates when content like this gets published, sign up to the newsletter.

Source Code

Accompanying source code for this blog post can be found on BitBucket.

References

• https://www.eclemma.org/jacoco/trunk/doc/maven.html
• https://www.eclemma.org/jacoco/trunk/doc/prepare-agent-mojo.html
• https://www.eclemma.org/jacoco/trunk/doc/report-mojo.html
• https://www.eclemma.org/jacoco/trunk/doc/check-mojo.html
• https://www.eclemma.org/jacoco/trunk/doc/counters.html
• http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html