Verifying End-to-End Test Code Coverage Using Jacoco Agent

Fact: end-to-end tests are critical if you want to make sure your software works as it should. To be 100% sure that you covered every (or almost every) possible branch of your business code, it is worth it to check what code has been invoked after your E2E suite finished successfully. The solution I am going to present may be much easier if you are able to stand up your application locally. Sometimes, though, it is not the case.

In this scenario, we are going to start two instances of a web application at the remote servers, with code invocation recording turned on. Next, we are going to run integration tests pointing to them one after another with different parameters, to test different paths. After the tests are done, we are going to pull the data from these servers to localhost, merge it and transform it to HTML report.

We are going to test this simple REST controller:

@RestController
public class ExampleController {

    @GetMapping(path = "/test")
    public int exampleMethod(@RequestParam int parameter) {
        if(parameter > 10) {
            return invokeThisBranch(parameter);
        } else {
            return invokeAnotherBranch(parameter);
        }
    }

    private int  invokeThisBranch(int parameter) {
        System.out.println("Taking this branch..");
        if(parameter < 10) {
            System.out.println("This cannot be tested..");
        }
        return 3;
    }

    private int invokeAnotherBranch(int parameter) {
        System.out.println("Taking another branch..");
        return 3;
    }
}

To make things simple, we will run two instances of the same application on our local machines (on different ports). Before we do that, we need to download Jacoco agent from this link. Remember that the version must match the version of your Jacoco plugin in Maven. This agent will be attached to JVM and record the code coverage. The problem is that you need to push it to your remote server manually (if you find a way to automate this clean way, let me know). If you are using IntelliJ, add this to VM options in your Run Configuration (or add this to startup script on a remote server):

-javaagent:<path-to-agent>/jacocoagent.jar=port=36320,destfile=jacoco-it.exec,output=tcpserver

By specifying the output as a tcpserver and providing a port, you are enabling further connection from the Maven plugin, which will download a data file (jacoco-it.exec).

Our test suite contains one test, which will cause different methods to be invoked depending on the parameter value.

@RunWith(SpringRunner.class)
public class TestCoverageExampleApplicationTests {

    private TestRestTemplate testRestTemplate = new TestRestTemplate();

    @Test
    public void anotherBranchTest() {
        //given
        int appPort =  Integer.parseInt(System.getProperty("app.port"));
        int parameter = Integer.parseInt(System.getProperty("parameter"));

        //when
        Integer result = testRestTemplate.getForObject(
                "http://localhost:" + appPort + "/test?parameter=" + parameter, Integer.class);

        //then
        assertThat(result).isEqualTo(3);
    }
}

We want to make sure that even though the code coverage was spread among instances, we will merge it correctly to be able to see the whole picture. Let’s run the test - different paths for different instance:

mvn test -Dapp.port=8080 -Dparameter=5
mvn test -Dapp.port=8081 -Dparameter=15

Before we generate a coverage report, we need to configure Jacoco plugins in the pom.xml file. The full version can be found on my GitHub account:

<plugins>
    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>${jacoco.version}</version>
        <executions>
            <execution>
                <id>pull-test-data</id>
                <phase>post-integration-test</phase>
                <goals>
                    <goal>dump</goal>
                </goals>
                <configuration>
                    <destFile>${project.build.directory}/jacoco-it-${app.host}:${app.port}.exec</destFile>
                    <address>${app.host}</address>
                    <port>${app.port}</port>
                    <reset>false</reset>
                    <skip>${skip.dump}</skip>
                </configuration>
            </execution>
            <execution>
                <id>merge-test-data</id>
                <goals>
                    <goal>merge</goal>
                </goals>
                <configuration>
                    <destFile>target/jacoco-it.exec</destFile>
                    <skip>${skip.dump}</skip>
                    <fileSets>
                        <fileSet implementation="org.apache.maven.shared.model.fileset.FileSet">
                            <directory>target</directory>
                            <includes>
                                <include>*it*.exec</include>
                            </includes>
                        </fileSet>
                    </fileSets>
                </configuration>
            </execution>
        </executions>
        <configuration>
            <append>true</append>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>${antrun-plugin.version}</version>
        <executions>
            <execution>
                <id>generate-report</id>
                <phase>post-integration-test</phase>
                <goals>
                    <goal>run</goal>
                </goals>
                <configuration>
                    <skip>${skip.int.tests.report}</skip>
                    <target>
                        <taskdef name="report" classname="org.jacoco.ant.ReportTask">
                            <classpath path="${basedir}/target/jacoco-jars/org.jacoco.ant.jar"/>
                        </taskdef>
                        <mkdir dir="${basedir}/target/coverage-report"/>
                        <report>
                            <executiondata>
                                <fileset dir="${basedir}">
                                    <include name="target/jacoco-it*.exec"/>
                                </fileset>
                            </executiondata>
                            <structure name="jacoco-multi Coverage Project">
                                <group name="jacoco-multi">
                                    <classfiles>
                                        <fileset dir="target/classes"/>
                                    </classfiles>
                                    <sourcefiles encoding="UTF-8">
                                        <fileset dir="src/main/java"/>
                                    </sourcefiles>
                                </group>
                            </structure>
                            <html destdir="${basedir}/target/coverage-report/html"/>
                            <xml destfile="${basedir}/target/coverage-report/coverage-report.xml"/>
                            <csv destfile="${basedir}/target/coverage-report/coverage-report.csv"/>
                        </report>
                    </target>
                </configuration>
            </execution>
        </executions>
        <dependencies>
            <dependency>
                <groupId>org.jacoco</groupId>
                <artifactId>org.jacoco.ant</artifactId>
                <version>${jacoco.ant.version}</version>
            </dependency>
        </dependencies>
    </plugin>
</plugins>

Let’s create an automated script for coverage report generation:

mvn jacoco:dump@pull-test-data -Dapp.host=localhost -Dapp.port=36320 -Dskip.dump=false
mvn jacoco:dump@pull-test-data -Dapp.host=localhost -Dapp.port=36321 -Dskip.dump=false
mvn jacoco:merge@merge-test-data -Dskip.dump=false
mvn antrun:run@generate-report -Dskip.int.tests.report=false

The script will pull the data files from two remote servers (localhost, in our case), merge them into one file, and then generate a report from it.

The final report is a joy to analyze:

Our class:

As you can see, even though different code was invoked in different instances, the final report contains both paths.

Some real-world problems you may encounter:

• If you use bytecode manipulators like Lombok or AspectJ, Jacoco won’t be able to find a source code that matches the invoked lines; you can use auto-value or immutables instead of Lombok for some use cases and spring-aop instead of AspectJ.
• If you write your tests in Spock and you want to upload your jacoco-it.exec files to Sonar to show the code coverage there, you have to make sure Groovy’s expressive method names will be correctly transformed in the failsafe report - you need to add org.sonar.java.jacoco.JUnitListener as a listener.

The full code can be found on my GitHub account.