Calculating Java Code Coverage for Non-JVM e2e Tests Suite and More With JCov

1. What Is Dynamic Instrumentation Code Coverage and Why Do We Want It?

Java code coverage tools, like these embedded in IDEs or provided as CI environments plugins, are great, but they have one limitation — the tests you run have to also be written in Java or other JVM language. What if you have suites of tests in other, non-JVM languages and would like to know what is covered and what is not?

I've faced such an issue — we had a really big suite of e2e REST API tests written in Python and executed them against big Java application running on Tomcat. We wanted to track where these tests go in the code. But how to check it?

A possible solution is the so-called dynamic instrumentation of Java code using Java agents. Shortly speaking, a Java agent triggers the moment of class-loading and transforms class bytecode to save some statistics about executed lines. This is magic applied in practice.

According to my research, there are two reasonable solutions providing (among other features) the dynamic instrumentation:

• Jacoco — having an agent mode, recognizable by some people (4090 Stack Overflow matches as of February 2019), not necessarily due to its capabilities for outside-Java-triggered execution, but for its Maven plugins integrated with Jenkins plugins, Sonar plugins, etc. I used it in some of the previous projects to observe code coverage from release to release, so it was my first shot. Disappointingly, for this use case, it didn't work well. More on this in the Why Not Jacoco section
• JCov — having an agent mode, developed as a tool targeted at Java developers (not a typo, I mean people actually developing Java, yep), probably originating in the depths of Oracle basements, and known by a small group of unicorns (9 Stack Overflow posts in three years and one reasonable presentation). You have to check out and build it yourself. I went there, I'm alive, and I'm coming with a practical tutorial.

I used it for generating code coverage reports while running e2e Python tests, but of course, with such a flexible agent, you can track coverage of the code called any way you want. Postman suite? No problem. Clicking around the GUI wondering how the heck it works inside without staring at the debugger? Here you go.

2. Dynamic Vs. Static Instrumentation

As I mentioned before, the instrumentation (the process of altering the bytecode of examined classes) may be dynamic, and this is the flavor I used, but it can also be static. With static instrumentation, you mutate the class files before you run them to gather coverage statistics. In comparison to dynamic instrumentation, it makes execution faster (as there is no overhead on classloading) and consumes much fewer memory resources in general. JCov also supports static instrumentation, I haven't tried it though, as dynamic mode was more suitable for me. You can find more information on static instrumentation mode in the linked JavaOne presentation from JCov developers themselves and also in the indispensable verbose help mode built in the jcov.jar

3. Setup and Operating Instructions for JCov in Dynamic Mode for Tomcat Application

1. Building the Tool Itself

It is not that easy to start with JCov. Actually, you have to build it yourself using manually downloaded dependencies, as the site with released versions doesn't really work. I have built the final jcov.jar for you, and also I provided intermediate dependencies copied to my repository to save you the hassle of looking up the dependencies over the net. But I still recommend to download the sources yourself, as with documentation being scarce going to the sources can show you some hidden functionalities. At least, that was my case.

If you want to prepare it yourself, check out the steps below:

These are official building instructions, although they seem to be a bit 'vintage,' as JCov supports modern Java versions and readme still refers to JDK5. I built it on JDK8.

• Clone jcov mercurial repo: hg clone http://hg.openjdk.java.net/code-tools/jcov

• You will need the following jar dependencies built/downloaded manually to build JCov (I posted the exact versions I used, but maybe others will work, too):

Asm-7.0.jar
Asm-tree-7.0.jar
Asm-util-7.0.jar
jtharness-4_5-dev-bin-b27-19_apr_2013/lib/javatest.jar

I have found them in various places on the web. You can also download them from my GitHub repository.

3. Edit file/build/build.properties, setting paths to the aforementioned jars.

# path to asm libraries
asm.jar = /asm-7.0.jar
asm.tree.jar = /asm-tree-7.0.jar
asm.util.jar = /asm-util-7.0.jar

# path to the javatest library (empty value allowed if you do not need jtobserver.jar)
javatestjar = /javatest.jar

4. Go to /build directory and execute 'ant' command. If you got all of the paths, well this should finally build jcov.jar file in the /JCOV_BUILD/jcov_/directory.

2. Using the JCov Tool

jcov.jar embeds all functions needed to generate coverage report, from instrumenting classes to compiling it in .html (or other formats) report itself.

The JCov tool has many sub-tools and usage scenarios — what's below is a concrete case where dynamic instrumentation is used to generate and dump coverage data on the fly from the application server, which was not pre-instrumented before running.

The phases to generate such a report are as follows:

1. Attach jcov.jar to server startup configuration in Java agent mode, exposing commands server

2. Start the application server

3. Command the agent to dump data to file (to get rid of server startup Java coverage from statistics )

4. Run cases where coverage interests us, by any means on the application server (anyhow, for instance, python nosetests, postman suite, even manual clicking around the GUI or browser automation plugin)

5. Dump the coverage data again to file

6. Generate HTML or another report from coverage data

Now, let's go through these phases in details:

1. Attach jcov.jar to server startup configuration in Java agent mode, exposing commands server.

In the case of the Tomcat server, the configuration went as follows:

Let's explain the params:

• file=important-coverage-data.xml,merge=gensuff — the root of filename where the coverage data will be dumped to. The 'gensuff' merge option makes JCov create a separate file for coverage data every time dumping is requested with a random suffix. There are also other modes of operation, for instance, overwriting existing data in an XML disk file or merging its content. For my case, the separate files with suffixes seemed to be the most practical mode
• Log, log.level — you know what it is
• include=com\.mycompany* — makes the Java agent instrument only classes in selected packages. By default, all classes are instrumented, but beware of that — it may cause excessive memory consumption, even OOM errors for bigger systems. Most probably, you're not interested in coverage data for the external libraries anyway. Note that you can use multiple include and exclude statements to have a more sophisticated configuration (refer to jcov.jar built-in help for more information)
• agent.port=3336 — in this configuration, a simple command accepting the server will be started inside agent, waiting for a signal to dump coverage data to file. Other options are just dumping data to file on VM exit or using so-called 'Grabber' module (read further)

2. Start the server

Start the server however you do it in normal operation — just make sure it picks up the configuration from point 1. In this concrete configuration, you will see that it works if something listens on localhost/3336.

3. Command the agent to dump data to file

In the agent.port configuration, commanding to dump current coverage statistics to XML file turns out to be really crude. Connect to the server using: telnet localhost 3336

Every time you send 'save' string to the server, it will save the next part of coverage data to an XML file and respond with 'saved' string to your telnet session.

Of course, you can automate it, if you need to dump the excessive amounts of data periodically, even in bash, for instance:

Or any other telnet client integrated with your testing environment. This will dump a new portion of data every minute.

Remember: you can merge the results later with Merge command from jcov.jar

4. Run cases where coverage interests you, by any means on the application server (anyhow, for instance, python nosetests, postman suite, even manual clicking around the GUI or browser automation plugin)

5. Dump the coverage data again to file

As in step three, my practice was to dump data just before running the suite and just after, and then using just the second file to calculate coverage report.

6. Generate HTML or another report from coverage data

To generate the report in HTML navigable format, along with the covered lines marked in class sources, execute a command like:
java -jar jcov.jar RepGen -sourcepath /path/to/module/one/src:/path/to/module/two/src /path/to/coverage-data.xml

This will generate, by default, fully navigable HTML report in/report directory below where jcov.jar resides.

4. Troubles I've Encountered and You May Also

Apart from the time I've spent on trivial things, like having to build it with good old Ant or figuring how to signal Agent to dump contents to file, there are some problems that you may stumble upon:

• Out-Of-Memory Errors/ HeapSize — don't be surprised if your application throws OOMs at you or starts to act strangely. This may be GC dying there. Instrumenting the classes and storing execution statistics may use quite a lot of memory. I had to increase my HeapSize a lot to start up a big set of Tomcat applications, even after including only my company classes in the filter. Also, it happened to me that if I haven't applied ANY filter (just took all the classes), sometimes, the JVM instantly crashed at startup. But with filters, everything was alright.
• Multiple source trees — this was covered in the tutorial above but I would like to mention it again. Help for jcov.jar mentions source location — singular. But in fact, you can add multiple directories separated with your operating system separator (you can easily find it in JCov sources, going from the parameter name).
• Spring/CGLib bytecode altering — sometimes, Spring resorts to generating class proxies, which apparently don't just nicely delegate calls to old implementations but dynamically subclass them. Effectively, this means that the original class is lost in favor of a newly generated one. You will see its statistics in the report, but without relation to the original .java source. This is another reason to reconsider using Java class inheritance nowadays!
• Grabber module — in this article, I demonstrated how to use a server embedded in the agent, but in theory, according to JavaOne presentation, the standard way to do it is using so-called Grabber server, to which the Java agent connects. I've successfully set up the Grabber server, but the client didn't seem to work, so I resorted to server embedded in the agent.
• Untouched classes — in this example, only loaded classes are instrumented, so remember that if the class is not covered by tests at all, you won't see it in the report. If you have such need, you have to generate so-called 'template' beforehand.

5. So, What's Next? -hv Is Your Friend

JCov is quite a mysterious tool in terms of community and documentation, but it's really worth noting that it has quite reasonable help built into the jcov.jar itself.

It has many features that go beyond this introduction. For instance, merging existing coverage data files, filtering results, and even generating diffs from build to build.

Start from typing:java -jar jcov.jar

You will get a description of the modules of jcov.jar. You can get verbose info about every part by the  -hv option, which stands for 'help-verbose.' For instance:

It is also worth exploring the source code itself near places where command line parameters are processed.

Lastly, take a detailed look at the JavaOne presentation I mentioned.

6. Why Not Jacoco?

At first, Jacoco seemed to be a better choice, with much higher adoption, bigger community, IDE and CI integrations, but at least, in my case, it basically didn't work. After importing Jacoco coverage data to IntelliJ, it was clear that the lines that were executed were marked as not executed in the report, which beats the purpose. Maybe, it was the faulty plugin. I tried to generate a report outside of the IDE using Jacoco tools themselves, but the report generation crashes when there are two classes of the same name in different packages. And with as big a codebase as mine, the case made this tool useless. I want to stress that, in the past, I've had good experiences with this product while using Maven/Jenkins plugins for tests executed in JVM, so for you, it may be worth giving a shot.

7. Sources

• My repository with JCov.jar and prerequisites built

• JavaOne JCov talk slides

• https://wiki.openjdk.java.net/display/CodeTools/How+To+Build+JCov