Thanks to the variety of language tools available these days, it is not hard to measure and track code coverage of a JS application. My go-to solution involves Istanbul as the coverage tool, and either Karma or Venus.js as the test runner, and this setup works with various popular unit test libraries out there. If you are new to this, I recommend checking out my past blog posts on this subject:
- Karma and Istanbul with Mocha
- Karma and Istanbul with Jasmine
- Karma and Istanbul with QUnit
- Istanbul and Venus.js (Mocha, Jasmine, QUnit)
And yet, the work does not stop there. Wouldn't it be fantastic if the code coverage report became another conduit of feedback for a contributor? Is it possible to track down every single pull request, then check if the changes associated with that pull request would diminish the coverage?
The answer is yes. The key to this process is in utilizing a hosted code coverage service. In this post I will cover (pun intended) my current favorite, Codecov.io, although there are many out there.
Thanks to a set of rich features, integrating Codecov.io into your open-source project is very easy. For a start, you do not need to create a dedicated account as you can just authenticate using Github. Furthermore, Codecov.io has a built-in support for Github (as well as other hosted Git such as Bitbucket), choosing a project to be added to your dashboard is trivial.
Keep in mind that Codecov.io displays the coverage information of your project. Your build process still needs to produce that coverage information. Also, I assume that you have a continuous integration system, which runs the build process every time there is a new check-in or when someone has a feature branch in a pull request. For many FLOSS project, Travis CI is the most common solution, although there are a few other hosted CI services out there.
To start using Codecov.io, first we need to enable the coverage information in Cobertura format. I have played with different coverage formats and I discovered that Cobertura is the most suitable (your mileage may vary and things can change from time to time). If you use Istanbul directly, you can use its
report command to generate the coverage information in the right format (refer to the documentation for more details). With our setup, I modified a section in the Karma configuration file,
This ensures that Karma tells Istanbul to produce another coverage information, in addition to the default lcov, in the format that we want, Cobertura. You can test this, simply execute
npm test and after a while, you will spot the file
coverage/cobertura/cobertura-coverage.xml that contains the coverage information. This is what we need to send to Codecov.io. There are multiple ways to do that, the easiest is to use codecov.io package. You can use this package by running:
In this example,
package.json is modified to look like this:
Thus, everytime you invoke
npm run ci on your Travis CI job, the tests will be executed and the coverage information will be sent to Codecov.io.
To setup the dashboard, login to Codecov.io and add the repository as a new project. Codecov.io maintains a nice mapping of project URL. For example, the coverage dashboard for this example repo github.com/ariya/coverage-mocha-istanbul-karma is codecov.io/github/ariya/coverage-mocha-istanbul-karma. The next time you kick a build on the project, the dashboard will display the coverage information as sent from the build process.
If that works flawlessly, now you want to enable its pull request integration. Go to the project page and choose Integration and Setup, Pull Request Comment. Now you can determine various ways Codecov.io will comment on every pull request. For a start, you may want to enable Header and Compare Diff.
In the example repo, I have created a pull request, github.com/ariya/coverage-mocha-istanbul-karma/pull/3, that demonstrated a coverage regression. In that pull request, there is a commit that aims to optimize the code but that optimization does not include an additional unit test. This triggers the following response from Codecov.io, a feedback that is rather obvious:
With the build process that produces the coverage information, combined with a service such as Codecov.io, it is easy to keep untested code away from your project!