Code Coverage On GitHub PRs With Coveralls.io

With Coveralls.IO it’s possible to get code coverage on all pull requests submitted. For Kentor.AuthServices I’ve set up AppVeyor builds that uses Coveralls.Net to upload coverage numbers to Coveralls.IO.

Why Coveralls.IO?

When I set up the code coverage I already had AppVeyor builds running. I wanted something that worked together with AppVeyor. I tried out a few options and quickly found out that I also wanted something that used the Visual Studio Code Coverage engine. The project had 100.00% coverage in Visual Studio when I started and when I tried another coverage engine it showed some lines as skipped. I wanted consistency with the Visual Studio Code Coverage, which coveralls could give me.

AppVeyor, Pull Requests and Secrets

AppVeyor can be configured in two ways: Through an appveyor.yml file in the repository or through a web interface. There is also a mechanism for handling secure settings such as API keys through environment variables. That way secrets can be protected in the AppVeyor settings page, while the general build settings are public in the appveyor.yml file. There is one caveat though: Those environment variables are not made available when building pull requests. The rationale behind is that otherwise someone could submit a pull request containing an altered appveyor.yml that prints out the environment variables to the build log. That decision makes sense.

When using Coveralls.IO with AppVeyor it leaves two options for getting coverage on pull requests: Check in the Coveralls.IO api key with the source, or use web based config. I went for the latter. Keeping the key directly in appveyor.yml makes it possible for anyone to upload spurious coverall results. I don’t know if that would really matter, but I opted to keep the API key secret through using web based config.

The Tooling

The code coverage is calculated using the tooling built into Visual Studio Enterprise. Those are supported on AppVeyor. So the actual coverage is calculated as part of running the unit tests during the build. Then the results need to be converted to the Coveralls.IO format and uploaded. For that I’m using the Coveralls.net package. Please note that the 0.7.0.0-beta0001 version or later is required for the pull request support. This means that as the time of posting this the package need to be installed with Install-Package coveralls.net -Pre.

The Configuration

These are the additional steps I took to enable exports to Coveralls.io from my AppVeyor builds. Following this guide assumes that you already have working AppVeyor builds, or that you can fill in the gaps yourself.

1. Disable configuration through appveyor.yml. Usually an appveyor.yml file takes precedence.
2. Set up an environment variable in the AppVeyor settings called COVERALLS_REPO_TOKEN containing the key from Coveralls.io.
3. Set up a custom powershell test script (see contents below). Ensure that you have a working runsettings file that excludes non-relevant projects from the code coverage. Typically you don’t want to check code coverage of the unit test code itself.

This is the PowerShell test script I’m using. It obviously needs some paths changed to match another project.

vstest.console.exe /logger:appveyor /Enablecodecoverage /settings:codecoverage.runsettings .\Kentor.AuthServices.Tests\bin\Debug\Kentor.AuthServices.Tests.dll

$result = $LASTEXITCODE

$coverageFilePath = Resolve-Path -path "TestResults\*\*.coverage"

$coverageFilePath = $coverageFilePath.ToString()

if(Test-Path .\coverage.coveragexml){ rm .\coverage.coveragexml }

& "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" analyze /output:coverage.coveragexml "$coverageFilePath"

$coveralls = "packages/coveralls.net.0.7.0-beta0001/tools/csmacnz.coveralls.exe"

& $coveralls --dynamiccodecoverage -i coverage.coveragexml --useRelativePaths

if($result -ne 0){
 exit $result
}

The coveralls command automatically detects if it is a Pull Request build or not when running on AppVeyor through enviornment variables. It’s also possible to specify those parameters through command line switches.

GitHub Integration

Coveralls.IO can be configured to automatically add notifications back to GitHub and include them as pre-merge tests. The failing criteria can be set either as an absolute number or as the maximum decrease in coverage allowed.

It’s Awesome

Finally I just want to say how much i like this solution. From the start, the AuthServices project didn’t even have CI builds. For every pull request I had to download the code, compile, run tests and check coverage manually. Now I don’t have to any more. The CI build ensures that the PR builds and tests works and Coveralls can make me confident that there is complete test coverage (which is required in the project).

Now I can just go straight for the merge button if the CI build and coverage pass and the diff looks good.