Is it relevant to have a code coverage target? In a talk at NDC Oslo 2014 Uncle Bob said that the only reasonable goal is 100%. On the other hand Mark Seemann recently said on Twitter and in a follow up blog post that “I thought it was common knowledge that nothing good comes from code coverage targets.” Those are two seemingly opposing views.
Before looking at the role of code coverage, I’d like to take a few steps back and look at the objectives of testing.
When working properly with test driven development, no production code is written unless there is a failing test driving the change. As soon as the test passes, no addition of production code is allowed until there is again a failing test. If that principle is followed, there will simply be no production code that is not covered by a test.
My objective is to achieve high code quality.
My way of achieving that is TDD.
An effect of that is that I do get full code coverage.
The reasoning above is an example of logical implication:
P → Q. Let
P= “high code quality” and
Q = “high code coverage”. Then
P → Q. In plain English it means that If
P is true,
Q has to be true too. In what cases can we draw a conclusion on the value of
P if we know the value of
Q? In exactly one case: If
Q is false, then
P also must be false. If
Q is true, we can’t say anything about the value of
In propositional logic, it is well known that an implication does not work backwards. In the case of code coverage and testing, it is well known that high code coverage is not the same as functional coverage or high quality. I’ve written about it and Martin Fowler shares a real horror story in his article on assertion free testing.
How to Interpret Code Coverage
Code coverage can be deceiving by giving us the comfort of a good metric without actually guaranteeing anything. But that doesn’t mean that it’s useless. The negation of an implication works backwards. If
P → Q and we want to ensure
P is true, we first have to ensure that
Q is true. So if the code coverage numbers drops, we can safely draw the conclusion that the functional coverage is lower too. That’s why I still think that 100% is the only reasonable code coverage target. That’s what I answered when Uncle Bob asked if anyone in the room had a coverage goal.
100% is that realistic? Hell no, no one can get 100% coverage. But, that’s the only meaningful goal. If your goal is 80%, then what you are saying is that it’s okay if 20% of the code doesn’t work. Your goal has to be 100%, even though you cannot hit this goal, it’s an asymptotic goal.
Refactoring and Redundant Code
There are two more cases where I think that code coverage can really help. One is during refactoring, when it can help ensuring that no extra functionality is accidentally added. When doing major refactoring, I think that it is very easy to accidentally introduce some extra feature. It could be a small error check or it could be a minor tweak to an algorithm. Keeping track of code coverage will clearly show if such an addition has been done.
There is also the opposite case, where refactoring moves functionality to a new place so that some old handling of the case is now unreachable code. That will also be detected by code coverage.
There is, however, one obstacle in the way for using code coverage as the detection mechanism for those cases. We need a way to make it immediately obvious that there is now a redundant or untested line of code somewhere. We need to make that line stand out. We need to make sure that it is the one line that breaks the perfect 100.0% code coverage score.
But, there are always things that cannot reasonably be tested. For example in Kentor.AuthServices there are several parts of the HttpModule API that are extremely hard to test because of how the IIS API is designed. That code is mostly plumbing code, with very few changes. There are very few conditional branches in the code, so a few simple high level integration tests will find any problems. I’ve come to the conclusion that it’s not worth the effort to write unit tests for those. But I still want the 100.0% code coverage score.
This is actually how the entire twitter thread by Mark Seemann started.
I use the
[ExcludeFromCodeCoverage] attribute to make deliberate deviations from the full-code-coverage standard. All code can’t be tested at a reasonable cost. So a full 100.0% code coverage target is not possible. Somewhere between 80% and 100% is realistic. But I want those exclusions to be deliberate. If I accepted 87.4% coverage, there would be no way that I could find if I accidentally missed coverage on another line (one that mattered). It’s the same as compiler warnings: the only policy that works in the long term is 0 warnings. Once in a while there are warnings that are safe to ignore – that’s where we use a
#pragma to suppress the warning in that specific place.
[ExcludeFromCodeCoverage] is the corresponding mechanism for code coverage.
In the end I don’t think that I, Mark Seemann or Uncle Bob are that far apart in our view on TDD. The code coverage is just a means to reach the real goal: high code quality. There is no value in a high code coverage number on its own.