When 100% Coverage Gives Us a False Sense of Security

I use code coverage tools on a regular basis only to ensure that the most complex areas of the code base are properly tested. I tend to bravely fight the temptation to get excited by this metric and waste valuable time adding meaningless tests (e.g. getters and setters) just to keep the numbers going up. I fell into temptation once, and I got in trouble.

While working on the latest version of the FEST’s JavaFX Maven plugin, I reached an impressive 100% code coverage that made me feel pretty confident that the plugin worked as expected. The truth is that I almost released this plugin without realizing it didn’t work at all.

100% code coverage may sound fantastic, but in this case it gave me a false sense of security. The underlying problem was not the number of tests or the coverage tool itself, but the assumptions made when writing those tests. The test suite consisted exclusively of unit tests testing classes in isolation, using mock or stub implementations of their collaborators. I worked hard to have it cover every single line of code I wrote, verifying that my assumptions of how the program should work were correct. In fact, the test results clearly reflected that the code was doing what I expected it to do.

The test suite didn’t include any functional tests, because I couldn’t figure out how to automate a test involving a command-line tool (Maven.) I originally thought it was an overkill to go into Maven internals and see how everything works, since the plugin I’m writing is pretty simple. I was actually being lazy.

Once I looked at the code coverage report, I felt pretty good about the health of the code base. I assumed everything will work, and I was ready to release. Just for curiosity, I launched a command prompt and typed “mvn fest-javafx:compile” expecting it to succeed. Sadly, it never worked.

After reading my code I finally realized the problem. I had two instance variables with the same name, just different case. One was JavaFXHome, a string that receives the value of the JavaFX home directory specified by a user (and injected by Maven.) The other one was javaFxHome, an instance of the class that figures out the path of the JavaFX home directory if it is not specified by the user. Maven was getting confused by the variable names and was trying to inject a value in the wrong variable. This failure proves that I made a wrong assumption of how Maven works, and my test suite didn’t cover it.

This functional test, even thought is manual, saved me from releasing faulty software.

Lesson learned

This was an interesting experience, and I got to learn something useful:

1. a high percentage of code coverage does not reflect the health of the code base
2. unit tests only verify that the programmer’s assumptions are correct
3. functional tests are a must to ensure that an application, as a whole, works as expected in front of the user
4. not all tests are worth automating
5. if automation is not possible, at least test manually

Feedback is always welcome! :)

From http://alexruiz.developerblogs.com/?p=1030