DZone
Java Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Java Zone > When 100% Coverage Gives Us a False Sense of Security

When 100% Coverage Gives Us a False Sense of Security

Alex Ruiz user avatar by
Alex Ruiz
·
Apr. 13, 10 · Java Zone · Interview
Like (0)
Save
Tweet
5.60K Views

Join the DZone community and get the full member experience.

Join For Free

I use code coverage tools on a regular basis only to ensure that the most complex areas of the code base are properly tested. I tend to bravely fight the temptation to get excited by this metric and waste valuable time adding meaningless tests (e.g. getters and setters) just to keep the numbers going up. I fell into temptation once, and I got in trouble.

While working on the latest version of the FEST’s JavaFX Maven plugin, I reached an impressive 100% code coverage that made me feel pretty confident that the plugin worked as expected. The truth is that I almost released this plugin without realizing it didn’t work at all.

100% code coverage may sound fantastic, but in this case it gave me a false sense of security. The underlying problem was not the number of tests or the coverage tool itself, but the assumptions made when writing those tests. The test suite consisted exclusively of unit tests testing classes in isolation, using mock or stub implementations of their collaborators. I worked hard to have it cover every single line of code I wrote, verifying that my assumptions of how the program should work were correct. In fact, the test results clearly reflected that the code was doing what I expected it to do.

The test suite didn’t include any functional tests, because I couldn’t figure out how to automate a test involving a command-line tool (Maven.) I originally thought it was an overkill to go into Maven internals and see how everything works, since the plugin I’m writing is pretty simple. I was actually being lazy.

Once I looked at the code coverage report, I felt pretty good about the health of the code base. I assumed everything will work, and I was ready to release. Just for curiosity, I launched a command prompt and typed “mvn fest-javafx:compile” expecting it to succeed. Sadly, it never worked.

After reading my code I finally realized the problem. I had two instance variables with the same name, just different case. One was JavaFXHome, a string that receives the value of the JavaFX home directory specified by a user (and injected by Maven.) The other one was javaFxHome, an instance of the class that figures out the path of the JavaFX home directory if it is not specified by the user. Maven was getting confused by the variable names and was trying to inject a value in the wrong variable. This failure proves that I made a wrong assumption of how Maven works, and my test suite didn’t cover it.

This functional test, even thought is manual, saved me from releasing faulty software.

Lesson learned

This was an interesting experience, and I got to learn something useful:

  1. a high percentage of code coverage does not reflect the health of the code base
  2. unit tests only verify that the programmer’s assumptions are correct
  3. functional tests are a must to ensure that an application, as a whole, works as expected in front of the user
  4. not all tests are worth automating
  5. if automation is not possible, at least test manually

Feedback is always welcome! :)

From http://alexruiz.developerblogs.com/?p=1030

unit test Code coverage security Test suite

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • 5 Steps to Strengthen API Security
  • 12 Kick-Ass Software Prototyping and Mockup Tools
  • Modernize Legacy Code in Production: Rebuild Your Airplane Midflight Without Crashing
  • ETL, ELT, and Reverse ETL

Comments

Java Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo