@Ignore unit tests immediately if they fail!

In development teams, there is often discussion about using the @Ignore tag for failing tests.

In particular, developers who are very enthusiastic about writing unit tests often tend to be very dogmatic about it. They argue, "A test should never be ignored! It’s better to have the build server on Yellow if tests fail. That way we always have an overview of our problems and are forced to fix them!"

What might sound good at first is not a good idea, at least when it comes to large development teams.

If you have several commits from multiple developers and maybe even additional deliveries from different branches to the main development line, the build will definitely break from time to time. How long does it take to fix such an error? Sometimes it may be easy, but in some cases, it may take days to fix a broken test. Now let’s assume that we have implemented the rule that you should never ignore a test. This means that the build may stay Yellow for a long time.

If another developer now updates his workspace with the latest version from the trunk and runs the unit tests locally, he will see failing tests even if he didn’t make any mistakes! That’s pretty bad itself. But he then checks the build server and, after some time looking around, he realizes it wasn’t his fault, as the server is already in the Yellow state. Should he commit now? If he does, he will receive endless emails from the build server about the broken build state that he didn’t cause. If not, his work and maybe the work of others will be blocked until the test is fixed. In a typical, pretty hectic IT project, with it’s almost impossible to meet deadlines, this is not really an option.

What is the solution?

1. If you cause a build to break, @Ignore the failing test immediately and commit the change.
2. Now that the build server is Green again (or Blue in the case of Hudson), start fixing the test.
3. After you have fixed the test, commit the change and check to see if the build stays Green.
4. Have some kind of statistic page on the build server that lists all ignored tests – this allows you to easily track the tests that are currently disabled.

Don’t get me wrong: all failing unit tests should be fixed as fast as possible! But my failure when committing a change should never constrain other team members from doing their work.