Unit Test Runs Stale Code Because Code Coverage Is Enabled in Visual Studio

A quick post to (maybe) save some of you some troubleshooting time one day.

Executive summary: If your Visual Studio unit tests work when a debugger is attached and fail when there’s no debugger attached, make sure that the test is running the latest version of your code, and if it’s not, try turning off code coverage in your test run configuration.

During the last two weeks I wasted at least an accumulated 5 hours trying to understand why my Visual Studio unit tests fail even after I fixed the bug. In particular, I couldn’t wrap my head around why they fail when I run them without a debugger attached, and succeed spectacularly when running under a debugger.

What I eventually found is that my TestResults\…\Out directory, which contains the assemblies used by the test, has a stale version of my code in it—i.e., doesn’t contain the fixes I just made to address the failing unit tests. I figured this out by actually opening the assembly with Reflector and seeing, much to my horror, that my code changes haven’t made it into the binary.

Following Guy’s advice I ran mstest from the command line and it generated quite a few warnings about items that could not be deployed because they conflict with items that have been instrumented for code coverage. I turned off code coverage, and the problem went away.