Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

VSTS Build Failed Test Phase, but 0 Tests Failed

DZone's Guide to

VSTS Build Failed Test Phase, but 0 Tests Failed

Ever run into a situation where your build failed despite all your tests passing? One recounts he resolved this issue when he encountered it.

· Performance Zone ·
Free Resource

xMatters delivers integration-driven collaboration that relays data between systems, while engaging the right people to proactively resolve issues. Read the Monitoring in a Connected Enterprise whitepaper and learn about 3 tools for resolving incidents quickly.

I had a strange situation where I had a build that suddenly started signaling that tests were failing, but actually zero tests failed.

Figure 1: No test failed, but the test phase was marked as failedFigure 2: The output of the Test Step action is, indeed, that the test failed.

As you can see in Figure 1, the test step is marked as failed, but, actually, I have not seen a single test fail; indeed a strange situation. To troubleshoot this problem, you need to select the failing step to verify the exact output of the task.

As you can see in Figure 2, the VS component that executes the tests is reporting an error during execution, this implies that, even if no test is failing, something wrong happened during the test. This happens if any test adapter writes to the console error during execution and it is done to signal that something went wrong.

Failing a test run if there is an error in the output is a best practice, because you should understand what went wrong and fix it.

To troubleshoot the error you need to scroll through all the output of the task, to find exactly what happened. Looking at the output I was able to find the reason why the test failed.

2017-11-17T14:43:19.6093568Z ##[error]Error: Machine Specifications Visual Studio Test Adapter - Error while executing specifications in assembly C:\A\_work\71\s\src\Intranet\Intranet.Tests\bin\Release\Machine.TestAdapter.dll - Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

If an adapter cannot load an assembly, it will output an error and the test run is marked as failed because, potentially, all the tests inside that assembly were not run. This is a best practice, you do not want to have some tests skipped due to load error and still have a green build.

Figure 3: How to exclude a specific dll from the test run.

The Machine Specification test adapter tried to run a test in Machine.TestAdapter.dll and this created a typeLoadException. Clearly, we do not need to run a test in that assembly, because TestAdapter.dll is the dll of the test adapter itself and does not contain any tests. This problem indeed happens after I upgraded the NuGet package of the runner, probably in the old version the TestAdapter.dll did not creates errors, but with the new version, it does throw an exception.

The problem lies in how the step is configured because, usually, you ask the test runner to run tests in all assemblies that contain 'test' in the name. The solution was to exclude that assembly from the test run.

And voilà, the build is now green again. The rule is, whenever the test step is failing but no test failed, most of the time is some test adapter that was not able to run.

Gian Maria.

Discovering, responding to, and resolving incidents is a complex endeavor. Read this narrative to learn how you can do it quickly and effectively by connecting AppDynamics, Moogsoft and xMatters to create a monitoring toolchain.

Topics:
performance ,performance testing ,testing problems

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}