Using xUnit and Pipeline

The JUnit plugin is the go-to test result reporter for many Jenkins projects, but it is not the only one available. The xUnit plugin is a viable alternative that supports JUnit and many other test result file formats.

Introduction

No matter the project, you need to gather and report test results. JUnit is one of the most widely supported formats for recording test results. For a scenario where your tests are stable and your framework can produce JUnit output, the JUnit plugin is ideal for reporting results in Jenkins. It will consume results from a specified file or path and create a report, and if it finds test failures, it will set the job state to "unstable" or "failed."

There are also plenty of scenarios where the JUnit plugin is not enough. If your project has some failing tests that will take some time to fix or if there are some flaky tests, the JUnit plugin's simplistic view of test failures may be difficult to work with.

No problem; the Jenkins plugin model lets us replace the JUnit plugin functionality with similar functionality from another plugin and Jenkins Pipeline lets us do this in a safe stepwise fashion where we can test and debug each of our changes.

In this post, I will show you how to replace the JUnit plugin with the xUnit plugin in Pipeline code to address a few common test reporting scenarios.

Initial Setup

I'm going to use the "JS-Nightwatch.js" sample project from my previous post to demonstrate a couple common scenarios that the xUnit handles better. I already have the latest JUnit plugin and xUnit plugin installed on my Jenkins server.

I'll be keeping my changes in my fork of the "JS-Nightwatch.js" sample project on GitHub, under the "blog/xunit" branch.

Here's what the Jenkinsfile looked like at the end of that previous post and what the report page looks like after a few runs:

Switching from JUnit to xUnit

I'll start by replacing JUnit with xUnit in my pipeline. I use the Snippet Generator to create the step with the right parameters. The main downside of using the xUnit plugin is that while it is Pipeline compatible, it still uses the verbose step() syntax and has some very rough edges around that, too. I've filed JENKINS-37611 but in the meanwhile, we'll work with what we have.

If I replace the junit step in my Jenkinsfile with that last step above, it produces a report and job result identical to the JUnit plugin but using the xUnit plugin. Easy!

Accept a Baseline

Most projects don't start off with automated tests passing or even running. They start with a people hacking and prototyping, and eventually they start to write tests. As new tests are written, having tests checked-in, running and failing can be valuable information. With the xUnit plugin, we can accept a baseline of failed cases and drive that number down over time.

I'll start by changing the Jenkinsfile to fail jobs only if the number of failures is greater than an expected baseline (in this case, four failures). When I run the job with this change, the reported numbers remain the same, but the job passes.

Next, I can also check that the plugin reports the job as failed if more failures occur. Since this is sample code, I'll do this by adding another failing test and checking the job reports as failed.

In a real project, we'd make fixes over a number of commits bringing the number of failures down and adjusting our baseline. Since this is a sample, I'll just make all tests pass and set the job failure threshold for failed and skipped cases to zero.

Allow for Flakiness

We've all known the frustration of having one flaky test that fails once every ten jobs. You want to keep it active so you can work on isolating the source of the problem, but you also don't want to destabilize your CI pipeline or reject commits that are actually okay. You could move the test to a separate job that runs the "flaky" tests, but in my experience, that just leads to a job that is always in a failed state and a pile of flaky tests that no one looks at.

With the xUnit plugin, we can keep the flaky test in main test suite but allow the job to still pass.

I'll start by adding a sample flaky test. After a few runs, we can see that the test fails intermittently and causes the job to fail, too.

I can almost hear my teammates screaming in frustration just looking at this report. To allow specific tests to be unstable but not others, I'm going to add a guard "suite completed" test to the suites that should be stable and keep the flaky test on its own. Then I'll tell xUnit to allow for a number of failed tests, but no skipped ones. If any test fails other than the ones I allow to be flaky, it will also result in one or more skipped tests and will fail the build.

After a few more runs, you can see the flaky test is still being flaky, but it is no longer failing the build. Meanwhile, if another test fails, it will cause the "suite completed" test to be skipped, failing the job. If this were a real project, the test owner could instrument and eventually fix the test. When they were confident they had stabilized the test they could add a "suite completed" test after it to enforce it passing without changes to other tests or framework.

Conclusion

This post has shown how to migrate from the JUnit plugin to the xUnit plugin on an existing project in Jenkins pipeline. It also covered how to use the features of xUnit plugin to get more meaningful and effective Jenkins reporting behavior.

What I didn't show was how many other formats xUnit supports, from CCPUnit to MSTest. You can also write your own XSL for result formats not on the known/supported list.