How I Learned to Stop Worrying and Love TestNG

Think back to your early 20′s. Remember going out on Halloween or New Year’s Eve? No sooner do you walk into a bar than someone in your group starts lobbying to go to some other bar. ‘Cuz it’s gonna be way better. (In your exasperation – you just ordered a drink for pete’s sake! –  you vow that next year you’re gonna find a house party instead.)

Apologies in advance, but… I’m gonna be “that someone”. Replace New Year’s Eve with every Monday through Friday. The bar you just walked into is JUnit. That other bar you should, like, totally make a b-line for is TestNG - an open-source automated testing framework.

Now, where this analogy breaks down (assuming it ever had legs to begin with), is that unlike the human confines of singular spacio-temporal existence™, you can use both JUnit and TestNG at once. Here are 3 reasons you should.

Groups

This is hands-down my favorite thing about TestNG because it’s the most flexible way to split your tests into sub-suites. Let’s jump right to an example, inspired by Bamboo (a CI server from Atlassian). Here we see two tests around ordering sub-steps, or "Tasks", within a build job:

Each test has been assigned to groups representing functional area of the application, level of technology stack, maturity level, and whether it should be run as a post-deploy smoke test. This is incredibly powerful because you can use these assignments at run time to pull tests from disparate classes or packages and run them inside a single job. A job that runs all the API-level tests having to do with configuration functionality, for example. Or a job that runs all the tests for features that are still in development and not really expected to pass yet (TDD, anyone??).

And you guessed it: the logical next step is to create jobs that cover the fully body of tests you wish to execute, and run those jobs in parallel inside a stage. Check out this post for more implementation details.

@DataProvider

Let’s say you’re developing, oh I dunno… a continuous integration server. Let’s also say that your CI server needs to support switching between branches from build to build. There are loads of permutations you want to test, but you’re verifying the same functionality for each of them.

What’s a dev to do? Duplicate the hell outta their test code, making explicit sets of tests for each repo type/branch type combination? Oh, the humanity! But wait: up there in the sky! It’s a bird… no, it’s a plane… no! it’s @DataProvider! Here’s a simple example from TestNG’s site:

The output of that test will be: 

But wait. Junit supports parameterized tests as of v.4.4, which is essentially what this is. So why bother with TestNG? First, TestNG’s implementation is slicker and easier than JUnit’s. Second, and more importantly, the reporting in TestNG is much clearer. TestNG tells you exactly which inputs were used for each execution, making it much easier to diagnose failures.

JUnit just tells you which array the parameters came from, and their positions therein – 'bout as clear as mud.

Skipped Tests

Adherents to the D.R.Y principal (and that’s most of us) rely on @BeforeTest or @BeforeClass set-up methods. But if something goes awry in set-up, there’s no point in trying to run the actual test. We know it just gonna fail. With TestNG, the default behavior is to simply skip a test if its upstream dependency barfs. I love this for two reasons.

First, executing the actual test is a waste of time and just means you wait that much longer to find out this revision of your code no es bueno. Second, when you look at your test results, your count of skipped tests makes it immediately clear that something fundamental to your application is borked. Much faster than diving into 350 test failures and, somewhere around number 20, noticing that they’ve all failed on exactly the same call.

And it’s not just set-up methods that can trigger a skip. If you’ve declared TestA as being dependent upon TestB, and TestA fails, TestB will be skipped. Here’s what that looks like in your build logs:

Two Can Play at This Game…

Incorporating TestNG into your arsenal is easy, and again: you can use it in concert with JUnit. Add it as a dependency in your Maven POM, or as an Ant task. Then get the TestNG plugin for Eclipse or IDEA and start sprucing up your tests. Visit testng.org for details.

Among the goodies in Bamboo’s latest release is native support for parsing TestNG results - @DataProvider, skipped tests… the whole works. Just add the TestNG Parser as a final task in the jobs that execute tests. If you’re running the tests by way of Maven, Ant or Gradle, be sure to disable test reporting in those tasks so there are no conflicts. The TestNG plugin for Jenkins also supports all three features I've talked about. 

Try it out and see if you love TestNG, too!

ps: For more tips and best practices, be sure to check out our new automated testing page with all sorts of helpful info. Even automation gurus are sure to find at least one new idea