Unit Testing Frameworks in C#: Comparing XUnit, NUnit, and Visual Studio

When you find yourself (or your company) with more code than anyone could ever test by hand, what can you do? Well, unit testing has always been the perfect solution, as you can run tests that check more data than a person could in a day in a matter of milliseconds. So, today I’ll take a look into a few popular C# unit testing frameworks and try them out first hand so you can choose which one best suits your project.

Unit tests can be run as often as you want, on as many different kinds of data as you want and with next to no human involvement beyond once the tests are written.

Not only that, but using code to test code will often result in you noticing flaws with your program that would have been very difficult to spot from a programmer’s viewpoint.

Popular C# Unit Testing Frameworks

The unit testing frameworks I’ll be testing are:

• NUnit
• XUnit
• Built-in Visual Studio testing tools

All of these unit testing frameworks offer a similar end goal: to help make writing unit tests faster, simpler, and easier! But, there are still a few key differences between them. Some are more focused towards powerful complex tests, while others rank simplicity and usability as a higher priority. 

First Up Is Microsoft’s Own Built-in Visual Studio Unit Testing Tools

In most versions since 2005, Visual Studio has come with a built in testing framework supported by Microsoft. This framework certainly wins the most points for installation. Though, if your copy of Visual Studio doesn’t come with it already included, you are going to have to jump through a few hoops to get it going. (We wrote a review of the 2017 version of Visual Studio here.)

This framework is the simplest of the three and uses an easy to understand method attribute structure (much like most testing frameworks) where you are able to add tags such as ‘[TestClass]’ and ‘[TestMethod]’ to your code in order to get testing.

Visual Studio even has a UI panel dedicated to visualizing your tests, which can be found under Test -> Windows -> Test Explorer.

Now, before we dive into trying out this testing framework, let’s introduce our example classes that need testing.

First, we have a Raygun, which we can fire and recharge. The only thing we need to keep track of with our Raygun is its ammo, which can run out.

We also have a bug, which we can shoot at with our Raygun. But this bug has the ability to dodge our attempts to shoot it.

If we shoot at a bug after it has just dodged, we will miss. Though, if we hit the bug square on, it’s safe to assume that it will be dead.

These two classes are defined as follows:

public class Raygun {

private int ammo = 3;

public void FireAt(Bug bug) {
if (HasAmmo()) {
if (bug.IsDodging()) {
bug.Miss();
}
else {
bug.Hit();
}
ammo--;
}
}

public void Recharge() {
ammo = 3;
}

public bool HasAmmo() {
return ammo > 0;
}
}

public class Bug {

private bool dodging;
private bool dead;

public void Dodge() {
dodging = true;
}

public void Hit() {
dead = true;
}

public void Miss() {
dodging = false;
}

public bool IsDodging() {
return dodging;
}

public bool IsDead() {
return dead;
}
}

Seems simple enough, but we need to make sure that our Rayguns and bugs behave as we want them to.

So then, it’s time to write some unit tests! (We wrote about how to write robust unit tests in C# here.)

First up, let’s try a simple situation where we want to shoot at and hit a bug.

What we would expect is that afterward the bug will be dead, and the Raygun will still have a bit of juice left in it.

Well, let's see if we are right:

[TestClass]
public class Class1
{
[TestMethod]
public void TryShootBug() {

Bug bug = new Bug();
Raygun gun = new Raygun();

gun.FireAt(bug);

Assert.IsTrue(bug.IsDead());
Assert.IsTrue(gun.HasAmmo());
}
}

These tags are what allow Visual Studio’s built-in testing framework to recognize this particular class as a class that contains unit tests, and to treat the method TryShootBug() as a test case, instead of just an ordinary method.

Since these tools are built for Visual Studio, running your tests from within Visual Studio is very simple.

Just right click on any [TestMethod] tags as shown:

And would you look at that, the test passed. Looks like our Raygun can at least hit a stationary bug.

Of course, this is only showing the bare basics of what Visual Studio’s testing tools can do.

Some other very useful tags you will surely be using are the [TestInitialize] and [TestCleanup] tags.

These tags allow you to specify code that is run before (initialize) and after (cleanup) every individual test is run.

So, if you want to reload your Raygun after every encounter like a stylish gunslinger, then this should do the trick:

[TestInitialize]
public void Initialize() {
gun = new Raygun();
}

[TestCleanup]
public void Cleanup() {
gun.Recharge();
}

<i>
</i>

Stylish.

While we are still talking about the Visual Studio testing tools I’ll quickly mention the [ExpectedException] tag, which is incredibly useful for when you want to deliberately cause an exception in your test (which you will certainly want to do at some point to make sure your program isn’t accepting data it shouldn’t).

Here’s a quick example of how you would write a test that results in an exception:

[TestMethod]
[ExpectedException(typeof(System.IndexOutOfRangeException))]
public void TryMakingHeapsOfGuns() {

Raygun[] guns = new Raygun[5];
Bug bug = new Bug();

guns[5].FireAt(bug);

}

Overall, the built-in Visual Studio testing tools do exactly what they say on the box. They are simple, easy to use, and handle all the basic testing functionality you would need. Plus, if you’re already working in Visual Studio, then they are already ready to use!

Next Up Is Arguably the Most Popular C# Testing Platform, NUnit

NUnit is an incredibly widely used tool for testing, and it serves as an excellent example of the open source unit testing frameworks. It’s a broad and powerful testing solution. In fact, it’s what we use here at Raygun for the bulk of our unit testing.

NUnit is installed via a NuGet package, which you can search for within Visual Studio.
The packages I’ve used for this example are NUnit and NUnit.ConsoleRunner, though you also have the option of installing a GUI-based plugin for Visual Studio.

NUnit uses a very similar attribute style system just like the visual studio testing tools, but now we will be referring to a [TestClass] as a [TestFixture], and a [TestMethod] as simply a [Test].

Now, let’s go back to our Rayguns and bugs and have a look at another example, but this time using NUnit.

Now in order to run this test using NUnit, we need to seek the command line (unless of course you’ve chosen to install a GUI based plugin.)

This time, let's make sure our dodges and ammo are working properly, so let's try and shoot a much more mobile bug:

[TestFixture]
public class NUnitTests {

[Test]
public void TryShootDodgingBug() {

Bug bug = new Bug();
Raygun gun = new Raygun();

bug.Dodge();
gun.FireAt(bug);

bug.Dodge();
gun.FireAt(bug);

bug.Dodge();
gun.FireAt(bug);

Assert.IsFalse(bug.IsDead());
Assert.IsFalse(gun.HasAmmo());
}
}

Notice the new [TestFixture] and [Test] tags.

First, you must make sure you are in your project’s root directory (e.g. C:\Users\yourUserName\Documents\Visual Studio 2015\Projects\YourProjectName) and then enter the following command in a new cmd window:

packages\NUnit.ConsoleRunner.3.6.0\tools\nunit3-console.exe  
YourProjectName\bin\Debug\YourProjectName.dll 

Assuming everything is set up properly, the NUnit console runner will run all the tests in your project and give you a nice little report on how things went:

Looks like our bug sure can dodge and our Raygun can certainly run out of ammo!

One feature of NUnit that makes it incredibly useful is the ability to include parameters in your tests!

This means that you can write a test case with arguments, then easily run the same test with a range of unique data. This removes the need to write unique test cases for every set of arguments you want to test.

Here’s a quick example test case we could use to make sure our Raygun was actually running out of ammo at the right time, in a much smarter way than before:

[TestCase(1)]
[TestCase(2)]
[TestCase(3)]
[TestCase(4)]
public void FireMultipleTimes(int fireCount) {

Bug bug = new Bug();
Raygun gun = new Raygun();

for(int i = 0; i < fireCount; i++) {
gun.FireAt(bug);
}

if (fireCount >= 3) {
Assert.IsFalse(gun.HasAmmo());
}
else {
Assert.IsTrue(gun.HasAmmo());
}
}

Excellent, with this one test case we were able to make sure a Raygun which has fired two shots still has ammo, while one that has fired three is empty. And thanks to the [TestCase] tag we were easily able to test a whole bunch of other values while we were at it!

Overall, NUnit is an excellent testing framework, and as you delve deeper into what it can offer, it surely exceeds what Microsoft’s built-in testing can offer.

Anyway, let’s look at our last testing framework, and our last attempt at shooting bugs with Rayguns!

If You Like the Sound of Facts and Theories, Then It's Time to Look at XUnit

XUnit is an open-source testing platform with a larger focus in extensibility and flexibility. XUnit follows a more community-minded development structure and focuses on being easy to expand upon.

XUnit actually refers to a grouping of frameworks, but we will be focusing on the C# version. Other versions include JUnit, a very well known testing framework for Java. XUnit also uses a more modern and unique style of testing, by doing away with the standard [test] [testfixture] terminology and using new fancy tags like Facts and Theories.

NUnit and XUnit are actually quite similar in many ways, as NUnit serves as a basis for a lot of the new features XUnit brings forward.

Note that XUnit is also installed via a NuGet package much like NUnit, which you can search for within Visual Studio. The packages I’ve used for this example are XUnit and XUnit.ConsoleRunner, though you also have the option of installing a GUI-based plugin for Visual Studio.

Much like the [TestCase] tag in NUnit, XUnit has its own solution to providing parameters to a test case. To do so, we will be using the new [InLineData] tag and Theories.

In general, a test case that has no parameters (so it doesn’t rely on any changing data) is referred to as a Fact in XUnit, meaning that it will always execute the same (so 'Fact' suits it pretty well). On the other hand, we have Theories, which refer to a test case that can take data directly from [InLineData] tags or even from an Excel spreadsheet.

So, with all these new fancy keywords in mind, let’s write a test in XUnit that uses a theory to test our bugs dodge ability:

[Theory]
[InlineData(true, false)]
[InlineData(false, true)]
public void TestBugDodges(bool didDodge, bool shouldBeDead) {

Bug bug = new Bug();
Raygun gun = new Raygun();

if (didDodge) {
bug.Dodge();
}

gun.FireAt(bug);

if (shouldBeDead) {
Assert.True(bug.IsDead());
}
else {
Assert.False(bug.IsDead());
}
}

This test covers both cases at once, where the bug dodges and survives, or doesn’t dodge and gets hit. Lovely!

Now, last step, let's run the XUnit test runner from the command line (note that much like NUnit, XUnit also has a GUI-based Visual Studio plugin available for you to run tests with).

First, you must make sure you are in your project’s root directory, just like NUnit (e.g. C:\Users\yourUserName\Documents\Visual Studio 2015\Projects\YourProjectName) and then enter the following command in a new cmd window:

packages\xunit.runner.console.2.1.0\tools\xunit.console.exe  
YourProjectName\bin\Debug\YourProjectName.dll 

Assuming everything is set up properly, the XUnit console runner will run all the tests in your project and let you know how your tests turned out.

Looks like our dodging tests passed!

Overall XUnit acts as the more contemporary version of NUnit, offering flexible and usable testing with a fresh coat of paint.

In Conclusion…

Regardless of which of the unit testing frameworks you use, you’re going to be getting all the basics. However, there are a few differences between them that I hope I’ve highlighted so you can choose the right one for your project. Whether it’s the convenience of Microsoft’s built-in unit testing framework, the solid and well-proven status of NUnit, or the modern take on unit testing that XUnit provides, there's always something out there that will give you exactly what you need!

Want to add an extra layer of protection for your code? Catch the errors that fall through the cracks with Raygun. Take a free trial here.