Test Behavior, Not State

Many developers and testers write automated tests. Sometimes they're unit tests, other times package level, and occasionally integration. There are many different types of tests, but there are a few characteristics of great tests. Today let's look at one specific characteristic: behavior versus state.

A common trap that many test crafters fall into is letting their test see too far into the class. Delving into the code inside the method, checking data structures, seeing what routines were called, and looking at variables are all signs that your test has gone too deep. By coupling the test so completely with the exact implementation of the method, you've made it impossible to refactor the code inside the method without changing the test as well. This creates more work for you (generally a bad thing), but also forces you to update the test to match the changes you've made. Adjusting the test gives you a change to change what's being tested, which allows bugs to creep in.

For example, say I've got a routine that grabs a list of students in a class, calculates each student's grade, then returns an average for the entire list. The method signature might look like this:

public Integer calculate_classes_average_grade( String class_name)

You pass in a class name and get a number in return. There are several good ways to test a routine like this. One way would be to pass in a class name, have it pull in known data from a database, let it calculate the average GPA and return it to your test harness. You'd then verify the return value. (You'd probably want to use Dave Hussman's "Nuke and pave" idea on the database to ensure a clean, known data set.)

Another way to test it would be to mock the database connection, have the mock return a known data set, then again, test the return value.

A third way would be to drive the code that uses this routine, testing it from a higher level. Perhaps a package level test or even a GUI test would work for this scenario.

In each case, we're using the code and checking the results. The test doesn't know anything about how the grades are retrieved (apart from the mocking). It only knows what the code should return. This frees up the development team to change the code inside the routine. Perhaps you want to make the code more efficient, change the data structures being used, or completely rewrite the internals of the class. If you've fallen into the trap of overly coupling your tests to your code, these changes will fail your tests, and you'll have two classes to rewrite.

Sometimes a developer will write a test that runs this routine, then checks the internal list of students to see if the correct number of students is in the list, then checks if the right students are in the list, and so on. For the moment let's ignore that that bit of code should be broken out into a subroutine so that you can do that check properly. Instead, let's look at what happens if you write a code that peeks into the class, grabs that data structure, and verifies the structure's internal state. When you give the test that level of insight into the class being tested, what happens when the data structure changes from an array to a hash table? What happens when the order of the students in the list changes? What happens when you want to run the test against a different data set?

In each of these scenarios, the test fails, so you have to update the code. This creates unnecessary work, wastes your time, and introduces the potential of "fixing" the test in a way that lets errors creep in.

So take a step back. Never let your automated tests, no matter their type or flavor, see inside the code they're testing. Use the class, the package, or the routine. Pass in the data values that make it run, and then check the results. This is one of the best ways to write solid, reusable tests and ensure you can refactor your code without having to update your tests at the same time.