Choosing an Acceptance Test Framework

I can still remember writing my first automated tests in a professional setting. Notice that I didn’t say unit tests, and for good reason. At the time, some 14 years ago, I hadn’t heard of unit tests. Instead, I simply automated the process of manually testing my software. 

This may sound somewhat facile, but it actually speaks to core principles in the programming profession. As a relatively inexperienced programmer, I understood the importance of testing my work. I also understood the importance of automating manual, error-prone process. And so, of my own accord, I examined and then automated my previously manual testing efforts.

Don’t get me wrong. By doing this, I reinvented a wheel simply because I did not know of its existence. Folks had created automated unit test frameworks for this exact purpose. Had I known, I could have better spent my time learning and using these things. But, in spite of the waste, I did learn something. I learned that, under the covers, test frameworks just represented yet another instance of automating an important manual process. 

What Is User Acceptance Testing (UAT)?

I led with a tale about unit tests because testing software components applies to everyone that writes software. I mean, you always test your own software, even if you don’t think you’re doing so. Whenever you compile, you test your code to see if it compiles. Granted, you aren’t executing the most sophisticated, high-value test known to man. But you are performing a test of sorts. 

But what if we look beyond our own dev boxes a bit? What if we look at other forms of testing?

For almost any software that we write, other stakeholders will perform other sorts of tests. These stakeholders include users or user-proxies, who perform an activity known as user acceptance testing (UAT). In its simplest incarnation, this involves users or their proxies using the software and evaluating whether or not they find it acceptable. 

This can come in various shapes and sizes. In some cases, actual users perform formalized beta tests, perhaps for pay. In other cases, someone from the QA group might do a quick run-through and give a thumbs-up or thumbs-down. But whatever happens, these tests capture the user’s experience and perspective.

The Essential Conundrum of Automated UAT

In my story about automating my own testing efforts, I could close the loop. In other words, I could use a programming language to write a program. I could then use that same programming language that wrote another program to test the first program. You could think of this as akin to an accountant making use of double entry bookkeeping. The accountant performs two distinct activities toward a single goal, but both still involve accounting.

Now think about user acceptance testing. This activity happens in the natural language of users, rather than in the explicit terms preferred by a compiler. While writing my own tests as an entry-level programmer, I might have typed, “assertIsFalse(newBalance < 0)”. But when discussing the issue with beta users, I would have heard instead something like, “it shouldn’t let me go negative on balance without at least giving me a warning or something.”

As a profession, we programmers have the same impulse to automate acceptance testing that I did so many years ago with my component testing. But, to do this, we need to overcome the challenges of going outside our profession’s domain. 

UAT Frameworks to the Rescue

Luckily, we have another parallel from my introductory story. UAT frameworks already exist, whether you currently use them or not. Clever people have figured out a way to bridge the gap between users’ conception of systems and automating the testing of those concepts.

These come in various flavors, but I’ll list two common ones. First, frameworks exist that “script” GUI interaction. To imagine this in its simplest form, think of “recording” something like a login, and then later “replaying” it with a subsequent version of the target software. This guards against regressions because you’re saying, “login should still work like so.” Technical folks can easily sit with users and design these sorts of tests.

The second common way to make this happen involves a language known as “Gherkin.” It asks users to phrase expectations in English (though very specifically formatted English) so that they can later be translated to software instructions. A typical Gherkin test case may say something like, “Given a valid username and password, when I use them to log in, then I expect to see the home page.” User stakeholders can craft these sorts of statements, and then a programmer can work on turning those into actual, executable code.

So with that backstory established, let’s look at how you might choose an acceptance test framework to use. Many exist, so narrowing it down is essential.

Does Tech Stack Matter?

As you start to look at tools that can automate acceptance tests, you’ll find that you have many options. As you peruse various compiled lists, you’ll notice talk of the language in which they’re implemented or what framework they target. You might find yourself naturally inclined to pick one that matches your own software under test.

Strictly speaking, you need not limit yourself this way. After all, these frameworks help you exercise your software but they don’t need to tightly integrate with it. If I had a REST endpoint written in Python, for instance, nothing says I need to use Python to generate HTTP requests for testing it. The same logic can apply with these frameworks.

That said, I recommend you favor frameworks that share a tech stack with your system. Learning skills around UAT automation is hard enough without also learning a new programming language. If a compelling case for one outside the tech stack presents itself, definitely consider it, but lean toward the familiar. 

Using the GUI or Not?

The next bit of advice I’ll offer for narrowing the field involves looking at how you want to interact with your code. As I mentioned earlier, you can roughly divide your acceptance test strategy by the question, “go through the GUI” or “go around the GUI?” 

Architecturally speaking, many applications use a pattern like MVC to neatly decouple application logic from how you view it (i.e. the GUI). In such cases, you can interact with the application at the layer just below the GUI, making “around the GUI” an attractive option. It offers considerably less brittleness than GUI tests, which often depend on frequently-changed concerns like the placement of elements on the screen. But then again, not all architectures lend themselves to this decoupling.

The decision about whether to go through your GUI or the layer below it will definitely help you eliminate a lot of options in your search.

Should You Roll Your Own?

I’ll conclude by addressing a question that seems to loom in any programming-related tool question. Should you consider building your own framework? I highly recommend against this approach.

I get it. Many exist and the issue gets further muddied as people bandy about terms like UAT, Behavior Driven Development (BDD), Acceptance Test Driven Development (ATDD), and seem to have a hard time agreeing which is which and which is a subset of what. It can scare you away from picking anything, if not abandoning the attempt to automate altogether.

But pick a tool and try it. If that doesn’t meet your needs, pick another one and try that. Don’t sweat the terms so much, and don’t let any confusion distract you from the broader goal of automating your user acceptance tests. You want them automated, and you don’t want to do what I did all of those years ago and, in spite of good intentions, reinvent inferior versions of the wheel.