The Art of Designing Automated Testing Tools

Designing automated testing tools can be a difficult task. Not only do you have to surface all of the complexities involved in a variety of test structures but you also have to make all of those complexities accessible to a variety of audiences.

Some people prefer to script their tests, working directly with commands, variables, and objects so they can manipulate the interactions without any of the restrictions imposed by a user interface. Others are not comfortable writing code and prefer to use a GUI-based tool to record actions and create assertions. Even when it comes to scheduling the tests to run, there are the CLI fans who want to hook the tests to automated build tools and others who want a centralized hub for scheduling tests in different environments.

As a tools vendor, you have to decide where to put your limited resources and time. Ultimately, it comes down to knowing your customers’ mindset – what do they expect from your tool? In some cases, one organization can have different requirements, depending on how they manage their automation. While it’s important for vendors to have a focused approach to building their tools, it’s also important that they design tools that provide choices to accommodate as many user preferences as possible.

Our friends at Infinio shared some thoughts with us about the difference between an “outside in” approach to writing automated tests versus an “inside out” approach, as well as their approach to managing those tests. This is the kind of conversation we love having with testers because it better informs how we build our products.

At SmartBear, we’ve tried to walk the line between providing easy to use tools for the novice while still allowing coders to get into the fabric of our tools and do their own scripting. It’s not always an easy line to walk – it’s important to stay in constant touch with your customers so you know what their preferences are and whether you have provided them with the right interfaces and capabilities. Sometimes that also means listening to their evolution over time – do they still want/need the same things they did a year ago? Two years ago?

Let us know what you think. Do you prefer a GUI-based automation tool, a script-based automation tool, or one that offers both capabilities?