Building a Test Automation Strategy

Ashley Hunsberger, Greg Sypolt and Chris Riley contributed to this post.

Bringing test automation into your organization is not as easy as writing and running a Selenium script. It involves first getting buy-in, building a team, establishing a strategy, and picking the right tools. During the Q&A portion of a recent webinar hosted by Chris Riley, Ashley Hunsberger, and Greg Sypolt, the presenters realized that these aspects of introducing test automation are well known, but not well understood. In our first post of the series we discussed getting buy-in. Below, in the second post, we discuss how to build a test automation strategy.

Getting started with test automation is easy. If you have a technically minded QA team, you can usually create your test script, sign up for a test cloud, and run the script in just a few hours. But keeping a test automation environment going for the long term is not as easy as any of us would like to believe. QA teams are generally better at building strategy than any other. And when it comes time to build a test automation environment, strategy is a key first element to both getting started and keeping it going.

When building a strategy, you have to address how the environment works, how the tests are run, how the test suite is maintained, the process of running tests, the design patterns of the test scripts, and more. Let’s look at questions from the webinar to address some ways to approach your test automation strategy.

Would you define what “traditional QA” is and what QA “was”?

Ashley: When I say traditional QA, I’m referring to waterfall. Teams got their requirements, engineers did their work, QA went off and did theirs, but couldn’t really do anything until dev was ‘complete’—and I use that term loosely. Devs wouldn’t know there was a bug in their code until sometimes weeks or months later because of testing cycles.

Greg: Traditional processes are designed for getting requirements, and developers and QA work in their silos. There is rarely any communication between developers and QA during the sprint. At the end of the sprint, developers will throw finished code over the wall at QA. This approach doesn’t allow for collaboration, and it leads to slow feedback, and no iterations during the sprint. Iterations are a key element to modern dev.

Can you define a list of essential skills every modern QA team member should have?(This blog post expands on this topic.)

Ashley: I still maintain that a QA mindset, whether via automation or not, is incredibly valuable in determining a holistic test strategy. You always need to be able to consider the end user, how the system works at a broader level than your team’s feature du jour, and what other types of testing to consider (not just automated tests, but accessibility, localization, performance, usability, security, exploratory). The key is how to incorporate that into your sprint and really push to the definition of done.

Greg: For us, it’s not a choice. We do not have dedicated DevOps resources as a Platform as a Solution (PaaS), so we are experimenting with the modern QA team member that can have both QA and DevOps responsibilities. QA is the gatekeeper of quality and it makes sense they maintain and champion the continuous integration tooling.

How do we write more tests faster?

Greg: Why do you need to write more tests? Everyone shares responsibility and follows DoD. Focus on building the right types of tests. It’s more about building a testing portfolio that identifies the areas of the application that would be a showstopper for end users, or that affect how your application makes money (at Gannett || USA Today, network ads testing is critical).

Ashley: I completely agree with Greg. Make sure you are writing the right tests instead of writing something just because you can. At Blackboard, we identify the most critical features and workflows and automate those, with as much unit and integration testing as possible, and some UI integration tests for our showstopper workflows.

Many times I feel like the DevOps Infrastructure problems have to be solved before I can do test automation. Is that true?

QA, Dev, and DevOps should share this task. It isn’t necessary for local development, but when it comes to scale and Continuous Integration (CI), it must be done up front (infrastructure first, not infrastructure last).

Greg: Read this post regarding the shared responsibilities. The team shares responsibilities for DevOps tasks. No one has been a champion for DevOps tasks. The modern QA position has become a technical role, the gatekeeper of quality, and may take on more DevOps responsibilities and tasks.

Ashley: Every company is different. At ours, QA does work closely with our DevOps team more and more as we transition to a modern delivery chain. We definitely still have our kinks, but we are still doing test automation. We are still working to be in the CI pipeline, but it doesn’t prohibit meaningful tests.

There is no question that QA is evolving. More tech, more strategy. But this is a fantastic opportunity for QA to become a first-class citizen. Given QA teams’ holistic view of the entire delivery chain (compared to IT Ops and developers consistently in the weeds), QA is best suited to build a strategy with DevOps to modernize development operations.