As the preceding research noted, the continuous testing adoption rate is extremely low (26%), even among organizations actively practicing Agile and DevOps. This seems shocking at first, but it makes sense once you explore the pattern that emerges at most organizations. Teams typically try to get started with continuous testing by automating some UI tests and integrating their execution into the continuous integration process. They achieve and celebrate small victories, but the process doesn’t expand. In fact, it decays. Why? It usually boils down to roadblocks that fall into the following three categories:
- Time and resources
Time and Resources
Teams severely underestimate the time and resources required for sustainable test automation. Yes, getting some basic UI tests to run automatically is a great start. However, you also need to plan for the time and resources required to:
- Keep notoriously brittle tests scripts from overwhelming the team with false positives.
- Create tests for every new/modified requirement (or determine where to focus your efforts and what you can skip).
- Establish a test framework that supports reuse and data-driven testing — both of which are essential for making automation sustainable over the long term.
- Keep individual tests and the broader test framework in sync with the constantly-evolving application.
- Execute the test suite — especially if you’re trying to frequently run a large, UI-heavy test suite.
- Determine how to automate more advanced use cases and keep them running consistently in a continuous testing environment (see the next section for more on this).
- Review and interpret test the mounting volume of test results (more on this later, too).
With Agile and DevOps, time for test creation, maintenance, execution, and analysis is extremely limited — but fast feedback is essential. How can you ensure that the most important things are sufficiently tested without delaying time to market?
It’s one thing to automate a test for a simple “create” action in a web application (e.g. create a new account and complete a simple transaction from scratch). It’s another to automate the most business-critical transactions, which typically pass through multiple technologies (SAP, APIs, mobile interfaces, and even mainframes) and require sophisticated setup and orchestration. You need to ensure that:
- Your testing resources understand how to automate tests across all the different technologies and connect data and results from one technology to another.
- You have the stateful, secure, and compliant test data required to set up a realistic test as well as drive the test through a complex series of steps — each and every time the test is executed.
- You have reliable, continuous, and cost-effective access to all the dependent systems that are required for your tests — including APIs, third-party applications, etc. that may be unstable, evolving, or accessible only at limited times.
Moreover, you also need a systematic way to flush out the critical defects that can only be found with a human evaluating application from an end-user perspective. Automation is great at rapidly and repeatedly checking whether certain actions continue to produce the expected results, but it can’t uncover the complex usability issues that significantly impact the end user experience.
Without fast, reliable feedback on how application changes impact the core end user experience, how do you know if a release will help the business or harm it?
The most commonly cited complaint with test results is the overwhelming number of false positives that need to be reviewed and addressed. When you’re just starting off with test automation, it might be feasible to handle the false positives. However, as your test suite grows and your test frequency increases, addressing false positives quickly becomes an insurmountable task. Ultimately, many teams either start ignoring the false positives (which erodes trust in the test results and continuous testing initiative) or giving up on test automation altogether.
When DevOps and continuous delivery initiatives come into play, another critical issue with results emerges: they don’t provide the risk-based insight needed to make a fast go/go-no decision. If you’ve ever looked at test results, you’ve probably seen something like this:
What does this really tell you? You can see that…
- There’s a total of 53,274 test cases.
- Almost 80% of those tests passed.
- Over 19% of them failed.
- About 1% did not execute.
But would you be willing to make a release decision based on these results? Maybe the test failures are related to some trivial functionality. Maybe they are the most critical functionality, the “engine” of your system. Or maybe your most critical functionality was not even thoroughly tested. Trying to track down this information would require tons of manual investigative work that yields delayed, often-inaccurate answers.
In the era of Agile and DevOps, release decisions need to be made rapidly — even automatically and instantaneously. Test results that focus on the number of test cases leave you with a huge blind spot that becomes absolutely critical, and incredibly dangerous, when you’re moving at the speed of Agile and DevOps.
If your results don’t indicate how much of your business-critical functionality is tested and working, you can’t rely on them to drive automated delivery pipelines. Manual review and assessment will be required…and that’s inevitably going to delay each and every delivery.