Over a million developers have joined DZone.
Refcard #246

Continuous Testing

Transforming Testing for Agile and DevOps

By introducing continuous testing to your software development pipeline, you can release business-differentiating software more efficiently, get important feedback sooner, and control business risk. In this Refcard, you’ll learn about what continuous testing really involves, why continuous testing is crucial for effective Agile and DevOps teams, and what the five-level path to continuous testing looks like.


Brought to you by

Free .PDF for easy Reference

Written by

Wayne Ariola CMO, Tricentis
Refcard #246

Continuous Testing

Transforming Testing for Agile and DevOps

By introducing continuous testing to your software development pipeline, you can release business-differentiating software more efficiently, get important feedback sooner, and control business risk. In this Refcard, you’ll learn about what continuous testing really involves, why continuous testing is crucial for effective Agile and DevOps teams, and what the five-level path to continuous testing looks like.

Free .PDF for easy Reference

Written by

Wayne Ariola CMO, Tricentis

Brought to you by

Table of Contents

Continuous Testing vs. Test Automation

Traditional Testing Doesn’t Work

What Is Continuous Testing?

Comparing Continuous Testing With Test Automation

Continuous Testing and Agile + DevOps Success

Key Findings

Business Risk Must Be Accurately and Continuously Managed

Key Recommendations

Ruthlessly Automate Your Testing and QA Processes, End-to-End

Prioritize Test Execution Predicated Around Business Risk

Additional Research Findings

The Top Continuous Testing Roadblocks

The Path to Continuous Testing


Section 1

Continuous Testing vs. Test Automation

Like Lucy and Ethel struggling to keep pace at the chocolate factory, many software testers have been scrambling to keep pace with accelerated processes — then along comes the supervisor proclaiming, “You’re doing splendidly! Speed it up!”

As expectations associated with testing are changing, legacy testing platforms aren’t keeping up. Legacy testing platforms take a “heavy” approach to testing. They rely on brittle scripts, deliver slow end-to-end regression test execution, and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18%, on average (8% for enterprises). In a polling question asked at industry webinars and trade shows, respondents overwhelming reported that the results of test automation to date have been “so-so.”

Section 2

Traditional Testing Doesn’t Work

Recent changes across the industry are demanding more from testing while making test automation even more difficult to achieve. There are several reasons for this:

  • Application architectures are increasingly more distributed and complex, embracing cloud, APIs, microservices, etc. and creating virtually endless combinations of different protocols and technologies within a single business transaction.
  • Thanks to Agile, DevOps, and continuous delivery, many applications are now released anywhere from every two weeks to thousands of time a day. As a result, the time available for test design, maintenance, and especially execution decreases dramatically.
  • Now that software is the primary interface to the business, an application failure is a business failure. This is true even for a seemingly minor glitch that could have severe repercussions if it impacts the user experience. As a result, application-related risks have become a primary concern for even non-technical business leaders.

Given that software testers are facing increasingly complex applications, they are expected to deliver trustworthy go/no-go decisions at the new speed of modern business. More of the same traditional testing approaches won’t get us there. We need to transform the testing process as deliberately and markedly as we’ve transformed the development process. This requires more effective test automation. And it requires a different approach called continuous testing.

Section 3

What Is Continuous Testing?

As mentioned previously, continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release as rapidly as possible. Continuous testing does not require any specific type of testing approach (e.g. shift left, shift right…) or testing tools. However, it does require that:

  • Actionable feedback is delivered to the right stakeholder at the right time.
  • Testing occurs across all phases of the software delivery pipeline.

Test automation is designed to produce a set of pass/fail data points correlated to user stories or application requirements. Continuous testing, on the other hand, focuses on business risk and providing insight on whether the software can be released. To achieve this shift, we need to stop asking, “Are we done testing?” and instead concentrate on, “Does the release candidate have an acceptable level of business risk?”

There are five key attributes to continuous testing:

  1. Assesses business risk coverage as its primary goal.
  2. Establishes a safety net that helps the team protect the user experience.
  3. Requires a stable test environment to be available on demand.
  4. Seamlessly integrates into the software delivery pipeline and DevOps toolchain.
  5. Delivers actionable feedback appropriate for each stage of the delivery pipeline.
Section 4

Comparing Continuous Testing With Test Automation

The main differences between continuous testing and test automation can be grouped into three broad categories: risk, breadth, and time.


Businesses today have not only exposed many of their internal applications to the end user; they also have developed vast amounts of additional software that extends and complements those applications. For example, airlines have gone far beyond exposing their once-internal booking systems. They now let customers plan and book complete vacations, including hotels, rental cars, and activities. Exposing more and more innovative functionality to the user is now a competitive differentiator — but it also increases the number, variety, and complexity of potential failure points.

Large-scale “software fails” have such severe business repercussions that application-related risks are now prominent components of a business’ public financial filing. Given that notable software failures resulted in an average -4.06 percent decline in stock price (which equates to an average of negative $2.55 billion loss of market capitalization), it’s not surprising that business leaders are taking note — and expecting IT leaders to take action.

If your test cases weren’t built with business risk in mind, your test results won’t provide the insight needed to assess risks. Most tests are designed to provide low-level details on whether user stories are correctly implementing the requirements — not high-level assessments of whether a release candidate is too risky to release. Would you automatically stop a release from taking place based on test results? If not, your tests aren’t properly aligned with business risks.

To be clear: We’re not suggesting that low-granularity tests aren’t valuable; we’re stating that more is needed to stop high-risk candidates from going out into the wild.

Here are some ways that testers can address risk:

  • Understand the risks associated with the complete application portfolio.
  • Map risks to application components and requirements (which are then mapped to tests).
  • Use a test suite that achieves the highest possible risk coverage with the least amount of test cases.
  • Always report status that shows risk exposure from business, technical, performance, and compliance perspectives.


Even if a business manages to steer clear of large-scale software fails that make the headlines, even seemingly minor glitches can still cause trouble these days. If any part of the user experience fails to meet expectations, you run the risk of losing that customer to a competitor. You also risk brand damage if that user decides to expose issues to social media.

Just knowing that a unit test failed or a UI test passed doesn’t tell you whether the overall user experience is impacted by recent application changes. To protect the end-user experience, run tests that are broad enough to detect when an application change inadvertently impacts functionality which users have come to rely on.

Here are some ways to address testing breadth:

  • Define and execute complete end-to-end tests that exercise the application from the user’s perspective.
  • Provide integrated support for all the technologies involved in critical user transaction (web, mobile, message/API-layer, SAP and packaged apps, etc.).
  • Simulate service virtualization for dependent components required to exercise complete end-to-end transactions that are neither available nor configurable for repeated testing.
  • Ensure that tests and service virtualization assets are populated with realistic and valid data each and every time the tests are executed.
  • Performing exploratory testing to find user experience issues that are beyond the scope of automated testing (e.g. usability issues).


Now that the speed at which organizations ship software has become a competitive differentiator, the vast majority of organizations are turning to Agile and DevOps to accelerate their delivery processes.

When automated testing emerged, it focused on testing internal systems that were built and updated according to waterfall development processes. Systems were all under the organization’s control, and everything was completed and ready for testing by the time the testing phase was ready to start. Now that Agile processes are becoming the norm, testing must begin in parallel with development; otherwise, the user story is unlikely to be tested and deemed “done” within the extremely compressed iteration time frame (often two weeks).

If your organization has adopted DevOps and is performing continuous delivery, software may be released hourly — or even more frequently. In this case, feedback at each stage of the process can’t just be “fast” — it must be nearly instantaneous. If quality is not a top concern for your application (e.g. if there are minimal repercussions to doing a rollback when defects are discovered in production), running some quick unit tests and smoke tests on each release might suffice. However, if the business wants to minimize the risk of faulty software reaching an end user, you need some way to achieve the necessary level of risk coverage and testing breadth — fast.

For testing, there are several significant impacts:

  • Testing must become integral to the development process (rather than a “hygiene task” tacked on when development is complete).
  • Tests must be ready to run almost as soon as the related functionality is implemented.
  • The organization must have a way to determine the right tests to execute at different stages of the delivery pipeline (smoke testing upon check-in, API/message layer testing after integration, and end-to-end testing at the system level).
  • Each set of tests must execute fast enough that it does not create a bottleneck at the associated stage of the software delivery pipeline.
  • A way to stabilize the test environment is needed to prevent frequent changes from causing an overwhelming number of false positives.

Here are some ways for testers need to address time pressures:

  • Identify which test cases are critical for addressing top business risks.
  • Define and evolve tests as the application constantly changes.
  • Rebalance the test pyramid so that most tests execute at the API layer, which is at least 100 times faster than UI test execution.
  • Integrate tests into the delivery pipeline.
  • Run distributed tests across multiple VMs, network computers, or in the cloud as appropriate.
  • Enlist service virtualization and synthetic data generation/TDM so that testing doesn’t need to wait on data or environment provisioning.
Section 5

Continuous Testing and Agile + DevOps Success

New Forrester research (The Definitive Software Quality Metrics for Agile + DevOps) reveals that continuous testing, done properly, is a key differentiator between DevOps and Agile leaders and DevOps and Agile laggards.

Delivering high-quality software fast is no longer a luxury — it is a prerequisite for digital success and mandatory for customer loyalty and acquisition. We are in an era in which customers are empowered to choose and switch to the company that can provide them the best experience quickly and at minimal cost. Organizations that cannot keep up with their customers’ expectations and adapt to new business paradigms will find themselves losing customers and business to competitors that can.

How can firms deliver with quality at speed? Implementing Agile and DevOps best practices and building in quality is the first step — and this is not trivial. Automation is the lever that development teams pull to go faster. However, if not properly managed, automation can jeopardize quality and increase risk. To ensure that quality issues and unacceptable risk do not negate the velocity benefits of automation, firms must ensure they are tracking the right metrics to measure quality throughout development testing and deployment.

Tricentis commissioned Forrester Consulting to evaluate current software development and delivery priorities as well as key metrics tracked throughout the software development life cycle. To explore this topic, Forrester conducted an online survey with 603 global enterprise respondents responsible for their firms’ Agile and/or DevOps strategies.

Section 6

Key Findings

Five Core Practices Separate Successful DevOps Leaders From Laggards

Firms that are more mature in Agile and DevOps do five key things differently:

  1. Allocate proper testing budgets and focus on upgrading their testing skills.
  2. Implement continuous testing to meet the demands of release frequency and support continuous delivery.
  3. Include testers as part of their integrated delivery teams.
  4. Automate end-to-end functional testing.
  5. Shift testing left to be earlier in the development lifecycle.

For firms that follow these five best practices, testing is a way to gain speed and efficiency rather than be a bottleneck to delivering faster.

While most Agile and/or DevOps firms have implemented some of these five core testing practices, few have actually implemented all of them — only about a quarter of respondents say they somewhat or completely agree with all five best practice statements. This is an important distinction. While many CXOs and high-level decision-makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers — and their behaviors and attitudes support that assertion.

Section 7

Business Risk Must Be Accurately and Continuously Managed

Automating software delivery without being able to accurately measure software quality is dangerous. Firms must understand the business risk — the chance of a negative user experience — that each software application carries when deciding whether to release. Unfortunately, most firms today admit they have gaps in their ability to accurately assess and manage business risk in testing.

Leaders Ruthlessly Drive End-to-End Test Automation

With a better indication of risk from their testing metrics, DevOps leaders can relentlessly pursue automation to develop with quality at speed. They do this by prioritizing end-to-end automated business process test cases. They automate test design and test execution, and they orchestrate the automation of the overall dev-test-deploy process.

Section 8

Key Recommendations

Realistically assess and up-level your continuous testing practices for Agile + DevOps. While many CXOs believe their firms are ahead of the curve on DevOps, our research shows that only about a quarter of firms are following testing best practices. Make sure your firm is implementing continuous testing as part of its DevOps strategy.

Make Business Risk the Driver for Your Metrics Program

Most firms acknowledge gaps in their ability to manage business risk with the right metrics but are nevertheless optimistic on risk coverage. This is dangerous for extreme automation — and exacerbated as delivery velocity and volume increase. First, you must clearly define risk. Second, you must continuously measure your exposure versus that risk. Next, use that understanding to promote releases. By focusing test automation efforts on processes and transactions that that carry higher business risk, you reduce the chance that critical defects slip into production.

Section 9

Ruthlessly Automate Your Testing and QA Processes, End-to-End

If your goal is to deliver high-quality software faster (and it should be), then you need to automate your software development pipeline. More mature Agile + DevOps firms understand that automation is critical to driving release velocity, and they are reaping the benefits. Automating end-to-end testing is a key step in implementing continuous testing and a top priority for Agile + DevOps leaders today.

Section 10

Prioritize Test Execution Predicated Around Business Risk

Understanding which tests deliver the greatest risk coverage is a significant advantage when speed matters. The ability to prioritize test execution depends on the implementation of a model that is collaboratively developed among the development, testing, and business stakeholders.

Section 11

Additional Research Findings

For more research findings, see the infographic or read the complete Definitive Software Quality Metrics for Agile + DevOps report.

Section 12

The Top Continuous Testing Roadblocks

As the preceding research noted, the continuous testing adoption rate is extremely low (26%), even among organizations actively practicing Agile and DevOps. This seems shocking at first, but it makes sense once you explore the pattern that emerges at most organizations. Teams typically try to get started with continuous testing by automating some UI tests and integrating their execution into the continuous integration process. They achieve and celebrate small victories, but the process doesn’t expand. In fact, it decays. Why? It usually boils down to roadblocks that fall into the following three categories:

  1. Time and resources
  2. Complexity
  3. Results

Time and Resources

Teams severely underestimate the time and resources required for sustainable test automation. Yes, getting some basic UI tests to run automatically is a great start. However, you also need to plan for the time and resources required to:

  • Keep notoriously brittle tests scripts from overwhelming the team with false positives.
  • Create tests for every new/modified requirement (or determine where to focus your efforts and what you can skip).
  • Establish a test framework that supports reuse and data-driven testing — both of which are essential for making automation sustainable over the long term.
  • Keep individual tests and the broader test framework in sync with the constantly-evolving application.
  • Execute the test suite — especially if you’re trying to frequently run a large, UI-heavy test suite.
  • Determine how to automate more advanced use cases and keep them running consistently in a continuous testing environment (see the next section for more on this).
  • Review and interpret test the mounting volume of test results (more on this later, too).

With Agile and DevOps, time for test creation, maintenance, execution, and analysis is extremely limited — but fast feedback is essential. How can you ensure that the most important things are sufficiently tested without delaying time to market?


It’s one thing to automate a test for a simple “create” action in a web application (e.g. create a new account and complete a simple transaction from scratch). It’s another to automate the most business-critical transactions, which typically pass through multiple technologies (SAP, APIs, mobile interfaces, and even mainframes) and require sophisticated setup and orchestration. You need to ensure that:

  • Your testing resources understand how to automate tests across all the different technologies and connect data and results from one technology to another.
  • You have the stateful, secure, and compliant test data required to set up a realistic test as well as drive the test through a complex series of steps — each and every time the test is executed.
  • You have reliable, continuous, and cost-effective access to all the dependent systems that are required for your tests — including APIs, third-party applications, etc. that may be unstable, evolving, or accessible only at limited times.

Moreover, you also need a systematic way to flush out the critical defects that can only be found with a human evaluating application from an end-user perspective. Automation is great at rapidly and repeatedly checking whether certain actions continue to produce the expected results, but it can’t uncover the complex usability issues that significantly impact the end user experience.

Without fast, reliable feedback on how application changes impact the core end user experience, how do you know if a release will help the business or harm it?


The most commonly cited complaint with test results is the overwhelming number of false positives that need to be reviewed and addressed. When you’re just starting off with test automation, it might be feasible to handle the false positives. However, as your test suite grows and your test frequency increases, addressing false positives quickly becomes an insurmountable task. Ultimately, many teams either start ignoring the false positives (which erodes trust in the test results and continuous testing initiative) or giving up on test automation altogether.

When DevOps and continuous delivery initiatives come into play, another critical issue with results emerges: they don’t provide the risk-based insight needed to make a fast go/go-no decision. If you’ve ever looked at test results, you’ve probably seen something like this:

Image title

What does this really tell you? You can see that…

  • There’s a total of 53,274 test cases.
  • Almost 80% of those tests passed.
  • Over 19% of them failed.
  • About 1% did not execute.

But would you be willing to make a release decision based on these results? Maybe the test failures are related to some trivial functionality. Maybe they are the most critical functionality, the “engine” of your system. Or maybe your most critical functionality was not even thoroughly tested. Trying to track down this information would require tons of manual investigative work that yields delayed, often-inaccurate answers.

In the era of Agile and DevOps, release decisions need to be made rapidly — even automatically and instantaneously. Test results that focus on the number of test cases leave you with a huge blind spot that becomes absolutely critical, and incredibly dangerous, when you’re moving at the speed of Agile and DevOps.

If your results don’t indicate how much of your business-critical functionality is tested and working, you can’t rely on them to drive automated delivery pipelines. Manual review and assessment will be required…and that’s inevitably going to delay each and every delivery.

Section 13

The Path to Continuous Testing

Based on our experience guiding enterprise testing teams to steer clear of the common roadblocks and achieve the mature continuous testing critical for DevOps and Agile success, Tricentis developed a Continuous Testing Maturity Model. We’ve found that this is the most efficient path to rolling out continuous testing in a way that’s sustainable for the team — and valuable to leaders aiming to accelerate delivery without incurring unacceptable business risk.

You can use this model to assess where you stand today and understand what’s needed to progress from one level to the next.

Image title

Level 1: The Typical Starting Point

At this initial level, the key metric is the number of test cases. All test cases are designed based on tester intuition. Testing is performed manually or partially automated with a script-based approach (which results in a high rate of false positives that require constant maintenance). Testers must manually ensure test data suitability ( e.g. by localizing and refining test data) and wait for dependencies to be provisioned in test environments. Any API testing is the domain of developers.

Level 2: Aligned

A risk assessment has been completed and risk coverage is now the key metric of test case definition and execution. Test automation still focuses on the UI, but now uses model-based test automation (MBTA), which significantly reduces false positive rates and maintenance efforts. Since there is still no comprehensive test data management in place, automation primarily focuses on new data object creation rather than complex administrative use cases.

Level 3: Managed

Session-based exploratory testing is introduced to expose risks that specification-based testing cannot find (e.g. in functionality implemented beyond the boundaries of the specification). Additional test cases are defined via combinatorial test case design methodologies such as linear expansion. If functionality is exposed via APIs, API testing is introduced at the tester level. MBTA-driven UI testing is extended in areas where API testing is not applicable or effective. Test automation is introduced into continuous integration through initial integrations with build and deployment tools.

Level 4: Mature

Test data management now provides the test data needed to enable continuous, consistent test automation. Service virtualization ensures that testing can proceed even if dependent components are unstable or unavailable. The introduction of both TDM and service virtualization enables more sophisticated API testing, end-to-end testing, and continuous test execution. Tests can now be executed continuously as part of the software delivery pipeline — providing instant feedback on the business risk associated with the software release candidate.

Level 5: Optimized

Comprehensive test automation has been established and is supported by sophisticated, stateful service virtualization and test data generation/provisioning. Metrics are in place to monitor and continuously improve the effectiveness of the software testing process. Continuous testing is fully integrated into continuous integration and the continuous delivery pipeline. The transformation into “DevTestOps” via process, people, and product is achieved.

Section 14


As software becomes the key to creating a competitive advantage across all markets, enterprises no longer enjoy the luxury of selecting either “speed” or “quality” when delivering software. Both are critical. Now that agile practices have matured and DevOps initiatives have entered the corporate agenda, continuous integration, continuous testing, and continuous delivery have emerged as key catalysts for enabling quality at speed. Of the three, continuous testing is by far the most challenging. 

While continuous integration is primarily a tool-driven activity and continuous delivery is a tool- and team-driven activity, continuous testing involves tools, teams, individuals, and services. Building and integrating code changes is certainly important. However, if the automated delivery process cannot identify how changes impact business risk or disrupt the end-user experience, then the increased frequency and speed of continuous integration and continuous delivery could become more of a liability than an asset.

Executed correctly, continuous testing serves as the centerpiece of the Agile downstream process — executing automated tests as part of the software delivery pipeline in order to provide risk-based feedback as rapidly as possible. Mastering continuous testing is essential for controlling business risk given the increased complexity and pace of modern application delivery.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}