Continuous Testing Insights from Forrester’s Latest Wave

Forrester just released The Forrester Wave: Global Continuous Testing Service Providers, Q1 2019. The focus was on evaluating the top Continuous Testing Service Providers and rating how they stack up. However, you’ll want to scan this report even if you’re not in the market for Continuous Testing services.

Diego Lo Giudice continues his mission of helping IT leaders understand how quality processes must evolve for Agile and DevOps success, and what specific actions are needed to bridge the ever-widening gap between where we  are and where we need to be.

You can read the complete report here, courtesy of Accenture (one of the leaders named in the report). Here’s a quick summary of some of the key insights…

Test Data Management (TDM) is Gaining Traction

The results from the 2019 survey indicated a greater interest in test data management (TDM) than the previous (2017) survey. What’s behind this increased interest? The report explains, “End-to-end Continuous Testing requires self-service, fast-life-cycle updates of secured test data. A strong testing process won’t produce strong quality if you don’t feed it the right data at the right time.”

Performance Testing Needs to (Dramatically) Shift Left

Digital, mobile, and IoT are making early performance testing essential. Forrester recommends a much more dramatic “shift left” than most companies are prepared for, though: “Shift-left means running performance tests as early as possible during development to ensure designs and components work — many also call this ‘unit’ or ‘local’ performance testing.”

Functional Testing, TDM, and Left-Shifted Performance Testing Are the Most Impactful for Quality at Speed

Over two-thirds of respondents cited functional test automation (UI-, data-, or API-driven) as the most impactful automation service for achieving quality at speed. Here’s how all the best practices ranked (from most important to least):

1. Functional test automation
2. Test data management
3. Performance testing (left-shifted)
4. Service virtualization
5. TDD/BDD
6. Functional test design automation (model-based, AI-driven, or other)
7. Exploratory testing
8. Mobile testing
9. API test automation
10. Security testing
11. Cloud-based test environment provisioning
12. Testing in production

Test Automation Rates Remain Dismally Low (But Many Companies are Making Strides)

Forrester recommends test automation rates of over 80% to meet the demands of two-week Agile sprints. However, only 12% of surveyed reference clients achieved that level of test automation. One bright spot: more than half of the reference clients working with Continuous Testing service providers such as Accenture, Infosys, Sogeti, and Wipro now automate 50%+ of their functional testing.