Continuous Testing: What, Why, and How?

Agile, DevOps, and Continuous Delivery are part of every techie’s vocabulary today. We all want to be Agile, release software faster, and create innovative products like the Silicon Valley giants and startups. Many aspects of this transformation are very well understood, and to an extent successfully adopted. However, one area where there is still considerable confusion and adoption challenges is that of testing. When the software release cycle shrinks from years or months to weeks, days, or even faster; how do we reshape our testing practices to ensure the software released in production does not break and end up frustrating users, rather than delighting them?

Clearly, this is a common challenge, given that most DevOps shops still consider testing as the most frustrating bottleneck in software production.

Continuous Testing is the answer, but what exactly is Continuous Testing and how do you go about implementing it? Wikipedia defines Continuous Testing as “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” But this definition misses the essence, nature, and magnitude of the transformation that Continuous Testing signifies. While automation is certainly a big part, Continuous Testing is fundamentally about transforming testing from being a time-boxed event in a linear process to embedding it as a fundamental and ongoing aspect of every activity though the software delivery cycle.

The goal of Continuous Testing in Agile environments should be “in-sprint testing.” Regardless of whether your sprint is two weeks or four weeks, the goal should be to complete all types of testing within that sprint so that every sprint ends with a fully tested, ready-to-ship software. In fact, the very continuous delivery best practices show that you simply cannot do continuous delivery without continuous testing. And if you think that your sprint is too short to allow for comprehensive testing, most likely you are thinking about it wrong!

7-Steps to Adopt Continuous Testing

1. Define Tests Early, Even Before Writing Code!

Lack of clear requirements or incorrect interpretation thereof leads to rework and delays. Techniques such as Behavior-Driven Development (BDD), Acceptance Test-Driven Development (ATDD), and Model-Based Testing using tools like cucumber/gherkin, and CA Agile Requirements Designer (ARD) allow business stakeholders, product managers, developers and testers alike to ensure that requirements are clearly documented and communicated, test cases clearly defined, and test scripts created ahead of time to enable swift testing of code.

2. Optimize Tests and Coverage

Some organizations default to “running all the tests all the time” to ensure code coverage. This wastes resources and lengthens test cycles, without actually ensuring adequate code coverage. Save time, money and resources by testing only what you need to test. Visual models allow various paths to be explored and optimized such that only handful of tests provide maximum coverage. It is also possible to import test cases, remove duplicates and distribute and allocate optimized test cases in existing tools such as Rally, Jira, HP ALM, JIRA and more.

3. Shift-Left

To achieve “in-sprint” testing, shift testing left – so that tests can be run earlier in the development cycle. Developers test as they go; the CoE provides expertise, specialized systems and services. Test automation spans UI, functional, performance, and security. Teams work together and are focused on business value delivered to the customer. This requires developer-friendly tools as well as a culture shift.

4. Provide Complete Test Environments

The ability to provide test environments is critical to achieving continuous testing. Eliminate blocks and reduce wait times by providing complete test environments on demand with dev-friendly tools (as code, CI/CD integrations, supported open source). These environments should include:

• Virtual services – provide robust simulation for unavailable, inaccessible, or still-in-development services. Developers and testers can continue to work in parallel by using the virtual service to simulate the response from the actual service.
• Test data on demand – helps ensure teams can perform comprehensive tests with production-like data.
• Ephemeral environments – ready on demand, decommissioned after use.

5. Get the Right Test Data

The inability to get robust test data causes significant delays in many application release cycles. To accurately test new functionality, test data should be as close as possible to what the application will encounter in production. If the test data lacks certain real-world characteristics (i.e. actual fields, data specifications, negative scenarios, etc.), the tests are unlikely to find many potential issues or break the application where there are weak points. Understanding data models in order to extract the right data is a special skill in and of itself. And although production data is clearly the most realistic data to use for testing, data privacy regulations often restrict its use. Below, we'll take a look at how CA Test Data Manager lets you copy production data and mask it to protect sensitive information, while maintaining the characteristics that make production data desirable for testing: realistic and referentially intact data across rows. Test data can also be generated synthetically from scratch using TDM when production data simply cannot be used.

6. Don’t Forget to Shift Right!

Shift right to use data from the dev cycle and from production to optimize test cycles, fine-tune tests, and build optimal regression suites. Shift right techniques include real and synthetic user monitoring, canary deployments, A/B testing, chaos engineering, etc. For example, by shifting right, you can identify which features are being used in production and ensure your regression tests cover those features. In the same way, you can release a new feature to a small group of users (internal or external), understand the impact of that feature in production, and adjust as needed.

Many of the nimblest companies like Facebook and Netflix rely heavily on shift-right testing. Gartner recently released a report on the shift-right testing, calling it a “must” for adopting DevOps practices.

7. Use Data and Metrics to Continuously Improve

Build cross-team collaboration and continuously improve with actionable analytics and feedback loops. Continuous delivery and continuous testing are journeys. Make sure all your teams have KPIs and access to data that will inform additional optimization. Don’t just collect data that is easy to collect, collect and report on the data that will provide actual insights to continuous improvement. Look to release orchestration applications to deliver this insight and a “single source of truth” for all teams across the application delivery lifecycle.

To learn more about continuous testing challenges and practices, check out the CT Academy Video Series.