When Does Continuous Testing Happen?

The following post is adapted from the e-book A CTO’s Guide to Continuous Testing.

Continuous testing is the principle of testing your software as often as possible, from early on in development and continuing through release. Generally, continuous testing involves everything from unit tests and automated testing suites to manual tests run by humans.

Implementing a continuous testing workflow that fits into every stage of your development processes helps your team save time and resources. If you’re curious about what continuous testing really means and are wondering how it looks in action, here’s Rainforest CTO Russell Smith’s introduction to what continuous testing means at different stages of production.

Continuous testing starts from the beginning of development and continues through production. The aim of continuous testing is to ship your code faster and give your customer high-quality releases. We advocate “shifting left,” or moving testing earlier in the development process. The more you can “shift left” with continuous testing, the earlier you’ll find bugs and the faster you’ll be able to get your product to market.

Continuous Testing in Development

Testing should start when the code is still in a developer’s hands, before it has been merged into the main code base. Unit testing is the primary focus of continuous testing in development. Some companies are opting for test-driven development (TDD) strategies that put test writing ahead of writing code. Either way, unit tests and some functional testing should start as early as possible to catch and resolve bugs quickly.

Continuous Testing in Staging

In staging, testing should take on a more holistic, customer-oriented focus. Your QA team or developers should focus on pushing your application to its limits to find new bugs, then write tests that catch them later. Staging is where exploratory and end-to-end testing are usually introduced, in addition to the functional and regression tests that started in development. Usability, integration, and acceptance testing take place in staging as well to make sure that the product meets standards of design and usability for your customers.

Continuous Testing in Production

By the time a feature reaches production, its test cases should start to become relatively stable. At this point, automation becomes a more viable option because automated tests are less likely to be brittle when the product is not changing frequently.

Smoke and sanity tests are another important part of testing in production. These “safety net” tests are often run manually, to ensure that the product or feature is functioning correctly. An engineer or QA person will usually check a release just after it’s done and run through a couple of core test flows. It is important that these smoke and sanity tests emulate the actual user experience, so they are not often automated. Because Rainforest leverages human testers to run tests, it can be an effective tool for executing smoke and sanity tests at scale.