How the Automation Fallback Method Can Help Your Test Automation Strategy

Automated tests are a great tool for QA teams and can help speed up testing significantly. But test automation has its own costs that, if not addressed properly, can create drag for fast-moving teams. To allow teams to get the benefits of test automation, they need to take a more dynamic approach to their test execution and management workflow.

The automation fallback method leverages crowdsourced testing to support automated test suites, helping teams to reduce the time and resources required for continuous test coverage. In this article, we'll take a look at how automation fallback works, and why QA teams can benefit from incorporating it into their testing strategy.

The Impact of Test Automation on Release Cycles

We've talked about the long-term costs of test automation before, from the challenge of hiring out a skilled team to the ongoing cost of managing brittle test suites. But your automated tests can also impact productivity every time you prepare to ship new code to production.

Automated tests are inherently brittle and flaky, and require frequent maintenance to stay aligned with code as it changes. Every automated test failure requires an assessment as to whether the test case is broken, or if there truly is a bug. If the test case is broken, it must be refactored before it can provide valuable quality feedback. All of these activities require time and skilled resources. When a failure happens right before a scheduled release, teams may try to push ahead without the appropriate coverage - risking shipping bugs to customers - or delaying the release - reducing their competitive advantage in the market.

How to Use the Automation Fallback Method

To use the automation fallback method, every automated test case should have a corresponding crowd-executed manual test case. If the automated test returns a failure, then the crowdsourced test kicks off automatically. If the manual test fails as well, then the issue is likely a real bug. If the manual tests pass, then the issue may be a broken automation test.

The Benefits of Automation Fallback

The additional context given by the secondary test case kickstarts the triage process and helps the team understand where they should dig deeper. Teams can then recover time from triaging issues to be used for fixing broken tests or working with developers to identify real bugs.

Additionally, automation fallback helps mitigate the impact that broken automated tests can have on deployment timelines. When teams rely on automated tests for confidence in the quality of their code before launch, broken tests leave a potentially dangerous gap in coverage. Automation fallback decouples refactoring automated test cases from overall QA. Even if automated test cases cannot immediately be updated, the organization does not lose the confidence in quality they need to ship code.

Automation Fallback in Action: CareCloud

Rainforest customer CareCloud is a prime example of the "automation fallback" method in action. CareCloud uses Rainforest to speed up execution of many of their manual regression tests, and to help their automation team determine how to script automated tests.

Each automated test is based on a corresponding Rainforest test. Because of this, CareCloud's team is able to automatically kick off the Rainforest test if the automated test fails. This process ensures that the test automation team has as much insight into what went wrong as possible, before they spend any time triaging the failure. "Instead of having someone to manually go in and triage what's failing, using Rainforest helps us save about an hour every day for the automation team," says Zachory Strike, who leads the test automation team at CareCloud.

Learn more about how CareCloud uses Rainforest by reading their case study, or our interview with QA Engineer Ryan von Rosenberg on What QA Teams Should Know About Rainforest.

Finding Better Ways to Test Software

Many teams treat test automation as the final stage of a test's lifecycle, but in reality, test automation is more effective when it's part of a dynamic ecosystem of test cases. By using the automation fallback, teams can leverage automation for faster testing without letting broken tests pile up and create drag on development goals.

How does your team handle brittle or flaky automated test cases? We'd love to hear your approach to keeping your test suite healthy and effective. Tweet @rainforestqa to share your strategy!