Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Test Automation: Reduce Time to Analyze and Report Failures

DZone's Guide to

Test Automation: Reduce Time to Analyze and Report Failures

This article provides some quick tips on reducing the time your automated tests take and making them overall more efficient.

· DevOps Zone ·
Free Resource

Is the concept of adopting a continuous everything model a daunting task for your fast moving business? Read this whitepaper to break down and understand one of the key pillars of this model in Continuous Governance: The Guardrails for Continuous Everything.

There are many basics expectations from an automation testing framework, like code reusability, human-readable report generation, and easy maintenance. These expectations are widely discussed in every forum, blog, and meetup with few exceptions.

However, once we create a framework that meets all expectations and start using it in real time, there will be many other problems and requirements to emerge. These new problems may not be a must-have feature, but having those features will ensure the success of test automation.

One such problem we recently encountered is that in multiple cycles of regression testing, the same test cases failed again and again due to the same bug, and automation developers had to validate these scripts multiple times. This portrayed a picture that the number of failures were not reducing in each regression cycle; developers had to debug the failure only to find out that it failed due to an existing bug, while increasing the time to analyze and report test results.

When the number of automated tests and failures are fewer, this may not look like a problem. Once the number of automated tests increases - assume that we have 2,000 - this will cause a problem in analyzing and reporting the bugs.

What Can We Do to Improve?

We can integrate the automation framework with the bug management system that we are using and tag a bug ID to each automation test case, if any. Tagging can be done using many different mechanisms, like

  • Creating groups with a bug ID and exclude or include groups during execution after validating the bug status, in the case of TestNG.
  • Simply keep test cases to a bug ID map in a Properties, XML, or JSON file. Before execution of a test, enable the framework to get bug IDs from this mapping file, if any, for the test case being executed, and validate the bug status; if the bug is Reopened/Open/New, skip the execution; if the bug is Resolved/Verified/Closed, proceed with execution.
  • We can even use custom annotations.

By doing this, we are not executing those test cases that we know will fail when executed. The results will be more green, easy to analyze for new failures, and time to report issues to respective stakeholders will be reduced.

Are you looking for greater insight into your software development value stream? Check out this whitepaper: DevOps Performance: The Importance of Measuring Throughput and Stability to see how CloudBees DevOptics can give you the visibility to improve your continuous delivery process.

Topics:
test automation ,regression testing ,software testing ,devops

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}