Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

How to Automatically Skip Tests Based on Defects Status

DZone's Guide to

How to Automatically Skip Tests Based on Defects Status

With testNG, you can skip failing tests, like known and open issues, to save time and keep them out of the failed test list. See how in this tutorial.

· DevOps Zone ·
Free Resource

Do you need to strengthen the security of the mobile apps you build? Discover more than 50 secure mobile development coding practices to make your apps more secure.

I have been running thousands of automated test cases as part of our nightly regression testing. One of the challenges in running this is analyzing the test results next morning – particularly when you have a large set of regression test cases. Even though I have tried to keep the test flakiness as low as possible, there could be many genuine failures in the test results due to open defects. It is good that your automated regression suite works great by reporting them as issues. But you need to spend some time to find out if they are new issues or some existing issues in the issue tracking system.

To further simplify our work, we could add a feature in our test framework to automatically decide if the particular test should be executed or not.  That is what we are going to see in this article.

Assumptions

  1. I assume you use TestNG or a similar framework.
  2. Your issue tracking system has APIs enabled. For this article, I will be using GitHub as our issue tracker.

Sample testNG Test

For better understanding, let's create a sample testNG test first.

 public class UserRegistrationTest {

      @Test
      public void launchRegistrationPage() {
          RegistrationPage.launch();
          Assert.assertTrue(RegistrationPage.isAt());
      }

      @Test(dependsOnMethods= "launchRegistrationPage")
      public void enterDemographicInformation() {
          RegistrationPage.enterDemographicInformation();
          RegistrationPage.submit();
          Assert.assertTrue(ProductsListPage.isAt());
      }

      @Test(dependsOnMethods= "enterDemographicInformation")
      public void chooseProduct() {
          ProductsListPage.chooseProduct(Products.DUMMY);
          ProductsListPage.submit();
          Assert.assertTrue(PaymentInformationPage.isAt());
      }

      @Test(dependsOnMethods= "chooseProduct")
      public void enterPayment() {
          PaymentInformationPage.enterPayment(PaymentMethods.CC);
          PaymentInformationPage.submit();
          Assert.assertTrue(OrderConfirmationPage.isAt());
      }


      @Test(dependsOnMethods= "enterPayment")
      public void orderConfirmation() {
          Assert.assertTrue(OrderConfirmationPage.isOrderSuccessful());
          OrderConfirmationPage.close();
      } 
}

In the above test, let's assume that we have been facing some issues with the payment system – so all the tests are failing. As it is a known issue, we do not want them to appear in the failed tests list, as it increases the failed test count and wastes our time. So, we would like to skip the test as long as the issue is still open.

testng-issue-status-1

IssueTrackerUtil:

In order to enhance our testNG framework, I create the following.

public enum IssueStatus {
    OPEN,
    CLOSED;
}


public class IssueTrackerUtil {

    private static final String ISSUE_TRACKER_API_BASE_URL = "https://api.github.com/repos/username/dummy-project/issues/";

    public static IssueStatus getStatus(String issueID) {
        String githubIssueStatus = "CLOSED";
        try {
            githubIssueStatus = Unirest.get(ISSUE_TRACKER_API_BASE_URL.concat(issueID))
                                       .asJson()
                                       .getBody()
                                       .getObject()
                                       .getString("state")
                                       .toUpperCase();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return IssueStatus.valueOf(githubIssueStatus);
    }

}

I used the Unirest library to make a call to the REST API. To include in your Maven project:

<dependency>
    <groupId>com.mashape.unirest</groupId>
    <artifactId>unirest-java</artifactId>
    <version>1.4.9</version>
</dependency>


Issue Status Listener

First, I would like to mention the open defects in the in testNG tests, as shown here.

@Issue("APP-1234")
@Test(dependsOnMethods= "chooseProduct")
public void enterPayment() {
  PaymentInformationPage.enterPayment(PaymentMethods.CC);
  PaymentInformationPage.submit();
  Assert.assertTrue(OrderConfirmationPage.isAt());
}

So, I create an Issue annotation.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Issue {
    String value() default "";
}

Now we need to get the issue ID for each test before it gets executed, call the above IssueTrackerUtil to get the issue status, and skip the tests based on the status. Let's use testNG’s IInvokedMethodListener, as shown here.

public class MethodIssueStatusListener implements IInvokedMethodListener {

    @Override
    public void afterInvocation(IInvokedMethod invokedMethod, ITestResult result) {
        // nothing to do
    }

    @Override
    public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult result) {

        Issue issue = invokedMethod.getTestMethod()
                                   .getConstructorOrMethod()
                                   .getMethod()
                                   .getAnnotation(Issue.class);

        if (null != issue) {
            if (IssueStatus.OPEN.equals(IssueTrackerUtil.getStatus(issue.value()))) {
                throw new SkipException("Skipping this due to Open Defect - " + issue.value());
            }
        }
    }

}

The rest is simple. Add the above listener in the testNG test class (usually in the base class or suite XML file) and mention the issue ID if any of the tests associated with the particular issue.

@Listener(MethodIssueStatusListener.class)
public class UserRegistrationTest {

      @Test
      public void launchRegistrationPage() {
          RegistrationPage.launch();
          Assert.assertTrue(RegistrationPage.isAt());
      }

      @Test(dependsOnMethods= "launchRegistrationPage")
      public void enterDemographicInformation() {
          RegistrationPage.enterDemographicInformation();
          RegistrationPage.submit();
          Assert.assertTrue(ProductsListPage.isAt());
      }

      @Test(dependsOnMethods= "enterDemographicInformation")
      public void chooseProduct() {
          ProductsListPage.chooseProduct(Products.DUMMY);
          ProductsListPage.submit();
          Assert.assertTrue(PaymentInformationPage.isAt());
      }

      @Issue("APP-1234")
      @Test(dependsOnMethods= "chooseProduct")
      public void enterPayment() {
          PaymentInformationPage.enterPayment(PaymentMethods.CC);
          PaymentInformationPage.submit();
          Assert.assertTrue(OrderConfirmationPage.isAt());
      }

      @Test(dependsOnMethods= "enterPayment")
      public void orderConfirmation() {
          Assert.assertTrue(OrderConfirmationPage.isOrderSuccessful());
          OrderConfirmationPage.close();
      } 
}

When we execute the above test, we get the results as shown here.

testng-issue-status

Updating Issue Status Automatically

Lets further enhance this!

Instead of skipping the tests, let's execute it every time.

  • If it fails, change the result to SKIPPED (because we know that it might fail. We just ran to see if there is any luck!).
  • If it passes, close the defect automatically.
public class IssueTrackerUtil {

    private static final String ISSUE_TRACKER_API_BASE_URL = "https://api.github.com/repos/username/dummy-project/issues/";
    private static final String ISSUE_TRACKER_USERNAME = "username";
    private static final String ISSUE_TRACKER_PASSWORD = "password";

    public static IssueStatus getStatus(String issueID) {
        String githubIssueStatus = "CLOSED";
        try {
            githubIssueStatus = Unirest.get(ISSUE_TRACKER_API_BASE_URL.concat(issueID))
                                       .asJson()
                                       .getBody()
                                       .getObject()
                                       .getString("state")
                                       .toUpperCase();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return IssueStatus.valueOf(githubIssueStatus);
    }

    public static void updateStatus(String issueID, IssueStatus status) {
        try {

            Unirest.post(ISSUE_TRACKER_API_BASE_URL.concat(issueID).concat("/comments"))
                    .header("accept", ContentType.APPLICATION_JSON.toString())
                    .basicAuth(ISSUE_TRACKER_USERNAME, ISSUE_TRACKER_PASSWORD)
                    .body("{ \"body\" : \" testng method passed now. closing automatically.\" }")
                    .asJson();

            Unirest.patch(ISSUE_TRACKER_API_BASE_URL.concat(issueID))
                    .header("accept", ContentType.APPLICATION_JSON.toString())
                    .basicAuth(ISSUE_TRACKER_USERNAME, ISSUE_TRACKER_PASSWORD)
                    .body("{ \"state\" : \""+ status.toString().toLowerCase() + "\" }")
                    .asJson();

        } catch (UnirestException e) {
            e.printStackTrace();
        }
    }

}

MethodIssueStatusListener is updated to take action after method invocation, as shown here.

public class MethodIssueStatusListener implements IInvokedMethodListener {

    @Override
    public void afterInvocation(IInvokedMethod invokedMethod, ITestResult result) {

        Issue issue = invokedMethod.getTestMethod()
                                    .getConstructorOrMethod()
                                    .getMethod()
                                    .getAnnotation(Issue.class);

        if (null != issue) {
            if(IssueStatus.OPEN.equals(IssueTrackerUtil.getStatus(issue.value()))){
                switch(result.getStatus()){
                    case ITestResult.FAILURE:
                            // no need to fail as we might have expected this already.
                            result.setStatus(ITestResult.SKIP);  
                            break;

                    case ITestResult.SUCCESS:
                            // It is a good news. We should close this issue.
                            IssueTrackerUtil.updateStatus(issue.value(), IssueStatus.CLOSED);
                            break;
                }
            }
        }
    }

    @Override
    public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult result) {
        // nothing to do
    }

}

That’s it! When I execute my testNG tests, based on the status of the tests, it updates the issue status accordingly.

new-issue

new-issue-auto-close

Summary

By using the GitHub API and testNG method listener, we are able to improve our existing framework to skip tests based on the status of the issue or update issue status based on the test results.

It saves our time by skipping the known issues. You could modify IssueTrackerUtil based on the issue tracking system you use.

Check out tips for blazing the way from agile to DevSecOps with security built into your mobile app toolchain.

Topics:
testng ,issue tracker ,test automation ,software testing ,acceptance testing ,devops ,tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}