{{announcement.body}}
{{announcement.title}}

How to Stop Test Suite After N Test Failures In Pytest

DZone 's Guide to

How to Stop Test Suite After N Test Failures In Pytest

At times, test scenarios are interdependent and one failure causes other failures. Avoid such cases in this pytest tutorial with maxfail and pytest.mark.incremental.

· Performance Zone ·
Free Resource

An exhaustive test-suite comprises many test cases that test different features on various combinations of browsers, platforms, and devices. Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing Selenium test automation.

The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).

In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.

Using @pytest.mark.incremental decorator to Stop Test Suite After N Test Failures

The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.

PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.

conftest.py (to skip test in Python with pytest)

Python
 




x
16


1
import pytest
2
from selenium import webdriver
3
import urllib3
4
import warnings
5
 
6
def pytest_runtest_makereport(item, call):
7
    if "incremental" in item.keywords:
8
        if call.excinfo is not None:
9
            parent = item.parent
10
            parent._previousfailed = item
11
 
12
def pytest_runtest_setup(item):
13
    previousfailed = getattr(item.parent, "_previousfailed", None)
14
    if previousfailed is not None:
15
        pytest.xfail("previous test failed (%s)" % previousfailed.name)
16
 
          


The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) to skip the test in Python with pytest.

How to Skip Test In Python with Pytest Using @pytest.mark.incremental

To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an automated browser testing example which contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.

Fixture with scope as the class is used to instantiate the Chrome browser instance. Implementation to skip tests in Python with a pytest is also included in conftest.py.

Conftest.py

Python
 




xxxxxxxxxx
1
24


 
1
import pytest
2
from selenium import webdriver
3
import urllib3
4
import warnings
5
 
6
def pytest_runtest_makereport(item, call):
7
    if "incremental" in item.keywords:
8
        if call.excinfo is not None:
9
            parent = item.parent
10
            parent._previousfailed = item
11
 
12
def pytest_runtest_setup(item):
13
    previousfailed = getattr(item.parent, "_previousfailed", None)
14
    if previousfailed is not None:
15
        pytest.xfail("previous test failed (%s)" % previousfailed.name)
16
        
17
@pytest.fixture(scope="class")
18
def driver_init(request):
19
    web_driver = webdriver.Chrome()
20
    request.cls.driver = web_driver
21
    yield
22
    web_driver.close()
23
    # web_driver.quit()
24
 
          


test_step.py

Python
 




xxxxxxxxxx
1
81


1
#Using @pytest.mark.incremental decorator to stop test suite after n test failure using Selenium test automation in Python with pytest 
2
import pytest
3
import pytest_html
4
from selenium import webdriver
5
from selenium.webdriver.chrome.options import Options
6
from selenium.webdriver.common.keys import Keys
7
import time
8
from time import sleep
9
import sys
10
import urllib3
11
import warnings
12
# @pytest.mark.incremental Stop Test Suite after N Test Failures in Pytest
13
@pytest.mark.usefixtures("driver_init")
14
@pytest.mark.incremental
15
class Test_Scenario_1:
16
    def test_1(self):
17
        self.driver.get('https://www.lambdatest.com/blog/')
18
        self.driver.maximize_window()
19
 
20
        expected_title = "LambdaTest | A Cross Browser Testing Blog"
21
        assert expected_title ==  self.driver.title
22
        time.sleep(5)
23
 
24
    def test_2(self):
25
        self.driver.get('https://www.google.com/')
26
        self.driver.maximize_window()
27
        title = "Google"
28
        assert title == self.driver.title
29
 
30
        search_text = "LambdaTest"
31
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
32
        search_box.send_keys(search_text)
33
 
34
        time.sleep(5)
35
        search_box.submit()
36
 
37
        time.sleep(5)
38
        
39
        # Click on the LambdaTest HomePage Link
40
        # This test will fail as the titles will not match
41
        title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
42
        lt_link = self.driver.find_element_by_xpath("//h3[.='LambdaTest: Cross Browser Testing Tools | Free Automated ...']")
43
        lt_link.click()
44
 
45
        time.sleep(10)
46
        assert title == self.driver.title   
47
        time.sleep(2)
48
 
49
    # As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
50
    # will be marked as xfailed
51
    def test_3(self):
52
        self.driver.get('https://www.lambdatest.com/')
53
        self.driver.maximize_window()
54
 
55
        expected_title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest"
56
        assert expected_title ==  self.driver.title
57
        time.sleep(5)
58
 
59
@pytest.mark.usefixtures("driver_init")
60
@pytest.mark.incremental
61
class Test_Scenario_2:
62
    def test_4(self):
63
        self.driver.get('https://lambdatest.github.io/sample-todo-app/')
64
        self.driver.maximize_window()
65
 
66
        self.driver.find_element_by_name("li1").click()
67
        self.driver.find_element_by_name("li2").click()
68
 
69
        title = "Sample page - lambdatest.com"
70
        assert title ==  self.driver.title
71
 
72
        sample_text = "Happy Testing at LambdaTest"
73
        email_text_field =  self.driver.find_element_by_id("sampletodotext")
74
        email_text_field.send_keys(sample_text)
75
        time.sleep(5)
76
 
77
        self.driver.find_element_by_id("addbutton").click()
78
        time.sleep(5)
79
 
80
        assert self.driver.find_element_by_xpath("//span[.='Happy Testing at LambdaTest']").text == sample_text
81
 
          


As seen in the implementation, incremental testing is applied to class Test_Scenario_2 and Test_Scenario_2 via the @pytest.mark.incremental decorator.

Python
 




xxxxxxxxxx
1
41


 
1
......................................
2
......................................
3
from time import sleep
4
import sys
5
import urllib3
6
import warnings
7
......................................
8
......................................
9
 
10
@pytest.mark.usefixtures("driver_init")
11
@pytest.mark.incremental
12
class Test_Scenario_1:
13
    ..................
14
    ..................
15
    def test_2(self):
16
        self.driver.get('https://www.google.com/')
17
        self.driver.maximize_window()
18
        title = "Google"
19
        assert title == self.driver.title
20
 
21
        search_text = "LambdaTest"
22
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
23
        search_box.send_keys(search_text)
24
 
25
        time.sleep(5)
26
        search_box.submit()
27
 
28
        time.sleep(5)
29
        
30
        # Click on the LambdaTest HomePage Link
31
        # This test will fail as the titles will not match
32
        title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
33
    ..................
34
    ..................
35
    
36
@pytest.mark.usefixtures("driver_init")
37
@pytest.mark.incremental
38
class Test_Scenario_2:
39
    ..................
40
    ..................
41
 
          


In test case test_2(), Google search for LambdaTest is performed. Once the search results are ready, a click on the first search result is done and the page title is compared with the expected title. For testing, we have deliberately introduced an error in the code and the Selenium test automation case fails as the titles do not match.

Python
 




xxxxxxxxxx
1
34


1
class Test_Scenario_1:
2
    ..................
3
    ..................
4
    def test_2(self):
5
        self.driver.get('https://www.google.com/')
6
        .................
7
      ..................
8
        search_text = "LambdaTest"
9
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
10
        search_box.send_keys(search_text)
11
 
12
        time.sleep(5)
13
        search_box.submit()
14
 
15
        time.sleep(5)
16
        
17
        # Click on the LambdaTest HomePage Link
18
        # This test will fail as the titles will not match
19
title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
20
    ..................
21
    ..................
22
 
23
    # As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests  in the same class
24
    # will be marked as xfailed
25
      def test_3(self):
26
      ..................
27
      ..................
28
@pytest.mark.usefixtures("driver_init")
29
@pytest.mark.incremental
30
class Test_Scenario_2:
31
    def test_4(self):
32
      ..................
33
      ..................
34
 
          


Test case test_3() which follows after test_2() will be skipped from execution and marked as xfail as the previous test [i.e. test_2()] in the same class has failed. Selenium test automation case test_4() which is a part of class Test_Scenario_2() is also marked for incremental testing.

Here is the command for execution:

py.test -rx --verbose --capture=no

Shown below is the execution snapshot:

execution snapshot

Here is the final status of the tests performed in the example:

test cases and execution status table

Test_2 executes and fails whereas test_3 execution is skipped due to failure of test_2. Hence, it is marked as xfail. Though the test_4 () passes here, even if it were to fail, the test would be marked as fail (and not xfail) as there is no dependency on the test from the other class.

In case, incremental testing is removed from the above implementation, all the tests would be executed. Only test_2() will fail whereas the remaining three tests would pass as the @pytest.mark.incremental decorator is removed from the implementation. At times you’d have to run multiple test cases in pytest from a single file, read this article to know more.

test scenario failed snapshot

Using Maxfail Command Line Option to Stop Test Suite After N Test Failures

PyTest offers a command-line option called maxfail which is used to stop test suite after n test failures. For example, to stop the test suite after n test failures, where the value of n is 1. Once a failure is encountered, all the subsequent tests are skipped.

Though it is not a good practice to develop inter-dependent Selenium test automation cases, in some scenarios, you may have to develop such tests. The maxfail option comes handy in such cases as unnecessary execution of dependent tests can be avoided. Shown below is the syntax of maxfail:

py.test --maxfail=<num>

How to Skip Test in Python With pytest Using Maxfail

To demonstrate the usage of maxfail, we take the automated browser testing example which was shown in the section How To Skip Test In Python With pytest Using @pytest.mark.incremental. The implementation-specific to skip test in Python with pytest will be removed, the remaining code remains the same.

conftest.py

Python
 




xxxxxxxxxx
1
18


 
1
import pytest
2
from selenium import webdriver
3
import urllib3
4
import warnings
5
 
6
def pytest_runtest_setup(item):
7
    previousfailed = getattr(item.parent, "_previousfailed", None)
8
    if previousfailed is not None:
9
        pytest.xfail("previous test failed (%s)" % previousfailed.name)
10
        
11
@pytest.fixture(scope="class")
12
def driver_init(request):
13
    web_driver = webdriver.Chrome()
14
    request.cls.driver = web_driver
15
    yield
16
    web_driver.close()
17
    # web_driver.quit()
18
 
          


test_step.py

Python
 




xxxxxxxxxx
1
78


 
1
#Using maxfail command-line operation to stop test suite after n test failure using Selenium test automation in Python with pytest 
2
import pytest
3
import pytest_html
4
from selenium import webdriver
5
from selenium.webdriver.chrome.options import Options
6
from selenium.webdriver.common.keys import Keys
7
import time
8
from time import sleep
9
import sys
10
import urllib3
11
import warnings
12
 
13
@pytest.mark.usefixtures("driver_init")
14
# @pytest.mark.incremental Stop Test Suite after N Test Failures in Pytest
15
class Test_Scenario_1:
16
    def test_1(self):
17
        self.driver.get('https://www.lambdatest.com/blog/')
18
        self.driver.maximize_window()
19
 
20
        expected_title = "LambdaTest | A Cross Browser Testing Blog"
21
        assert expected_title ==  self.driver.title
22
        time.sleep(5)
23
 
24
    def test_2(self):
25
        self.driver.get('https://www.google.com/')
26
        self.driver.maximize_window()
27
        title = "Google"
28
        assert title == self.driver.title
29
 
30
        search_text = "LambdaTest"
31
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
32
        search_box.send_keys(search_text)
33
 
34
        time.sleep(5)
35
        search_box.submit()
36
 
37
        time.sleep(5)
38
        
39
        # Click on the LambdaTest HomePage Link
40
        # This test will fail as the titles will not match
41
        title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
42
        lt_link = self.driver.find_element_by_xpath("//h3[.='LambdaTest: Cross Browser Testing Tools | Free Automated ...']")
43
        lt_link.click()
44
 
45
        time.sleep(10)
46
        assert title == self.driver.title   
47
        time.sleep(2)
48
 
49
    def test_3(self):
50
        self.driver.get('https://www.lambdatest.com/')
51
        self.driver.maximize_window()
52
 
53
        expected_title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest"
54
        assert expected_title ==  self.driver.title
55
        time.sleep(5)
56
 
57
@pytest.mark.usefixtures("driver_init")
58
# @pytest.mark.incremental
59
class Test_Scenario_2:
60
    def test_4(self):
61
        self.driver.get('https://lambdatest.github.io/sample-todo-app/')
62
        self.driver.maximize_window()
63
 
64
        self.driver.find_element_by_name("li1").click()
65
        self.driver.find_element_by_name("li2").click()
66
 
67
        title = "Sample page - lambdatest.com"
68
        assert title ==  self.driver.title
69
 
70
        sample_text = "Happy Testing at LambdaTest"
71
        email_text_field =  self.driver.find_element_by_id("sampletodotext")
72
        email_text_field.send_keys(sample_text)
73
        time.sleep(5)
74
 
75
        self.driver.find_element_by_id("addbutton").click()
76
        time.sleep(5)
77
 
78
        assert self.driver.find_element_by_xpath("//span[.='Happy Testing at LambdaTest']").text == sample_text     


The example shown below is executed where we stop the test suite after n test failures and the maximum number of failures set to 2. Here is the execution command:

py.test -rx --verbose --capture=no --maxfail=2

As incremental testing is not enabled, all the four Selenium test automation cases will be executed and the execution stops once the number of failures becomes 2. Test case test_2() fails as the window title does not match the expected one, rest all the three test cases are executed. Out of 4 test cases, 1 test case fails and 3 test cases pass. Here is the execution snapshot:

out of 4 test cases, 3 pass and 1 fails

All In All

In this Python tutorial, we looked at how to stop test suite after n test failures. Skip tests in Python with a pytest can be followed in Selenium test automation scenarios where tests are inter-dependent and execution of dependent tests needs to be skipped in case the primary test case fails.

The @pytest.mark.incremental decorator is used to skip tests in Python with PyTest. The other mechanism to stop test suite after n test failures by using the maxfail command-line option. Refer to this article, to learn more about Python fixtures

This brings an end to this Python tutorial. I hope this article was informative, and you now know skip tests in python or how to stop test suite after n test failures. In case you still have any doubts, do reach out to us in the comment section below, and we’d be happy to help you with your problem. Happy Testing!

Topics:
cross browser testing, pytest, python, selenium automated testing, testing tutorial

Published at DZone with permission of Himanshu Sheth . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}