Making Your Selenium Tests ''Unbreakable''

Note: This article bases on the recheck-web version 1.1.0, which hasn't been officially published.

If you ever had a test that failed with a NoSuchElementException, because someone changed the website and the identification criteria that the test used to identify an element to check or interact with, this article is for you.

This is a follow-up to the previous tutorial about how to use recheck-web in Java/Selenium project with the help of Maven, and how to maintain tests using the recheck.CLI. In the following, we assume that you have recheck-web setup locally. On that basis, this tutorial is a step-by-step guide of how to use recheck-web to make your tests almost unbreakable.

The GUI Element Identification Problem

Element identification is well-known problem for test automation engineers. Both when simulation user interaction (e.g. when clicking a certain button) and when verifying a certain SUT property (e.g. the text value in an input field), it is necessary to reliably identify different elements on the GUI. This is usually done by an unambiguously identifying piece of information, such as the XPath, the label, and internal id or the X/Y coordinates.

Recheck addresses this problem in an interesting and unique way. The Golden Master based Difference Testing approach brings some unique benefits with it: It basically saves a complete copy of the last working state. After a breaking change, that copy can be used for comparison. If e.g. the label was used for identification, and the label changed, we can simply have a look in the old version (the Golden Master) and see which element has the given label. Then, using other identifying attributes like its xpath, id, name, class and others, we can then find the corresponding element with a different label in the current state. It is even better: not only do we have redundant information that we can easily keep up to date, thanks to the recheck.CLI. We also have a much more complete picture: all other elements that were on the website in the previous state as well. This allows for an easy 1-on-1 assignment of all previous elements to their current counterpart, allowing for really robust element identification.

To use this functionality, we can simply change our previous test in the following way:

       private RecheckDriver driver;
       private RecheckWebImpl re;

       @Before
       public void setUp() {
              re = new RecheckWebImpl();
              driver = new RecheckDriver( new ChromeDriver());
       }

Instead of using the regular generic RecheckImpl, we use an adapted RecheckWebImpl  and wrap the regular Selenium driver into a special RecheckDriver, that also is an instance of RemoteWebDriver. That way, maximal compatibility to other third party tools and test frameworks is ensured. We can create a new test that shows that functionality.

For that we can use an example page from the Selenium project itself. We can download the formPage.html from the Selenium GitHub repository. Let us create matching test for it. It could look like so:

public class MyUnbreakableTest {

RecheckDriver driver;

@Before
public void setup() {
driver = new RecheckDriver( new ChromeDriver());
}

@Test
public void check_order() throws Exception {
driver.startTest();
String url = Paths.get( "src/test/resources/formPage.html" ).toUri().toURL().toString();
driver.get(url);

driver.findElement(By.id("email")).sendKeys("Max");
driver.findElement(By.id("age")).sendKeys("16");
driver.findElement(By.name("login")).submit();

driver.capTest();
}

@After
public void tearDown() throws InterruptedException {
driver.quit();
}
} 

As in the previous tutorial, we execute this test twice (using e.g. mvn test). The first execution will fail as there is no Golden Master to compare against, but will create the Golden Master while doing so. The second time we execute this test, it should pass.

Now we want to edit the HTML code of the page and change the used identifiers of the elements, that the test interacts with. So, we edit the formPage.html file and change the lines 15-19 from 

<form method="get" action="resultPage.html" name="login">
    <input type="email" id="email"/>
    <input type="number" id="age"/>
    <input type="submit" id="submitButton" value="Hello there"/>
</form>

to something like

<form method="get" action="resultPage.html" name="newLoginName">
    <input type="email" id="userEmail"/>
    <input type="number" id="numberOfLifeYears"/>
    <input type="submit" id="submit" value="Hello there"/>
</form>

Because these identifiers are used in the test, this would make a typical Selenium test fail without an actual problem in the web site—what is usually referred to as “breaking the test”. To showcase and verify this problem, we can simply change the used driver to the default ChromeDriver and comment the recheck-specific capTest() and cap() method calls (using //). Executing this test with a quick mvn test results in the dreaded NoSuchElementException.

Now let's have this test pass by the power of recheck-web and by simply ignoring all differences. To do so, edit the .retest/recheck.ignore file and add  attribute=.*. This will ignore all attribute changes, including changes to id and name. Then let’s revert the changes to recreate our original test above using the RecheckDriver. If you re-execute your test (mvn test), it will now pass—although the referenced ids and name are not in the web site anymore!

However, if you have a closer look at the log output that is printed to the console during execution, you can see that it will now contain messages similar to the following:

*************** recheck warning ***************
The HTML name attribute used for element identification changed from 'login' to 'newLoginName'.
retest identified the element based on the persisted Golden Master.
If you apply these changes to the Golden Master , your test com.mycompany.MyUnbreakableTest will break.
Use the `By.name("newLoginName")` or `By.retestId("form")` to update your test MyUnbreakableTest.java:30.

This means that the RecheckDriver catched the NoSuchElementException. It then loaded the persisted Golden Master and found the element with that name within the old version. Then it created the 1-on-1 assignment as shown above and was able to correctly associate the new with the old element. Then it just used the new element in the test and continued. However, if you apply that change to the Golden Master, recheck will then not be able to identify the element based on the outdated identification criteria anymore. So, you have to update your test when you apply the change—or else it will then break. So, in a sense, we only postponed the problem until the update of the Golden Master.

To really get rid of that problem altogether, we now can use the retestId, as suggested in the log. If you have a look at one of the Golden Master files (e.g. /src/test/resources/retest/recheck/MyUnbreakableTest/check.00.recheck/retest.xml), you can see that every element has an attribute called retestId. This retestId is a virtual and constant identifying attribute. We call it virtual, because it never actually shows up on the GUI—and therefore it is never affected by GUI changes, making it constant. Recheck uses the RetestIdProvider configured in the RecheckOptions that are passed to the RecheckDriver to generate that retestId. It can be any String value, as long as it is unique within the Golden Master. When you update your test after such a breaking change, you can now reference the retestId instead of other volatile identifying attributes, such as the HTML id or name, XPath or the like.

Now we can adapt the above test as suggested by the log output to be similar to the following (use the values of your log output):

       @Test
       public void check_order() throws Exception {
              driver.startTest();

              String url = Paths.get( "src/test/resources/formPage.html" ).toUri().toURL().toString();
              driver.get(url);

              driver.findElement(By.retestId("input")).sendKeys("Max");
              driver.findElement(By.retestId("input-e3f18")).sendKeys("16");
              driver.findElement(By.retestId("form")).submit();

              driver.capTest();
       }

Make sure to also import the correct By (i.e. import  de.retest.web.selenium.By). When we execute this test, it passes. Now we can remove the attribute=.* again from our recheck.ignore file to make the test fail as expected—because there actually was a difference. Using your recheck.CLI, you can now apply that change as you want—your test will not break anymore.

How Other Tools Do It

If the chosen identifying piece of information changes over time (e.g. the button label is changed) or within different environments (e.g. on a screen with a different resolution), the element in question can either not be identified anymore or is identified falsely (e.g. the wrong text field is used). This typically results in annoying false positives, i.e. failing tests where the tested functionality works as expected and merely the test itself is broken. Depending on when that problem arises, it can even be hard to detect (e.g. when a text is entered in a wrong field early during test execution).

AI can be applied to help identify the correct element using a multitude of identification criteria (e.g. XPath, label, id, class, X/Y coordinates), using the “looks” of the element by applying some form of image recognition, or by choosing the historically most stable single identification criterion. All of those approaches have been used in the past. None of the aforementioned approaches will address the underlying problem: the continual change of the software. So, if the correct element is identified by a multitude of criteria, the approach is robust against changes to one or even multiple of those criteria. However, if such changes occur, the underlying “baseline” then needs to be updated to reflect those changes. Otherwise, confidence in the identification will decrease after multiple changes occur to different criteria throughout the lifetime of the software. Furthermore, at some point the remaining unchanged criteria will not yield a high enough confidence—thus only postponing the problem, not solving it. The same is true for image recognition: if the past and present image increase in difference, at some point the remaining confidence will not be high enough. Also for choosing the historically most stable single identification criterion—even if it was the most stable one, at some point it might still change.

Conclusion

In contrast to other tools, recheck-web supports the underlying process: the constant change of software. With recheck-web it is now possible to reference an virtual constant identifying attribute (the retestId) that doesn't actually show up in the HTML, and as such is not affected by any code change. As long as the element still exists on the website, this will keep your test from breaking as a result to changes of the HTML id, name or class attributes, or the XPath. You can utilize that power even if you do not want to work with Golden Master based difference testing, simply by ignoring all changes.