Full-Stack Test Automation Frameworks — API Usability, Part 1

In one of the last articles from the series, we talked about tons of problems that modern test automation frameworks should be able to solve. The full-stack test automation frameworks are the latest 5th generation tooling. They have some features that make them better than previous generations. Allowing you to use them in the newly emerging complex contexts. In 5 Must-Have Features of Full-Stack Test Automation Frameworks Part 1, we talked about the first five features such framework should have. Some of them were cross-platform, cross-technology and cloud readiness. Troubleshooting easiness, library customization, and easy knowledge transfer. Here, I am going to extend the list and add a new category: API usability. Also, some of the mentioned features solve the "Automated Tests Are Not Stable" problem.

API Usability — Locate Elements

Vanilla WebDriver Example

"Vanilla" means examples that use only standard WebDriver code without anything else. Look at this standard WebDriver automated test:

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    IWebDriver driver = new ChromeDriver();
    driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
    var promotionsLink = driver.FindElement(By.Id("Promotions"));
    promotionsLink.Click();
    Console.WriteLine(promotionsLink.TagName);
}

Generally, this is OK. However, for me, at least in C#, using the FindElement syntax requires additional effort. It is not fluent since you had to begin a new "chain" using the By static class. Moreover, since By contains static methods, you are limited to the built-in locators. Which means that if the promotions ID is uglier, something like- sf_colsOut-sf_1col_1_100-promotions. You need a locator such as By.IdEndingWith.

Also, imagine that you want to wait for the button to disappear after it is clicked. It is doable with WebDriver. You use code like this:

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    IWebDriver driver = new ChromeDriver();
    driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
    var promotionsLink = driver.FindElement(By.Id("Promotions"));
    promotionsLink.Click();
    var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
    wait.Until(ExpectedConditions.InvisibilityOfElementLocated(By.Id("Promotions")));
    Console.WriteLine(promotionsLink.TagName);
}

I am sure you noticed the code duplication By.Id("Promotions"). This is bad since, if the locator changes in the future, we will have to fix it multiple times. This happens because the IWebElementinterface didn't give us a way to access the By class after the element is located, which is a usability problem. Moreover, the full WebDriverWait usage is quite complicated in my opinion. It can be made much simpler and more convenient.

NOTE: Keep in mind that all comparisons and "issues" I describe don't mean that I don't like WebDriver libraries and tools. On the contrary, this is my favorite automation tool. As it is written on the SeleniumHQ homepage, "Selenium automates browsers." Because people use it in countless ways, it is created to be generic. Only one of its uses is automated testing. It is not a test automation framework, so it is entirely normal to miss some features or its APIs might not provide maximum usability in certain use cases.

Improved Example

Here is the same test, rewritten using the Bellatrix Test Automation Framework.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
    var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
    promotionsLink.Click();

    Console.WriteLine(promotionsLink.By.Value);
    Console.WriteLine(promotionsLink.WrappedElement.TagName);
}

The first noticeable difference is how we locate elements. Instead of using the By syntax, which is harder to type and non-extendable, we use CreateBy methods. Everything follows the natural writing flow, leveraging IntelliSense to the maximum degree. Moreover, you can automatically generate the code using code snippets.

Bellatrix contains CreateBy methods for all often-used locators. One of the coolest things is that Bellatrix's element includes By property (knowing how it was located).

If we need to wait for the button to be disabled, we can use the following code:

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
    var promotionsLink = App.ElementCreateService.CreateById<Button>("Promotions");
    promotionsLink.Click();

    promotionsLink.EnsureIsNotVisible();
}

As you can see, thanks to the built-in By property, the code duplication is skipped. Moreover, the syntax is simplified to the bare minimum. The Ensure method automatically waits until the button is disabled.

Documentation

API Usability — Wait for Elements

Vanilla WebDriver Example

It is relatively complex to wait for conditions with vanilla WebDriver. Maybe "complex" is not the most accurate word, but for sure, you will need additional classes/methods to skip code duplication. As mentioned, since you don't have access to how the WebDriver element was located, you need to specify this on multiple locations or think of a way to reuse the By locators.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    IWebDriver driver = new ChromeDriver();
    driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
    var promotionsLink = driver.FindElement(By.Id("Promotions"));

    promotionsLink.Click();

    var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
    wait.Until(ExpectedConditions.ElementToBeClickable(By.Id("Promotions")));
}

There is one more usability issue with the Wait-Until API. You see all ExpectedConditions static methods; they are not filtered based on the type of the element. Refer to the below example. ElementToBeSelected makes sense only for comboBoxes or select elements.

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
wait.Until(ExpectedConditions.ElementToBeSelected(By.Id("Promotions")));

Above, we pass a By locator for button, but we can use the ElementToBeSelected method.

One more thing: the usage is different depending on the methods you use.

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));  
wait.Until(ExpectedConditions.TextToBePresentInElement(promotionsLink, "Bellatrix"));

Instead of By locator, the method accepts IWebElement. In my opinion, these differences make the whole usage more confusing and complicated, which in the end, slows down the test's development.

Not only are the methods not filtered based on the type of element, but you see all kind of methods here, related to title, URL, and frames. This pollutes the API interface even more, which leads to longer periods while you find the right method.

Improved Example

How did we solve these problems in Bellatrix? You already saw an example of how we waited for the element not to be present on the page. But we go even further. To make the API most convenient, the EnsureSomeCondition methods are part of the element's API interface. Also, you will see only the relevant methods for the element, based on its type. For example, for an Anchor element, you won't see the EnsureIsDisabled method, since HTML anchor elements don't have a disabled attribute.

The usage is identical for all elements since you don't have to specify locators a second time.

var promotionsLink = App.ElementCreateService.CreateByLinkText<Button>("Promotions");
promotionsLink.Click();
promotionsLink.EnsureIsDisabled();

Documentation

Automatically Handle All Synchronization Issues

Anotherfrequent problem that people face is related to elements still not present on the page or not meeting some condition. Usually, all of these problems can be handled entirely with WebDriver code, but it varies depending on what difficulty of error you try to solve. If you don't configure the library the right way, you may make your tests significantly slower.

Implicit vs. Explicit Waits

One way to handle the synchronization issues is through a global implicit wait timeout.

IWebDriver driver = new ChromeDriver();
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(30);

However, in some cases, you may need a larger wait interval. What do you do if this happens? One option is to increase the global timeout, but this will affect all existing tests. Another option is to mix implicit and explicit wait (showed in previous examples). But this is not recommended.

Here is a quote from Jim Evans (one of the Core Selenium contributors) on why this is not recommended:

"When you try to mix implicit and explicit waits, you've strayed into "undefined behavior." You might be able to figure out what the rules of that behavior are, but they'll be subject to change as the implementation details of the drivers change. So don't do it...Don't mix implicit and explicit waits. Part of the problem is that implicit waits are often (but may not always be!) implemented on the "remote" side of the WebDriver system. That means they're "baked in" to IEDriverServer.exe, chromedriver.exe, the WebDriver Firefox extension that gets installed into the anonymous Firefox profile, and the Java remote WebDriver server (selenium-server-standalone.jar). Explicit waits are implemented exclusively in the "local" language bindings. Things get much more complicated when using RemoteWebDriver, because you could be using both the local and remote sides of the system multiple times."

Full Stack Overflow Thread

Wait for Conditions Before First Element's Usage

In most tests, before you use some element (e.g. click or type text in it), you need it to be visible and existing on the web page. However, it needs to be clickable or selectable, which means that you need to use WebDriverWait not only for assertions/verifications but also before performing actions.

Here is a simple example where we wait for the element to exists before the first usage.

IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var promotionsLink = wait.Until(ExpectedConditions.ElementExists(By.Id("Promotions")));

promotionsLink.Click();

However, imagine that we need our element to fulfill more than two conditions: to exists and to be clickable. Shortly, you will realize that there isn't an easy way to do that. You can write something like this:

IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var promotionsLocator = By.Id("Promotions");
wait.Until(ExpectedConditions.ElementExists(promotionsLocator));
var promotionsLink = wait.Until(ExpectedConditions.ElementToBeClickable(promotionsLocator));

promotionsLink.Click();

There is always the alternative to open the ExpectedConditions source code, view how both methods work and combine them in your anonymous or normal function. But believe me, you will end up with much more complex code. Moreover, it is not rational to create such functions for each pair or set of conditions you might need.

Improved Example

How did we decide to handle these problems in Bellatrix? First, all elements are internally waited to exist before usage. This solves 80% of all use cases. After that as part of the CreateBy API, we added additional ToBeCondition methods which you can chain. Which means you can specify an unlimited number of conditions for each element that needs to be fulfilled before the element is returned.

var promotionsLink = App.ElementCreateService.CreateByLinkText<Button>("Promotions").ToBeVisible().ToBeClickable().ToExists();
promotionsLink.Click();
promotionsLink.EnsureIsDisabled();

Different Timeouts Problem

But this is not everything. If you use a single WebDriverWait instance, the timeout will be the same for each Until method and condition you use. But this may vary for each condition you use and each usage of these conditions. For example, I want in my test the timeout for ToBeVisible to be 15 seconds but for ToBeNotVisible to be 30 seconds. Next, for some optimized web pages, I may wish to set the ToBeVisible timeout to be 10 seconds, not 15. In vanilla WebDriver, this flexibility is missing, and you need to handle this with custom code somehow.

Here is a naive solution to the problem with vanilla WebDriver:

IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var wait15Seconds = new WebDriverWait(driver, TimeSpan.FromSeconds(15));
var wait30Seconds = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var wait45Seconds = new WebDriverWait(driver, TimeSpan.FromSeconds(45));
var promotionsLocator = By.Id("Promotions");
wait15Seconds.Until(ExpectedConditions.ElementExists(promotionsLocator)); // same 30 seconds
var promotionsLink = wait30Seconds.Until(ExpectedConditions.ElementToBeClickable(promotionsLocator));

promotionsLink.Click();

We have multiple WebDriverWait instances with different timeouts.

Different Timeouts Solution

In Bellatrix, we have a JSON configuration where we can fine-tune different aspects of the framework. There is a dedicated section called timeoutSettings, where you can change the default timeouts for the different wait conditions.

"timeoutSettings": {
    "waitForAjaxTimeout": "30",
    "sleepInterval": "1",
    "elementToBeVisibleTimeout": "30",
    "elementToExistTimeout": "30",
    "elementToNotExistTimeout": "30",
    "elementToBeClickableTimeout": "30",
    "elementNotToBeVisibleTimeout": "30",
    "elementToHaveContentTimeout": "15"
},

But you can also, further override these values directly in the methods. Below, we set the ToBeVisible timeout to 30 seconds, ToBeClickable to 20 seconds with sleep interval = 2 seconds, and ToBeVisible timeout to 10 seconds with a sleep interval of 1 second.

var promotionsLink = App.ElementCreateService.CreateByLinkText<Button>("Promotions").ToBeVisible(30).ToBeClickable(20, 2).ToExists(10, 1);
promotionsLink.Click();
promotionsLink.EnsureIsDisabled();

Documentation

Summary

Full-stack test automation frameworks have many features that can enable you to work in the new emerging complex contexts. Here, we primarily talked about test automation frameworks API usability — locating and waiting for elements. In the next articles of the series, we will discuss how to create additional locators, generate elements faster, and why you should care about this.