5 Must-Have Features of Full-Stack Test Automation Frameworks

Full-stack test automation frameworks are the latest 5th-generation tooling. They have some features that make them better than previous generations, allowing you to use them in the newly emerging complex contexts, such as supporting multiple OSes or testing mobile, desktop, web, and APIs. With the continuously increasing complexity of the problems that the tooling has to solve, the demand for troubleshooting and debugging increases.

Cross-Technology and Cross-Platform Readiness

Nowadays, engineers shouldn't be limited as to which OS they use. By definition, frameworks should be completely generic, and they shouldn't restrict their users. This means that they should be completely cross-platform, supporting Windows, Linux, and MacOS.

For example, our test automation framework Bellatrix is entirely written on .NET Core and .NET Standard. Through cross-platform templates, we distribute it on each OS using only the native CLI. After that, you can use your favorite IDE- Visual Studio (Windows or Mac) or Visual Studio Code. Using the same editor makes the development and discussing of possible problems more comfortable.

By cross-technology readiness, I mean to be able to write tests for different technologies such as web, mobile, desktop, and APIs. For me, this also includes a similar API. In Bellatrix, we tried to make the API for different modules as identical as possible. Below are a few examples.

Bellatrix Web

[TestClass]
[Browser(BrowserType.Firefox, BrowserBehavior.ReuseIfStarted)]
[ExecutionTimeUnder(2)]
public class BellatrixBrowserBehaviourTests : WebTest
{
    [TestMethod]
    [Browser(BrowserType.Chrome, BrowserBehavior.RestartOnFail)]
    public void BlogPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var blogLink = App.ElementCreateService.CreateByLinkText<Anchor>("Blog");
        blogLink.EnsureIsVisible();

        blogLink.Click();
    }
}

Documentation

Bellatrix Desktop

[TestClass]
[App(Constants.WpfAppPath, AppBehavior.RestartEveryTime)]
[ExecutionTimeUnder(2)]
public class ControlAppTests : DesktopTest
{
    [TestMethod]
    public void MessageChanged_When_ButtonHovered_Wpf()
    {
        var button = App.ElementCreateService.CreateByName<Button>("LoginButton");
        button.Hover();
        var label = App.ElementCreateService.CreateByName<Button>("successLabel");
        label.EnsureInnerTextIs("Sucess");
    }
}

Documentation

Bellatrix API

[TestClass]
[ExecutionTimeUnder(2)]
public class CreateSimpleRequestTests : APITest
{
    [TestMethod]
    public void GetAlbumById()
    {
        var request = new RestRequest("api/Albums/10");        
        var client = App.GetApiClientService();

        var response = client.Get<Albums>(request);
        response.AssertContentContains("Audioslave");
    }
}

Documentation

Cloud Readiness

Some companies have their own farms of devices and computers with various browser configurations. However, nowadays, cloud providers such as SauceLabs, BrowserStack, or CrossBrowserTesting are a reasonable solution to the problem. These integrations can help you to perform pixel-perfect layout testing on various devices.

A major requirement for the full-stack test automation frameworks is to allow you to execute your tests in the cloud without complex configurations effortless.

In Bellatrix, we make that possible through the usage of a single attribute. Our implementation is based on the observer design pattern.

[TestClass]
[SauceLabs(BrowserType.Chrome, "62", "Windows", BrowserBehavior.ReuseIfStarted, recordScreenshots: true, recordVideo: true)]
public class SauceLabsTests : WebTest
{
    [TestMethod]
    public void PromotionsPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
        promotionsLink.Click();
    }
}

You have similar attributes for BrowserStack and CrossBrowserTesting.

[BrowserStack(BrowserType.Chrome,
    "62",
    "Windows",
    "10",
    BrowserBehavior.ReuseIfStarted,
    captureNetworkLogs: true,
    captureVideo: true,
    consoleLogType: BrowserStackConsoleLogType.Verbose,
    debug: true,
    build: "myUniqueBuildName")]

Documentation

Easy Knowledge Transfer

Documentation is not enough. I bet that most of you that have more than a few years of experience in the field of automated testing and have a custom framework spent countless hours teaching new colleagues how to write "proper" tests using the team's framework. I believe that there should be a more automated process allowing people to learn by themselves. I think one way to do it is to utilize so-called getting started solutions or starter kits. Projects that have examples with explanations on how to write tests and why we use a particular method or not.

For each test technology, we created a similar starter kit for Bellatrix. Each of them explains the features of the framework with detailed real-world examples. Moreover, it contains detailed comments for each part. Something that makes it ideal for self-learning is that after each chapter, it offers exercises that people can do themselves.

Different features are grouped in separate folders, and at the end of each chapter, there is a TODO file containing the exercises.

Troubleshooting Ease

With increasing test count and complexity, it will be even more critical that the tests be maintainable. A significant part of this effort is easier troubleshooting and better support for locating errors.

A big part of maintainability is troubleshooting existing tests. Most in-house solutions or open-source ones don't provide lots of features to make your life easier. This can be one of the most time-consuming tasks. You can have 100 failing tests and find out whether there is a problem with the test or a bug in the application. If you use plugins or complicated design patterns, debugging the tests will be much harder, requiring lots of resources and expertise.

Two of the ways Bellatrix as full-stack test automation framework handles these problems are through full-page screenshots and video recording on test fail. Again, following the paradigm of similar APIs, we use these features through attributes.

Bellatrix Full-page Screenshots

[TestClass]
[ScreenshotOnFail(true)]
[Browser(BrowserType.Chrome, BrowserBehavior.ReuseIfStarted)]
public class FullPageScreenshotsOnFailTests : WebTest
{
    [TestMethod]
    public void PromotionsPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
        promotionsLink.Click();
    }
}

If you want to read how it works internally, you can read the article: Capture Full Page Screenshots Using WebDriver with HTML2Canvas.js.

Bellatrix Video Recording

[TestClass]
[VideoRecording(VideoRecordingMode.OnlyFail)]
[Browser(BrowserType.Chrome, BrowserBehavior.ReuseIfStarted)]
public class VideoRecordingTests : WebTest
{
    [TestMethod]
    public void PromotionsPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
        promotionsLink.Click();
    }
}

Library Customization

One of the hardest things to develop is a way to allow these generic frameworks to be extendable and customizable. The whole point of creating a shared library is to be used by multiple teams across the company. However, different teams work in different contexts. They may have to test somewhat different things, so the library code as-is may not be working out of the box for them. Thus, the engineers should be able to customize parts to fit their needs.

Below, you can find a couple of examples of how this is possible with Bellatrix. Next, I will show you how it works if you wish to implement it in your framework.

Override Actions Globally

[TestClass]
[Browser(BrowserType.Firefox, BrowserBehavior.RestartEveryTime)]
public class OverrideGloballyElementActionsTests : WebTest
{
    public override void TestsArrange()
    {
        Button.OverrideClickGlobally = (e) =>
        {
            e.ToExists().ToBeClickable().WaitToBe();
            App.JavaScriptService.Execute("arguments[0].click();", e);
        };
        Anchor.OverrideFocusGlobally = CustomFocus;
    }
    private void CustomFocus(Anchor anchor)
    {
        App.JavaScriptService.Execute("window.focus();");
        App.JavaScriptService.Execute("arguments[0].focus();", anchor);
    }
    [TestMethod]
    public void PurchaseRocketWithGloballyOverridenMethods()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        Select sortDropDown = App.ElementCreateService.CreateByNameEndingWith<Select>("orderby");     
        Anchor addToCartFalcon9 = 
        App.ElementCreateService.CreateByAttributesContaining<Anchor>("data-product_id", "28").ToBeClickable();
        Span totalSpan = App.ElementCreateService.CreateByXpath<Span>("//*[@class='order-total']//span");
        sortDropDown.SelectByText("Sort by price: low to high");
        addToCartFalcon9.Focus();
        addToCartFalcon9.Click();
        totalSpan.EnsureInnerTextIs("95.00€", 15000);
    }
}

Here, through the OverrideClickGlobally delegate, we change the default behavior of all buttons in the framework without recompiling the library. In the example, instead of using the default WebDriver implementation, we change the Click method to use JavaScript instead. Usually, we execute this once for all tests, so the right place to call the override is in the AssemlyInitialize method of MSTest that is performed once per assembly.

Documentation

Override Actions Locally

Another possible customization is through local override which means that the behavior will be changed only for the currently executing test.

Button.OverrideClickLocally = (e) =>
{
    e.ToExists().ToBeClickable().WaitToBe();
    App.JavaScriptService.Execute("arguments[0].click();", e);
};

Documentation

Extensibility Through Events

public class DebugLoggingButtonEventHandlers : ButtonEventHandlers
{
    protected override void ClickingEventHandler(object sender, ElementActionEventArgs arg)
    {
        DebugLogger.LogInfo($"Before clicking button. Coordinates: X={arg.Element.WrappedElement.Location.X} Y={arg.Element.WrappedElement.Location.Y}");
    }
    protected override void HoveringEventHandler(object sender, ElementActionEventArgs arg)
    {
        DebugLogger.LogInfo($"Before hovering button. Coordinates: X={arg.Element.WrappedElement.Location.X} Y={arg.Element.WrappedElement.Location.Y}");
    }
}
[TestClass]
[Browser(BrowserType.Chrome, BrowserBehavior.RestartEveryTime)]
public class ElementActionHooksTests : WebTest
{
    public override void TestsArrange()
    {
        App.AddElementEventHandler<DebugLoggingButtonEventHandlers>();
    }
    [TestMethod]
    public void PurchaseRocketWithGloballyOverridenMethods()
    {
        // some test logic
    }
}

Another way to extend Bellatrix is to use the controls hooks. This is how BDD logging and highlighting are implemented. For each method of the control, there are two hooks- one that is called before the action and one after. For example, the available hooks for the button are:

• Clicking - an event executed before button click
• Clicked - an event executed after the button is clicked
• Hovering - an event executed before button hover
• Hovered - an event executed after the button is hovered over
• Focusing - an event executed before button focus
• Focused - an event executed after the button is focused

You need to implement the event handlers for these events and subscribe them. Bellatrix gives you again a shortcut- you need to create a class and inherit the {ControlName}EventHandlers. In the example, DebugLogger is called for each button event printing to Debug window the coordinates of the button. You can call external logging provider, making screenshots before or after each action, the possibilities are limitless.

Documentation

There are many more ways how you can enable extensibility and customization of your full-stack test automation framework. You can check how we do it in the Extensibility section of Bellatrix documentation. 

Next, I will show you how it works internally and how to do it yourself.

API Usability — Locate Elements

Vanilla WebDriver Example

Vanilla stands for examples that use only standard WebDriver code without anything else. Look at this standard WebDriver automated test.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    IWebDriver driver = new ChromeDriver();
    driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
    var promotionsLink = driver.FindElement(By.Id("Promotions"));
    promotionsLink.Click();
    Console.WriteLine(promotionsLink.TagName);
}

Generally, this is quite OK. However, for me at least in C# using the FindElement syntax requires additional effort. It is not fluent since you had to begin a new "chain" using the By static class. Moreover, since By contains static methods, you are limited to the built-in locators. This means that if the promotions ID is uglier, something like sf_colsOut-sf_1col_1_100-promotions, you need a locator such as By.IdEndingWith.

Also, imagine that you want to wait for the button to disappear after it is clicked. It is doable with WebDriver. You use code like below.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    IWebDriver driver = new ChromeDriver();
    driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
    var promotionsLink = driver.FindElement(By.Id("Promotions"));
    promotionsLink.Click();
    var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
    wait.Until(ExpectedConditions.InvisibilityOfElementLocated(By.Id("Promotions")));
    Console.WriteLine(promotionsLink.TagName);
}

I am sure you noticed the code duplication: By.Id("Promotions"), which is bad since if the locator changes in the future we will have to fix it multiple times. This happens because IWebElement interface didn't give us a way to access the By class after the element is located which is a usability problem. Moreover, the full WebDriverWait usage is quite complicated, in my opinion. It can be made much simpler and more convenient.

NOTE: Keep in mind that all comparisons and "issues" I describe, doesn't mean that I don't like WebDriver libraries and tools. On the contrary, this is my favorite automation tool. As it is written on the SeleniumHQ homepage: "Selenium automates browsers." Because people use it in countless ways, it is created to be generic. Only one of its usages is automated testing. It is not testing automation framework, so it is entirely normal to miss some features or its APIs not provide maximal usability in certain use cases.

Improved Example

Here is the same test, rewritten using the Bellatrix Test Automation Framework.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
    var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
    promotionsLink.Click();

    Console.WriteLine(promotionsLink.By.Value);
    Console.WriteLine(promotionsLink.WrappedElement.TagName);
}

The first noticeable difference is how we locate elements. Instead of using the By syntax which is harder to type and not extendable, we use CreateBy methods. Everything follows the natural writing flow leveraging to the maximal degree on IntelliSense. Moreover, you can automatically generate the code using code snippets.

Bellatrix contains CreateBy methods for all often-used locators. One of the coolest things is that Bellatrix element includes By property (knowing how it was located). 

If we need to wait for the button to be disabled, we can use the following code.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
    var promotionsLink = App.ElementCreateService.CreateById<Button>("Promotions");
    promotionsLink.Click();

    promotionsLink.EnsureIsNotVisible();
}

As you can see thanks to the built-in By property, the code duplication is skipped. Moreover, the syntax is simplified to the bare minimum. The Ensure method automatically waits until the button is disabled.

Documentation

API Usability — Wait for Elements

Vanilla WebDriver Example

It is relatively complex to wait for conditions with vanilla WebDriver. Maybe "complex" is not the most accurate word but for sure you will need additional classes/methods to skip code duplication. As mentioned, since you don't have access to how the WebDriver element was located you need to specify this on multiple locations or think of a way how to reuse the By locators.

[TestMethod]
public void OpenBellatrixDemoPromotions()
{
    IWebDriver driver = new ChromeDriver();
    driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
    var promotionsLink = driver.FindElement(By.Id("Promotions"));

    promotionsLink.Click();

    var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
    wait.Until(ExpectedConditions.ElementToBeClickable(By.Id("Promotions")));
}

There is one more usability issue with the Wait-Until API. You see all ExpectedConditions static methods, there are not filtered based on the type of the element. Check the below example. ElementToBeSelected makes sense only for comboBoxes or select elements.

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
wait.Until(ExpectedConditions.ElementToBeSelected(By.Id("Promotions")));

Above, we pass By locator for the button but we can use the ElementToBeSelected method.

The usage is different depending on the methods you use.

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));  
wait.Until(ExpectedConditions.TextToBePresentInElement(promotionsLink, "Bellatrix"));

Instead of By locator, the method accepts IWebElement. In my opinion, these differences make the whole usage more confusing and complicating which in the end slows down the tests' development.

Not only the methods are not filtered based on the type of the element but you see all kind of methods here- related to title, URL, and frames, which pollutes the API interface even more, which leads to longer periods while you find the right method.

Improved Example

You already saw an example of how we waited for an element not be present on the page. But we go even further. To make the API most convenient the EnsureSomeCondition methods are part of the elements API interface. Also, you will see only the relevant methods for the element based on its type. For example, for Anchor element, you won't see the EnsureIsDisabled method since HTML anchor elements don't have a disabled attribute.

The usage is identical for all elements since you don't have to specify locators a second time.

var promotionsLink = App.ElementCreateService.CreateByLinkText<Button>("Promotions");
promotionsLink.Click();
promotionsLink.EnsureIsDisabled();

Automatically Handle All Synchronization Issues

Another frequent problem that people face is related to elements still not present on the page or not meeting some conditions. Usually, all of these problems can be handled entirely with WebDriver code, but it varies depending on what difficulty you try to solve. If you don't configure the library the right way, you may make your tests significantly slower.

Implicit Vs. Explicit Waits

One way to handle the synchronization issues is through a global implicit wait timeout.

IWebDriver driver = new ChromeDriver();
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(30);

However, in some cases, you may need a larger wait interval. What do you do if this happens? One option is to increase the global timeout, but this will affect all existing tests. Another option is to mix implicit and explicit wait (shown in previous examples). But this is not recommended.

Here is a quote from Jim Evans (one of the Core Selenium contributors) about why this is not recommended.

"When you try to mix implicit and explicit waits, you've strayed into "undefined behavior". You might be able to figure out what the rules of that behavior are, but they'll be subject to change as the implementation details of the drivers change. So don't do it."

"Don't mix implicit and explicit waits. Part of the problem is that implicit waits are often (but may not always be!) implemented on the "remote" side of the WebDriver system. Which means they're "baked in" to IEDriverServer.exe, chromedriver.exe, the WebDriver Firefox extension that gets installed into the anonymous Firefox profile, and the Java remote WebDriver server (selenium-server-standalone.jar). Explicit waits are implemented exclusively in the "local" language bindings. Things get much more complicated when using RemoteWebDriver, because you could be using both the local and remote sides of the system multiple times."

Full Stack Overflow Thread

Wait for Conditions Before First Element's Usage

In most tests, before you use some element- click or type text in it, you need it to be visible and existing on the web page. However, it needs to be clickable or selectable. This means that you need to use WebDriverWait not only for assertions/verifications but also before performing actions.

Here is a simple example where we wait for the element to exist before the first use.

IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var promotionsLink = wait.Until(ExpectedConditions.ElementExists(By.Id("Promotions")));

promotionsLink.Click();

However, imagine that we need our element to fulfill more than two conditions- to exist and to be clickable. Shortly you will realize that there isn't an easy way to do that. You can write something like this.

IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var promotionsLocator = By.Id("Promotions");
wait.Until(ExpectedConditions.ElementExists(promotionsLocator));
var promotionsLink = wait.Until(ExpectedConditions.ElementToBeClickable(promotionsLocator));

promotionsLink.Click();

There is always the alternative to open the ExpectedConditions source code — view how both methods work and combine them in your anonymous or normal function. But believe me, you will end up with much more complex code. Moreover, it is not rational to create such functions for each pair or tripe of conditions you might need. 

Improved Example

How did we decide to handle these problems in Bellatrix? First, all elements are internally waited to exist before usage. This solves 80 percent of all use cases. After that as part of the CreateBy API, we added additional ToBeCondition methods which you can chain. This means you can specify an unlimited number of conditions for each element that needs to be fulfilled before the element is returned.

var promotionsLink = App.ElementCreateService.CreateByLinkText<Button>("Promotions").ToBeVisible().ToBeClickable().ToExists();
promotionsLink.Click();
promotionsLink.EnsureIsDisabled();

Different Timeouts Problem

But this is not everything. If you use a single WebDriverWait instance, this means that the timeout will be the same for each Until method and condition you use. But this may vary for each condition you use and each usage of these conditions. For example, I want in my test the timeout for ToBeVisible to be 15 seconds but for ToBeNotVisible 30 seconds. Next, for some optimized web pages I may wish to set the ToBeVisible timeout to be 10 seconds, not 15. In vanilla WebDriver, this flexibility is missing, and you need to handle this with custom code somehow.

Here is a naive solution to the problem with vanilla WebDriver.

IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var wait15Seconds = new WebDriverWait(driver, TimeSpan.FromSeconds(15));
var wait30Seconds = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var wait45Seconds = new WebDriverWait(driver, TimeSpan.FromSeconds(45));
var promotionsLocator = By.Id("Promotions");
wait15Seconds.Until(ExpectedConditions.ElementExists(promotionsLocator)); // same 30 seconds
var promotionsLink = wait30Seconds.Until(ExpectedConditions.ElementToBeClickable(promotionsLocator));

promotionsLink.Click();

We have multiple WebDriverWait instances with different timeouts.

Different Timeouts Solution

In Bellatrix, we have a JSON configuration where we can fine-tune different aspects of the framework. There is a dedicated section called timeoutSettings, where you can change the default timeouts for the different wait conditions.

"timeoutSettings": {
    "waitForAjaxTimeout": "30",
    "sleepInterval": "1",
    "elementToBeVisibleTimeout": "30",
    "elementToExistTimeout": "30",
    "elementToNotExistTimeout": "30",
    "elementToBeClickableTimeout": "30",
    "elementNotToBeVisibleTimeout": "30",
    "elementToHaveContentTimeout": "15"
},

But you can also, further override these values directly in the methods. Below, we set ToBeVisible timeout to be 30 seconds, ToBeClickable 20 seconds with sleep interval = 2 seconds, ToBeVisible timeout = 10 seconds with sleep interval 1 second.

var promotionsLink = App.ElementCreateService.CreateByLinkText<Button>("Promotions").ToBeVisible(30).ToBeClickable(20, 2).ToExists(10, 1);
promotionsLink.Click();
promotionsLink.EnsureIsDisabled();

Documentation

Summary

Full-stack test automation frameworks have many features that enable you to work in emerging complex contexts. Above, we talked about how necessary is the tooling to be cross-platform and cross-technology ready. These tools should make it easy for you to execute your tests in the cloud so you can test your applications on multiple devices and OSes. A major must-have for such frameworks is to give you ways to find why your tests failed, such as screenshots and videos. Last but not least, since they are used across a wide range of teams' contexts, they should provide ways for customization and extensibility.