Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

5 Must-Have Features of Full-Stack Test Automation Frameworks — Part 1

DZone 's Guide to

5 Must-Have Features of Full-Stack Test Automation Frameworks — Part 1

These examples of test automation framework features shows what is needed for cross-platform and cloud readiness, troubleshooting, and more.

· DevOps Zone ·
Free Resource

In the last article from this series, we talked about tons of problems that modern test automation frameworks should be able to solve. Full-stack test automation frameworks are the latest 5th-generation tooling. They have some features that make them better than previous generations, allowing you to use them in the newly emerging complex contexts, such as supporting multiple OSes or testing mobile, desktop, web, and APIs. With the continuously increasing complexity of the problems that the tooling has to solve, the demand for troubleshooting and debugging increases.

In the next couple of articles, I am going to talk about some of these features. Here we will look at the first five of them.

Cross-Technology and Cross-Platform Readiness

Nowadays, engineers shouldn't be limited which OS they use. By definition, frameworks should be completely generic, and they shouldn't restrict their users. This means that they should be completely cross-platform, supporting Windows, Linux, and MacOS.

For example, our test automation framework Bellatrix is entirely written on .NET Core and .NET Standard. Through cross-platform templates, we distribute it on each OS using only the native CLI. After that, you can use your favorite IDE- Visual Studio (Windows or Mac) or Visual Studio Code. Using the same editor makes the development and discussing possible problems more comfortable.

By cross-technology readiness, I mean to be able to write tests for different technologies such as web, mobile, desktop, and APIs. For me, this also includes a similar API. In Bellatrix, we tried to make the API for different modules as identical as possible. Below are a few examples.

Bellatrix Web

[TestClass]
[Browser(BrowserType.Firefox, BrowserBehavior.ReuseIfStarted)]
[ExecutionTimeUnder(2)]
public class BellatrixBrowserBehaviourTests : WebTest
{
    [TestMethod]
    [Browser(BrowserType.Chrome, BrowserBehavior.RestartOnFail)]
    public void BlogPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var blogLink = App.ElementCreateService.CreateByLinkText<Anchor>("Blog");
        blogLink.EnsureIsVisible();

        blogLink.Click();
    }
}

Documentation

Bellatrix Desktop

[TestClass]
[App(Constants.WpfAppPath, AppBehavior.RestartEveryTime)]
[ExecutionTimeUnder(2)]
public class ControlAppTests : DesktopTest
{
    [TestMethod]
    public void MessageChanged_When_ButtonHovered_Wpf()
    {
        var button = App.ElementCreateService.CreateByName<Button>("LoginButton");
        button.Hover();
        var label = App.ElementCreateService.CreateByName<Button>("successLabel");
        label.EnsureInnerTextIs("Sucess");
    }
}

Documentation

Bellatrix API

[TestClass]
[ExecutionTimeUnder(2)]
public class CreateSimpleRequestTests : APITest
{
    [TestMethod]
    public void GetAlbumById()
    {
        var request = new RestRequest("api/Albums/10");        
        var client = App.GetApiClientService();

        var response = client.Get<Albums>(request);
        response.AssertContentContains("Audioslave");
    }
}

Documentation

Cloud Readiness

Some companies have their own farms of devices and computers with various browser configurations. However, nowadays, cloud providers such as SauceLabsBrowserStack, or CrossBrowserTesting are a reasonable solution to the problem. These integrations can help you to perform pixel perfect layout testing on various devices.
A major requirement for the full-stack test automation frameworks is to allow you to execute your tests in the cloud without complex configurations effortless.

In Bellatrix, we make that possible through the usage of a single attribute. Our implementation is based on the observer design pattern.

[TestClass]
[SauceLabs(BrowserType.Chrome, "62", "Windows", BrowserBehavior.ReuseIfStarted, recordScreenshots: true, recordVideo: true)]
public class SauceLabsTests : WebTest
{
    [TestMethod]
    public void PromotionsPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
        promotionsLink.Click();
    }
}

You have similar attributes for BrowserStack and CrossBrowserTesting.

[BrowserStack(BrowserType.Chrome,
    "62",
    "Windows",
    "10",
    BrowserBehavior.ReuseIfStarted,
    captureNetworkLogs: true,
    captureVideo: true,
    consoleLogType: BrowserStackConsoleLogType.Verbose,
    debug: true,
    build: "myUniqueBuildName")]

Documentation

Easy Knowledge Transfer

Documentation is not enough. I bet that most of you that have more than few years of experience in the field of automated testing and have a custom framework spent countless hours teaching new colleagues how to write "proper" tests using the team's framework. I believe that there should be a more automated process allowing people to learn by themselves. I think one way to do it is to utilize so-called getting started solutions or starter kits. Projects that have examples with explanations on how to write tests and why we use a particular method or not.

For each test technology, we created a similar starter kit for Bellatrix. Each of them explains the features of the framework with detailed real-world examples. Moreover, it contains detailed comments for each part. Something that makes it ideal for self-learning is that after each chapter, it offers exercises that people can do themselves.

Bellatrix Getting Started Web

Different features are grouped in separate folders, and at the end of each chapter, there is a TODO file containing the exercises.

Troubleshooting Ease

With increasing test count and complexity, it will be even more critical that the tests be maintainable. A significant part of this effort is easier troubleshooting and better support for locating errors.

A big part of maintainability is troubleshooting existing tests. Most in-house solutions or open-source ones don't provide lots of features to make your life easier. This can be one of the most time-consuming tasks. You can have 100 failing tests and find out whether there is a problem with the test or a bug in the application. If you use plugins or complicated design patterns, debugging the tests will be much harder, requiring lots of resources and expertise.

Two of the ways Bellatrix as full-stack test automation framework handles these problems are through full-page screenshots and video recording on test fail. Again, following the paradigm of similar APIs, we use these features through attributes.

Bellatrix Full-page Screenshots

[TestClass]
[ScreenshotOnFail(true)]
[Browser(BrowserType.Chrome, BrowserBehavior.ReuseIfStarted)]
public class FullPageScreenshotsOnFailTests : WebTest
{
    [TestMethod]
    public void PromotionsPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
        promotionsLink.Click();
    }
}

If you want to read how it works internally, you can read the article: Capture Full Page Screenshots Using WebDriver with HTML2Canvas.js.

Bellatrix Video Recording

[TestClass]
[VideoRecording(VideoRecordingMode.OnlyFail)]
[Browser(BrowserType.Chrome, BrowserBehavior.ReuseIfStarted)]
public class VideoRecordingTests : WebTest
{
    [TestMethod]
    public void PromotionsPageOpened_When_PromotionsButtonClicked()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
        promotionsLink.Click();
    }
}

The video recording works on all OSes. We use FFmpeg video recorder internally. In some of my next articles, I will describe in details how to implement it yourself. It works similarly to the video recording on tests failing that I wrote about in the past.

Library Customization

One of the hardest things to develop is a way to allow these generic frameworks to be extendable and customizable. The whole point of creating a shared library is to be used by multiple teams across the company. However, different teams work in different contexts. They may have to test somewhat different things, so the library code as-is may not be working out of the box for them. Thus, the engineers should be able to customize parts to fit their needs.

Below, you can find a couple of examples of how this is possible with Bellatrix. In the next articles, I will show you how it works if you wish to implement it in your framework.

Override Actions Globally

[TestClass]
[Browser(BrowserType.Firefox, BrowserBehavior.RestartEveryTime)]
public class OverrideGloballyElementActionsTests : WebTest
{
    public override void TestsArrange()
    {
        Button.OverrideClickGlobally = (e) =>
        {
            e.ToExists().ToBeClickable().WaitToBe();
            App.JavaScriptService.Execute("arguments[0].click();", e);
        };
        Anchor.OverrideFocusGlobally = CustomFocus;
    }
    private void CustomFocus(Anchor anchor)
    {
        App.JavaScriptService.Execute("window.focus();");
        App.JavaScriptService.Execute("arguments[0].focus();", anchor);
    }
    [TestMethod]
    public void PurchaseRocketWithGloballyOverridenMethods()
    {
        App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
        Select sortDropDown = App.ElementCreateService.CreateByNameEndingWith<Select>("orderby");     
        Anchor addToCartFalcon9 = 
        App.ElementCreateService.CreateByAttributesContaining<Anchor>("data-product_id", "28").ToBeClickable();
        Span totalSpan = App.ElementCreateService.CreateByXpath<Span>("//*[@class='order-total']//span");
        sortDropDown.SelectByText("Sort by price: low to high");
        addToCartFalcon9.Focus();
        addToCartFalcon9.Click();
        totalSpan.EnsureInnerTextIs("95.00€", 15000);
    }
}

Here, through the OverrideClickGlobally delegate, we change the default behavior of all buttons in the framework without recompiling the library. In the example, instead of using the default WebDriver implementation, we change the Click method to use JavaScript instead. Usually, we execute this once for all tests, so the right place to call the override is in the AssemlyInitialize method of MSTest that is performed once per assembly.

Documentation

Override Actions Locally

Another possible customization is through local override which means that the behavior will be changed only for the currently executing test.

Button.OverrideClickLocally = (e) =>
{
    e.ToExists().ToBeClickable().WaitToBe();
    App.JavaScriptService.Execute("arguments[0].click();", e);
};

Documentation

Extensibility Through Events

public class DebugLoggingButtonEventHandlers : ButtonEventHandlers
{
    protected override void ClickingEventHandler(object sender, ElementActionEventArgs arg)
    {
        DebugLogger.LogInfo($"Before clicking button. Coordinates: X={arg.Element.WrappedElement.Location.X} Y={arg.Element.WrappedElement.Location.Y}");
    }
    protected override void HoveringEventHandler(object sender, ElementActionEventArgs arg)
    {
        DebugLogger.LogInfo($"Before hovering button. Coordinates: X={arg.Element.WrappedElement.Location.X} Y={arg.Element.WrappedElement.Location.Y}");
    }
}
[TestClass]
[Browser(BrowserType.Chrome, BrowserBehavior.RestartEveryTime)]
public class ElementActionHooksTests : WebTest
{
    public override void TestsArrange()
    {
        App.AddElementEventHandler<DebugLoggingButtonEventHandlers>();
    }
    [TestMethod]
    public void PurchaseRocketWithGloballyOverridenMethods()
    {
        // some test logic
    }
}

Another way to extend Bellatrix is to use the controls hooks. This is how BDD logging and highlighting are implemented. For each method of the control, there are two hooks- one that is called before the action and one after. For example, the available hooks for the button are:

  • Clicking - an event executed before button click
  • Clicked - an event executed after the button is clicked
  • Hovering - an event executed before button hover
  • Hovered - an event executed after the button is hovered over
  • Focusing - an event executed before button focus
  • Focused - an event executed after the button is focused

You need to implement the event handlers for these events and subscribe them. Bellatrix gives you again a shortcut- you need to create a class and inherit the {ControlName}EventHandlers. In the example, DebugLogger is called for each button event printing to Debug window the coordinates of the button. You can call external logging provider, making screenshots before or after each action, the possibilities are limitless.

Documentation

There are many more ways how you can enable extensibility and customization of your full-stack test automation framework. You can check how we do it in the Extensibility section of Bellatrix documentation. There will be a whole new series about this feature. There, I will show you how it works internally and how to do it yourself if you want.

Summary

Full-stack test automation frameworks have many features that enable you to work in emerging complex contexts. In the first article about these features, we talked about how necessary is the tooling to be cross-platform and cross-technology ready. These tools should make it easy for you to execute your tests in the cloud so you can test your applications on multiple devices and OSes. A major must-have for such frameworks is to give you ways to find why your tests failed, such as screenshots and videos. Last but not least, since they are used across a wide range of teams' contexts, they should provide ways for customization and extensibility.

Topics:
test automation ,full stack ,devops ,software testing ,cross platform testing

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}