{{announcement.body}}
{{announcement.title}}

How to Test the Test Automation Framework: Types of Tests

DZone 's Guide to

How to Test the Test Automation Framework: Types of Tests

Before you use the test automation framework, it's important to go through a number of tests for the framework itself.

· DevOps Zone ·
Free Resource

Nowadays, more and more companies are building test automation frameworks based on WebDriver and Appium for testing their web and mobile projects. A big part of why there are so many flaky tests is that we don't treat our tests as production code. Moreover, we don't treat our framework as a product. I will present to you all the challenges during the testing of our test automation framework. In the first article from the series, I gave you an overview of our test infrastructure like what tools we use for source control, packages storage, CI and test results visualization. In this second part, we will discuss various types of testing that we did to verify that each aspect of our framework is working as expected — functional, compatibility, integration, usability, installability and many more.

What We Have to Test

Before I share the details of our test infrastructure, I need to explain what we have to test. Our test automation framework is written on .NET Core in C#. This way we achieve the cross-platform requirement. It also has modules for API, desktop, web, Android, iOS and load testing. Generally, there are two ways to use the framework. Most of our clients use it by installing NuGet packages. (NuGet is the package manager for .NET. NuGet client tools provide the ability to produce and consume packages.) To ease the process of configuring the projects we provide a Windows UI installer and CLI installer for Windows and OSX. The UI installer also adds various projects, item templates, and snippets for better user experience in Visual Studio IDE.

The second way of using the framework is if you are an enterprise user and have access to the full source code. The usage is slightly different, since we have removed all security protections like licensing and obfuscation. So, we need to test it separately.

Functional Testing

Most of our functional tests are written with our framework against demo apps. For the web, for example, we have developed mocked local web pages so that the tests can be as fast as possible. The same is valid for API tests which are run against a local web service run together with the tests. As for the mobile tests, we tried different approaches — real devices, emulators, simulators and cloud providers. We wanted to execute them using the cloud providers, but the tests were slower and if you use a service which provides real devices, the device is not always free. So, we decided for beginning to use emulators and simulators in case of iPhone.

The first type of functional tests we have is to check whether the framework can work with the various UI controls — buttons, text boxes, date pickers, etc.

[TestMethod]
[TestCategory(Categories.Chrome)]
public void DateSet_When_UseSetDateMethodWithDayBiggerThan9_Chrome()
{
   var dateElement = App.ElementCreateService.CreateById<Date>("myDate");
   dateElement.SetDate(2017, 11, 15);
   Assert.AreEqual("2017-11-15", dateElement.GetDate());
}


The way we developed our framework if it is not used correctly, it throws the appropriate exception with a specific message mentioning what went wrong and what to do. This is something that needs to be validated.

[TestMethod]
public void DateSetThrowsArgumentException_When_Month0_Edge()
{
   var dateElement = App.ElementCreateService.CreateById<Date>("myDate");
   Assert.ThrowsException<ArgumentException>(() => dateElement.SetDate(2017, 0, 1));
}


As mentioned one of the key things that differ the frameworks from the software libraries is the extensibility. For each control, we support different hooks/events to which you can subscribe and execute your logic. Here is how we test it.

[TestMethod]
public void SettingDateCalled_BeforeActuallySetDate()
{
   Date.SettingDate += AssertValueAttributeEmpty;
   var dateElement = App.ElementCreateService.CreateById<Date>("myDate");
   dateElement.SetDate(2017, 7, 6);
   Assert.AreEqual("2017-07-06", dateElement.GetDate());
   Date.SettingDate -= AssertValueAttributeEmpty;
   void AssertValueAttributeEmpty(object sender, ElementActionEventArgs args)
   {
       Assert.AreEqual(string.Empty, args.Element.WrappedElement.GetAttribute("value"));
   }
}


We have another logic which allows the user to change globally how a particular method behaves. For example, if you are not happy with how we have implemented the button.Click()   then you can change how it is done — for example, using JavaScript.

[TestMethod]
public void LocallyOverriddenActionCalled_When_OverrideGetDateGloballyNotNull()
{
   Date.OverrideGetDateGlobally = (u) => "2017-06-01";
   var dateElement = App.ElementCreateService.CreateById<Date>("myDate");
   Date.OverrideGetDateLocally = (u) => "2017-07-01";
   Assert.AreEqual("2017-07-01", dateElement.GetDate());
   Date.OverrideGetDateGlobally = null;
}


For some features, such as BDD logging, instead of running browsers and devices, we use pure unit tests with mocking some of the dependencies. This is a feature that once the test completes, creates a human-readable log what happened.


private Mock<IBellaLogger> _mockedLogger;
public override void TestInit()
{
   _mockedLogger = new Mock<IBellaLogger>();
   App.RegisterInstance(_mockedLogger.Object);
}
[TestMethod]
public void CreatesBDDLog_When_ClickButton()
{
   var button = App.ElementCreateService.CreateByIdContaining<Button>("button");
   button.Click();
   _mockedLogger.Verify(x => x.LogInformation(
      It.Is<string>(t => t.Contains("Click control (ID = button)"))), Times.Once());
}


Here we use a mock to verify that the correct message is saved.

Another type of tests is where we check the localization of elements. In the case of Android, we have an advanced logic which will scroll down until the element is visible before performing the action.


[TestClass]
[Android(Constants.AndroidNativeAppPath,
   Constants.AndroidDefaultAndroidVersion,
   Constants.AndroidRealDeviceName,
   Constants.AndroidNativeAppAppExamplePackage,
   ".view.ControlsMaterialDark",
   AppBehavior.RestartEveryTime)]
public class ElementCreateSingleElementTests : AndroidTest
{
   [TestMethod]
   public void ElementFound_When_CreateByText_And_ElementIsNotOnScreen()
   {
       var textField = _mainElement.CreateByText<TextField>("Text appearances");
       textField.EnsureIsVisible();
   }
}


Learning Tests

What are the learning tests? These are not tests that test your product or framework directly. Instead, if your framework uses a third-party library such as WebDriver or Appium to automate the products, then you probably know that the usage changes from time to time and not everything is so stable as you wished. The learning tests are tests that use these low-level libraries in the ways you will use them later in your framework. Once you make these tests pass, then you can verify whether the upgrade of the low-level library was successful or not.

[TestMethod]
public void OrientationTest()
{
   IRotatable rotatable = ((IRotatable)_driver);
   rotatable.Orientation = ScreenOrientation.Portrait;
   Assert.AreEqual(ScreenOrientation.Portrait, rotatable.Orientation);
}
[TestMethod]
public void LockTest()
{
   _driver.Lock();
   Assert.AreEqual(true, _driver.IsLocked());
   _driver.Unlock();
   Assert.AreEqual(false, _driver.IsLocked());
}


Features Not to Automate

Many features are hard to be automated, or even if it is doable, the ROI is too high. We have different approaches to such functions. For example, we have the troubleshooting features part of the framework that took screenshots and videos in case of test failure. Instead of trying to automate whether the video was OK or not. We use these features ourselves during the process of analyzation of failed tests if there are any.

[VideoRecording(VideoRecordingMode.OnlyFail)]
[Android(Constants.AndroidNativeAppPath,
   Constants.AndroidDefaultAndroidVersion,
   Constants.AndroidDefaultDeviceName,
   Constants.AndroidNativeAppAppExamplePackage,
   ".view.Controls1",
   AppBehavior.ReuseIfStarted)]
public class VideoRecordingTests : AndroidTest


Image title

Upgradability Testing

Once there is a new version of Appium, WebDriver or any other third-party library we use, we first execute all learning tests. If there are any problems, we investigate and try to work around them. If we cannot, we submit an issue to the repository maintainers. If all learning tests are green, then we upgrade our code and execute all other tests to find out that the new upgrade is working.

All APIs change so does the API of our framework. We try not to introduce breaking changes, but we leave the old API for a couple of releases, mark it as obsolete and give instructions on what needs to be changed.

It is a good practice not to delete the old versions of the API immediately; instead, give your users time to migrate their code to the new versions. In such cases, we mark the old API with the ObsoleteAttribute by providing explanations for why the code is deprecated and what should be used instead.

Here is an example from the official WebDriver C# bindings GitHub repository.

public interface ITimeouts
{
   TimeSpan ImplicitWait { get; set; }

   [Obsolete("This method will be removed in a future version. Please set the ImplicitWait property instead.")]
   ITimeouts ImplicitlyWait(TimeSpan timeToWait);
}


Installability Testing

As mentioned we need to make sure the UI installer is working correctly. To do so, we have a checklist of things to verify that everything is running smoothly.

Also, as I told you previously, we have CLI installers for both platforms.

dotnet new -i Bellatrix.Web.GettingStarted
dotnet new Bellatrix.Web.GettingStarted


This is how the project is created in an empty folder. To test this behavior we created CI builds that perform it and verify that the projects can be built after the installation.

Image title


Portability Testing

Portability means "the ease with which the software product can be transferred from one hardware or software environment to another."

In the case of frameworks, it means can you run the same tests using a different browser or Android version, screen size, etc and the test to continue working. We promise such behavior to our users, so we need to check it.

[TestClass]
[Android(Constants.AndroidNativeAppPath,
   Constants.AndroidDefaultAndroidVersion,
   Constants.AndroidGalaxyDeviceName,
   Constants.AndroidNativeAppAppExamplePackage,
   ".view.Controls1",
   AppBehavior.ReuseIfStarted)]
public class ButtonControlTests : AndroidTest
[TestClass]
[Android(Constants.AndroidNativeAppPath,
   Constants.AndroidDefaultAndroidVersion,
   Constants.AndroidNokiaDeviceName,
   Constants.AndroidNativeAppAppExamplePackage,
   ".view.Controls1",
   AppBehavior.ReuseIfStarted)]
public class ButtonControlTests : AndroidTest


Interoperability Testing

What is interoperability? Here is the official definition: "The degree to which two or more components or systems can exchange information and use the information that has been exchanged."

This includes all features for tight integration with programming IDEs, like project or item templates, snippets and so on. These are from the features that we don't strive to automate. We can, but I have the feeling that the tests will be more complicated than needed. So, as with the installer, we perform most of these test cases manually.

Image title


Performance Testing

How do you do performance testing in the context of a test automation framework?

Especially for our mobile testing modules, we make sure that the tests are executed under a particular time. Test frameworks such as NUnit and MSTest give us the ability to fail tests if they are run over a specified amount of time.

[TestClass]
[IOS(Constants.IOSNativeAppPath,
   Constants.IOSDefaultVersion,
   Constants.IOSDefaultDeviceName,
   AppBehavior.RestartEveryTime)]
public class ButtonControlTests : IOSTest
{
   [TestMethod]
   [Timeout(30000)]
   public void ZeroReturnForButtonText_When_CallClickMethod()
   {
       var button = App.ElementCreateService.CreateByName<Button>("ComputeSumButton");
       button.Click();
       var answerLabel = App.ElementCreateService.CreateByName<Label>("Answer");
       answerLabel.EnsureTextIs("0");
   }


Security Testing

In our case, we have a licensing service integration in the framework, so we have covered it with lots of tests to be sure everything is running smoothly. Besides that, we use obfuscation for the DLLs (part of the NuGet packages) so that nobody can decompile them. It is doable again to create such automated tests. However, we check whether all of this is working from time to time by trying to hack them ourselves using some favorite tools.

Image title


Usability, Understandability, Learnability Testing

To check whether our framework is easy to use or how easy it is to learn to work with. We do a couple of things. First, we perform alfa and beta testing with chosen experts. They give us feedback before we release the final version of the API.

Another thing you can do is to hire professionals to evaluate the usability of the framework. Some of them work with a network of consultants and domain experts who evaluate the tool. At the end, they send you a report with the notes and suggestions.

Suitability Testing

Suitability testing is the capability of the software product to provide an appropriate set of functions for specified tasks and user objectives.

You can build the perfect library or framework, but if it doesn't do what your users want, then its existence is questionable. This is why we do a couple of things. Client interviews, asking what problems do they have and how they believe should be solved. If we don't have the chance to talk in person, then we send questionnaires and feedback forms, asking similar questions. Based on this feedback, the framework evolves meeting more closely the users' expectations.

Syntax Testing

Documentation is not enough. I bet that most of you that have more than few years of experience in the field of automated testing and have a custom framework spent countless hours teaching new colleagues how to write "proper" tests using the team's framework. I believe that there should be a more automated process allowing people to learn by themselves. I think one way to do it is to utilize getting started solutions or starter kits. Projects that have examples with explanations on how to write tests and why we use a particular method or not.

For each test technology, we created a similar starter kit for Bellatrix. Each of them explains the features of the framework with detailed real-world examples. Moreover, it contains detailed comments for each part. Something that makes it ideal for self-learning is that after each chapter, it offers exercises that people can do themselves.

Image title


We need to make sure that there aren't any syntax, grammar, and similar errors. We do this kind of testing using tools such as Grammarly.

Image title


Static Analysis

We use static analysis to make sure that all developers follow the same coding standards and best practices. We use two tools- StyleCop and editorConfig.

Image title


Both on built produce warnings and "warn" you when there is a problem with your code so that you can fix it fast. You can view all warnings in the Error List window. Once you double-click on a warning, the file where the issue is situated is opened and selected.



If you enjoyed this article and want to learn more about Automated Testing, check out this collection of tutorials and articles on all things Automated Testing.

Topics:
devops ,testing ,test automation ,testing framework ,automated testing ,testing tools

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}