TestOps Infrastructure From Beginning to End: Theoretical Aspects (Part 1)

Building a TestOps infrastructure from scratch or with little knowledge is a tricky challenge to face. Part 1 of this article is more on the theoretical side; Part 2 will cover case studies that could be integrated into your project.

Here are the most valuable features we are looking for when building a TestOps infrastructure.

1. Coverage of Multiple Platforms (Web, Mobile, API, Console)

The customer expects to have a cross-platform solution. Today, most of our customers have different clients — mobile, web, and REST API, and in some cases, even gaming consoles. Therefore, to make our tests truly integrational, we’ve developed our own all-in-one solution called Carina, which is completely open-source. Actually, you can use any framework you like, though using more complex solutions saves time and budget.

2. Test Maintainability Based on OOP Best Practices

Concerning test maintainability, you should stick to the main practices and good projecting patterns based on object-oriented programming. The latest trend is functional testing using page objects. From the proof-of-concept point-of-view, it may be right, but from the experience in real production, those tests are impossible to maintain.

I highly recommend using an abstraction of UI level for mobile automation when iOS and Android clients are quite alike. The abstraction of this layer with the corresponding iOS or Android implementation lets you reuse a huge amount of source code without making drastic changes in it.

Use page objects and UI components as much as you can to have the clear structure of your tests and to transfer the main business logic and locators onto page objects. Using UI components, some basic elements like navigation menus, various controls that are present on the pages, or screenshots of the app should be gathered into separate classes and separate UI objects.

The combination of REST calls and UI validations makes tests more stable. To implement this approach, we need another approach with data parametrization with the use of data sources:

• Parametrize tests from Excel and databases, making it possibile to reuse the same source code on different data sets;
• You get a large amount of tests, giving the opportunity to check different domain areas of the product you test.
• Combine REST API calls with the UI validations to make tests more fast and more stable.
  

3. Transparency of Automation Results for Management and Development

Managers like visual representation of automation results. Charts and e-mail-able reports are a must. Concerning visibility, some teams practice a certain approach when automation stands away from development and manual QA. When automation engineers get certain tasks, they automate a number of test cases, run them, control the results, do bug fixing themselves, and so on, which happens to be not the best way to set up a process. My opinion is that you need much more visibility in this process. First of all, it’s giving management what they want: boring tables representing the amounts of failed or passed tests, random stack traces tell them nothing. I recommend using the tools with e-mailable reports on a daily basis and various types of visualizations like graphs and tables that are clear to the management.

Development should get results ASAP; ideally, automation is integrated into the CI/CD process. Try to gain loyalty from your development team. Explain what your automated tests do and how to integrate the test running process into your CI. Make small test suites running for five to seven minutes or even faster (if it’s a REST test, for example). If it’s UI integration tests, they may take a little longer to run; give them an opportunity to understand and get those results as fast as possible. Developers enjoy this approach and it will lead to a loyal relationship between them and a QA automation department.

Manual QA should be able to read and interact with automation results. Give them a tool to check test results like artifacts, logs, or screenshots and an opportunity to interact with this stuff because you know yourselves that tests fail due to the weak performance of a testing environment or infrastructure issues with Selenium or Appium, for example. 5-10% of tests may fail, and if you’re going to manually check every test run, you’ll spend too much time on maintenance. Those tasks can be curated by your manual QA engineers.

All those parameters, visualizations, and reports let your QA management better track your velocity and the exact amount of tasks you are able to complete for a certain period of time, so further tasks would be more appropriate and you manage to complete them in time.

4. Full Integration Into the CI/CD Process

Automation is triggered on commit, on a regular basis or manually. Concerning integration into the CI process, you should let your test automation be triggered on a commit basis or a pull request basis, have the opportunity to start tests manually or on a regular basis (e.g. nightly regression or release regression), and so on. 

Tests are smartly grouped based on tested features and scope. Organize tests in an intellectual way — not only by the amount of tests like minimal acceptance, acceptance, and regression, but let there be opportunity to test features. Form test suites containing tests on profile management or user management so that if developers are doing bug fixing or writing new functionality, they can run a certain amount of tests and don’t have to wait long when regression or acceptance tests are running.

Flexible configuration allows for selecting the AUT environment, execution environment, etc. Allow for the possibility to run tests with flexible configurations: an option to choose an environment, browser, or mobile device on which a test should be run.

Everyone has experienced a situation when during the regression, five or 10 percent of tests fail, and you realize that it was due to the Selenium issues. For example, 50 tests out of 300 failed, and there is no use in rerunning the whole regression. In our open-source products, we’ve built a logic allowing you to rerun only failed tests or their sequences and to aggregate reports from two separate runs.

Open-source instruments on the market we recommend:

• We use Selenium for web automation and Appium for mobile automation.
• For the test lifecycle control, we use TestNG. In our opinion, it’s more flexible compared to JUnit — it allows to group tests and to set TestNG XMLs. We also use the Maven SureFire Plugin, which allows for running certain groups of tests under CI to parametrize them and to get the corresponding artifacts in the end.
• For real-time or historical reporting, we use RabbitMQ proper, and Elastic Stack for archiving those logs and searching through the archives.

Concerning the test execution environment, we use Selenoid, which is a new generation of SeleniumHub allowing us to run every testing session in a dockerized environment and in a separate Docker container, be it web tests on the most popular browsers like Chrome or Firefox, or mobile.

Our mobile automation is currently being run in an MCloud device farm — it’s our own internal project — a real device farm held in storages in our office. They’re connected to a server through hubs. The farm is used by both manual and automation QA engineers.

And Open STF is an excellent open-source tool allowing you to control the aggregation of the outside devices. Through the web interface, you can interact with devices: swipe, type, click, install apps, and so on.

For orchestration of the infrastructure components, we use Ansible. With the help of pre-made playbooks in YAML format, it allows us to deploy and install packages through SSH, to administer the access rights, to create files and configurations, and so on. Ansible allows us to do the above things automatically and not to spend too much time on them.

Our whole testing infrastructure, be it SeleniumGrid or AppiumGrid, is built in a dockerized environment — this means we use Docker containers. If the infrastructure is placed on platforms like Amazon or Google Cloud, you can administer the whole infrastructure by the means of Kubernetes.

We use Jenkins as a CI tool ubiquitously; it’s the most suitable open-source CI/CD tool. With the help of pipelines, we arrange test runs after a dev build and set up various configurations.

This was an outline of the theoretical part. The next part would consist of diving into the set-up process and pointing out the things which are crucial for the whole system to function properly.