Distributed and Parallel Tests Execution of WebDriver UI Tests
Distributed and Parallel Tests Execution of WebDriver UI Tests
Learn about performing distributed and parallel testing of WebDriver UI tests with the open-source distributed test runner Meissa.
Join the DZone community and get the full member experience.Join For Free
Containerized Microservices require new monitoring. Read the eBook that explores why a new APM approach is needed to even see containerized applications.
This is the third article from the series dedicated to our open-source distributed test runner called Meissa. You can read about our biggest test, executing 100,000 tests under 20 minutes.
Here, I will tell you why we built the tool and show you data from our research. To the end of the publication, you will know the most common ways to execute your tests in parallel or distributed, what are their advantages and disadvantages.
Why Did We Build Meissa?
To understand why we created the tool, I need to tell you what we needed to test. Our company specializes in building test automation framework and tooling for web, mobile, desktop, and APIs. It is not a visual application rather than a set of C# libraries and other tools to speed up the writing of tests scripts. My belief is that, when you build a testing tool, you need to make sure it is covered with lots of tests. For example, for the web part, we support 8 browsers, the five major ones and three headless. The web library uses WebDriver in its core but adds many capabilities on top of it, so we have lots of tests related to Selenium grid mode and the integration with the cloud providers. We have tons of other logic that needs to be tested, so we ended up with over 5000 UI tests. I skip all the unit tests since they execute quickly. Actually, most of the UI tests are quite fast too, since we use Bellatrix to test itself. But, if we execute them sequentially on a single machine, it takes more than 16 hours to execute them.
We need to be able to deliver updates a few times a day so this was unacceptable.
Now with Meissa, we execute them in under 4 hours, and this is without using the in-parallel Bellatrix feature. We only distribute all tests on four virtual machines in Azure.
Why Do You Need to Distribute Your Tests?
In our observation, the optimal number of test run processes is 1.5-2.0 x total number of cores on your machine, which means you are limited to the hardware you use. In the teams where I worked, most of the VMs had up to 2 CPUs. One "big" machine or many smaller ones? To answer this question, I need to explain the difference between horizontal and vertical scale. Horizontal scaling means you scale by adding more machines to your pool of resources, whereas vertical scaling means you scale by adding more power (CPU, RAM) to an existing one. However, typically, many smaller machines are cheaper than one big one. The VM clouds nowadays are the mainstream. Because of that, the price differences are insignificant.
Advantages of Distributed Parallel Test Execution
The most obvious reason is speed. Instead of executing the tests in 16 hours, they complete in under 4 hours. When all required tests can be run a couple of times in a business day, you will be able to release your application every day (if we talk about web projects). Even if it is not web, you can improve the quality of your app and shorten the testing cycle by executing all your tests as part of the CI process. This means you will have higher coverage in shorter throughput time. As you know, each time your tests execute, their ROI (return of investment) increases. Last, the more often you run all your tests, the probability of locating unstable/not-well-written tests rises.
Parallel vs Distributed Test Execution
I can define at least three types of parallel test execution:
Parallel- Single Machine - Multiple Processes
Some unit test frameworks have native parallel execution support like NUnit. You have, for example, 100 tests. If you run them on a single machine with 5 CPU Cores, on each core, 20 tests will be executed simultaneously. However, not all test frameworks support this option, and there are some major problems related to it, depending on the type of tests you want to run.
Distributed Testing- Multiple Machines - Single Process
Your second option is to run your tests at the same time on multiple machines and merge the results at the end. Usually, you need an additional complex tooling, for example, Microsoft Test Controller/Agents setup.
Parallel Distributed Testing- Multiple Machines - Multiple Processes
You can mix both approaches. In this case, you will use the complex tooling and, at the same time, run the tests in parallel on each machine.
Ways to Execute Your Tests in Parallel or Distributed
I had the task to perform research and think of a way to solve the problem. I read everything I could find for a couple of days, besides all the solutions I know from the companies I worked for in the past. The truth is, for seven years, the way people execute their tests has not really changed. Additionally, in my research we asked our followers a similar question in our survey: " Do you run your tests in parallel? If YES, how? " Their answers were almost identical to what I read.
The most common practice is to run your tests in parallel, based on the machine cores, and to leverage on the Selenium Grid if we are talking about UI tests. However, this solution has a couple of drawbacks. For example, most native runners can run tests in parallel only if they are in separate projects/containers/libraries. Meissa can run tests without additional attributes/annotations/configurations added to your source code. Using it makes the onboarding of the parallel tests execution easier and faster. If you use the in-parallel feature of your native runner, excellent! You can use it side by side with Meissa. The tests will be run with the same speed, you will have all the other benefits of Meissa, and most importantly, you will be able to distribute them on multiple machines.
Some popular solutions for testing environments now utilize Docker containers. One of my favorites is Selenoid. It is a Selenium Grid on steroids that gives you clean installations of specific browsers each time in dedicated containers. It is excellent for WebDriver tests if you test on Firefox, Chrome, or Opera. How does the Selenoid tool differ from Meissa? Well, it is not a test runner. You still need a way to execute your tests in parallel if you want to run your tests simultaneously. I considered it during my research. However, there were a couple of limitations that stopped us from using it. The tool doesn't use containers for InternetExplorer and Edge. It doesn't support Appium and running desktop apps. It is built primarily for the major browsers. Of course, it is open-source, and you can create new images on top of the existing ones, but for me, it is too much effort to make it work for these scenarios.
Most points are valid for the standard Selenium grid and its tuned versions, such as Jsonwired-Grid, Gridrouter, Selenograph. They are no test runners; they just give you browsers/devices configurations. If you run your tests on one process, even if you have 100 hub nodes, your tests will be executed one by one. So, you can use these tools together with Meissa to run the tests distributed.
Because of the lack of tooling, some people use their CI systems to execute their tests. The usual practice is to mark each test with attribute/annotation, such as Machine1, Machine2, and then use the slave/build capability of the CI tool to run them on these machines separately. However, for me, this is unacceptable since the "balancing" of the tests is manual and the test execution time is equal to the time of the slowest slave's sub-run. Moreover, when you add or remove tests, you need to calculate that time yourself. With Meissa, the whole process happens automatically without modifications of the tests source code, and the runner produces a single test results file.
Most of the time, I like the Microsoft tools. In my previous companies, we used MS test agents, but they have a few flaws. It is unpredictable how the tests are balanced.. They are hard to set up and troubleshoot, and the documentation is missing almost entirely. Last, they can run tests only on the Windows platform. And if you cancel the build, the machines can be left in an inconsistent state with open browsers and so on. Meissa is cross-platform and simplified to maximum, including everything you need in a single CLI. Moreover, it has the most essential features of the MS distributed tests runner but can handle adequately any cancellations and execution issues.
When Is It Appropriate to Use and When Is It Not?
When you go to buy bread from the bakery, you don't go with a shuttle. Same can be applied here. I believe Meissa is not the appropriate choice in many cases. I will mention a few. If you have only unit tests and they are executed for a few minutes without parallel test execution, then you can use the native tests runners. You don't need to bother with more machines or changing the tool. Another possibility is, if you are happy with your current solution, and you are satisfied with its speed, stability, and usability, you should stick to it. You have built experience using and maintaining it. I am against the usage of new technologies only because they are viral or someone else told me to do so. Maybe you are from this 15% of people that have built something custom and many people/teams in your organization use it. It may be quite time-consuming to change all your CI builds and train all of your colleagues.
Published at DZone with permission of Anton Angelov , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.