Parallel Testing With Robot Framework
Parallel Testing With Robot Framework
Get a grasp of Pabot, a Robot Framework tool which allows you to run your Robot Framework tests in parallel in order to reduce your overall test execution time.
Join the DZone community and get the full member experience.Join For Free
Did you ever encounter the problem that your regression test execution time has become so large, that you needed to wait for hours in order to get the results? This is where Pabot comes to the rescue! Pabot enables parallel test execution for your Robot Framework tests. Let’s see how this works!
This post is the last in a series about Robot Framework, an automation test framework. In case you are not yet familiar with Robot Framework, you can take a look at the previous blogs about this topic:
- Automated Acceptance Testing With Robot Framework
- How to Write Data Driven Tests With Robot Framework
- Create Custom Robot Framework Libraries
It is fairly easy to set up a regression test with Robot Framework. You just run the
robot command and indicate you want to run all the tests in a particular directory. Robot Framework will execute the tests sequentially one after another and the results will be gathered in one output log and report. What will happen when you have a lot of regression tests? The test execution time will take for hours and we do not want that to happen when we want to be able to deliver changes fast to production. What to do in this case? You can limit the number of tests you run in the regression test, but this is not something you want to do and is only a temporary (bad) solution. You can e.g. split the regression test and run it separately on different machines. That could be a solution, but you will need something to gather the results together. Another option is to run the tests in parallel by means of Pabot. Pabot has been developed by Mikko Korpela, a core contributor to Robot Framework. He mainly has been occupied with RIDE (the Robot Framework IDE) development. Documentation and sources of Pabot can be found at GitHub.
In this post, we are going to explore how Pabot can be used.
Sources are available at GitHub.
2. Application Under Test
We need an application to test. We will use a simple Python script
wait.py containing three functions. Each function just waits for a certain amount of time.
We can invoke the functions by means of the command line:
We use a Python script as an application, but this can be any kind of application in any kind of programming language.
3. Our First Pabot Example
3.1 Create the Tests
We also need some tests. We create a separate Test Suite with one Test Case for each function. We create them in the
test directory and name them
test_wait_for_10s.robot. We tag them with the tag
basic, we will create more Test Suites and do not want to execute all of them all of the time. By means of tags, we can limit the tests we want to execute.
test_wait_for_3s.robot contains one Test Case which just invokes the Python command in order to execute the
wait_for_3s function. The other
*.robot files are similar to this one.
Execute the tests from within the root of the project directory and execute all the tests containing tag
basic in directory
All tests pass and when we take a look at the
log.html file, we notice that the elapsed time equals 18 seconds. This is as expected.
3.2 Run the Tests Using Pabot
Let’s see what happens when we run the tests with Pabot. First of all, we need to install Pabot.
We execute the same tests as we did before. Notice that you can use the same options as for the
Some things to notice here:
- Something is being stored in a file
.pabotsuitenames. We will come back to that later on.
- The tests are executed, each in their own process (unique PID’s).
- One log and report file are generated.
- We do not see the test output as we do when running by means of the
- The elapsed time is now only 10 seconds instead of 18 seconds.
log.html file also indicates these statistics. The tests are run in parallel and the elapsed time is determined by the longest running test, 10 seconds in our case.
Some other things to know:
- We ran the Test Suites in parallel, which is the default behavior. It is also possible to run the Test Cases in parallel. Therefore, you can add the option
--testlevelsplitto the command line options. Be aware that you first add the Pabot specific options to the command line and after these the standard
robotcommand line options.
- A directory
pabot_resultshas been created which contains per test a directory with the
output.xml, standard error output and standard output files.
- When adding the Pabot option
--verboseyou can view the console output of the tests being executed just like running the
4. Influence Execution of Tests
When you start running tests in parallel, it might be that some tests need to run in a specific order. It might be that tests are dependent on each other (intended or not intended) or some tests just need to run before other tests for some reason. It is possible to influence the execution order of the tests. This brings us back to the
.pabotsuitenames file we talked about earlier. Let’s take a look at the contents of this file:
The first four lines are not of interest for us, but the lines following contain the order of test execution.
The command line option
--ordering allows us to influence the test execution. We only need a file following the syntax of the
.pabotsuitenames file without the first four lines. We create a file
.pabotsuitenames-ordering-wait where we ensure that the 10 seconds test is executed before running any other test. We use the
#WAIT keyword for that:
Run the tests again with the predefined file:
In the console log, we clearly notice that the 10 seconds test must be ended before running any other test. The elapsed time is about 16 seconds (10 seconds for the first test and approximately 6 seconds for the 3 seconds and 5 seconds test which ran in parallel).
Sometimes it is required that tests are not allowed to run in parallel. We therefore can add a directory to the ordering file. All tests inside that directory will run sequentially. In order to verify this behavior, we add a directory
sequentially which contains copies of our three tests. We tag these tests with the
sequentially tag. The directory tree of the
test directory is the following:
We create a file
.pabotsuitenames-ordering-dir-sequentially and add this directory to the list of to be executed tests:
Execute the test:
We notice that all four tests (the three basic tests and the sequentially directory) are executed in parallel. The test in the
sequentially directory lasts for 18 seconds and that is exactly the elapsed time of the complete test execution. This can also be seen in the
Up till now, all available processes are being used, but this can also be limited. Therefore, use the
--processes command line option.
5. The Pabot Library
Running tests in parallel can cause some problems when the same resources are being used by different tests. This can cause unwanted side effects and influence the tests results of the different tests. The Pabot Library contains keywords in order to be able to cope with these kind of problems. We will explore one use case where we will set a lock upon a certain resource.
We copy the contents of the
test_wait_for_3s.robot file to files
test_acquire_lock2.robot. We tag them with the
lock tag. Assume that the function
wait_for_3s contains processing which causes problems when being executed in parallel (e.g. it writes or reads to the same file). We can solve this by using the
Acquire Lock and
Release Lock keywords from the Pabot library. With
Acquire Lock we request a lock on this part of the test and no other test will be able to acquire the lock with an identical name until the lock has been released. Besides that, we also need to import the
PabotLib library. We add the same locks to both test files.
Run the tests including the
--pabotlib command line option.
We notice that the elapsed time is almost 7 seconds, where we would have an elapsed time of about 3 seconds without any locking. The
log.html shows this even more clearly: The
Test Acquire Lock2 test lasts for 6 seconds whereas the
Test Acquire Lock1 test lasts for 3 seconds.
Pabot is a great tool for running your tests in parallel and to speed up the overall test execution. Just like running code in parallel, it can cause some challenges when you want to run your tests in parallel. Luckily, some features are available in order to overcome these challenges. However, beware that you will need to spend some effort in changing your Test Suites in order to be able to make full use of parallel test execution.
Published at DZone with permission of Gunter Rotsaert , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.