{{announcement.body}}
{{announcement.title}}

Parallel Testing With Robot Framework

DZone 's Guide to

Parallel Testing With Robot Framework

Get a grasp of Pabot, a Robot Framework tool which allows you to run your Robot Framework tests in parallel in order to reduce your overall test execution time.

· Performance Zone ·
Free Resource

Did you ever encounter the problem that your regression test execution time has become so large, that you needed to wait for hours in order to get the results? This is where Pabot comes to the rescue! Pabot enables parallel test execution for your Robot Framework tests. Let’s see how this works!

1. Introduction

This post is the last in a series about Robot Framework, an automation test framework. In case you are not yet familiar with Robot Framework, you can take a look at the previous blogs about this topic:

It is fairly easy to set up a regression test with Robot Framework. You just run the robot command and indicate you want to run all the tests in a particular directory. Robot Framework will execute the tests sequentially one after another and the results will be gathered in one output log and report. What will happen when you have a lot of regression tests? The test execution time will take for hours and we do not want that to happen when we want to be able to deliver changes fast to production. What to do in this case? You can limit the number of tests you run in the regression test, but this is not something you want to do and is only a temporary (bad) solution. You can e.g. split the regression test and run it separately on different machines. That could be a solution, but you will need something to gather the results together. Another option is to run the tests in parallel by means of Pabot. Pabot has been developed by Mikko Korpela, a core contributor to Robot Framework. He mainly has been occupied with RIDE (the Robot Framework IDE) development. Documentation and sources of Pabot can be found at GitHub.

In this post, we are going to explore how Pabot can be used.

Sources are available at GitHub.

2. Application Under Test

We need an application to test. We will use a simple Python script wait.py containing three functions. Each function just waits for a certain amount of time.

Python
 




x
21


 
1
import sys
2
import time
3
 
4
def wait_for_3s():
5
    time.sleep(3)
6
 
7
def wait_for_5s():
8
    time.sleep(5)
9
 
10
def wait_for_10s():
11
    time.sleep(10)
12
 
13
if __name__ == '__main__':
14
    actions = {'wait_for_3s': wait_for_3s,
15
               'wait_for_5s': wait_for_5s,
16
               'wait_for_10s': wait_for_10s}
17
 
18
    action = sys.argv[1]
19
    args = sys.argv[2:]
20
    actions[action](*args)
21
 
          


We can invoke the functions by means of the command line:

Shell
 




xxxxxxxxxx
1


 
1
$ python3 wait.py wait_for_3s


We use a Python script as an application, but this can be any kind of application in any kind of programming language.

3. Our First Pabot Example

3.1 Create the Tests

We also need some tests. We create a separate Test Suite with one Test Case for each function. We create them in the test directory and name them test_wait_for_3s.robottest_wait_for_5s.robot and test_wait_for_10s.robot. We tag them with the tag basic, we will create more Test Suites and do not want to execute all of them all of the time. By means of tags, we can limit the tests we want to execute.

The test_wait_for_3s.robot contains one Test Case which just invokes the Python command in order to execute the wait_for_3s function. The other *.robot files are similar to this one.

Plain Text
 




xxxxxxxxxx
1
12


1
| *** Settings ***   |
2
| Documentation      | Test the function wait_for_3s of the wait.py Python script
3
| Library            | OperatingSystem
4
 
5
| *** Variables ***  |
6
| ${APPLICATION}     | python3 wait.py
7
 
8
| *** Test Cases ***            |                 |
9
| Wait For 3s                   | [Documentation] | Just call the function wait_for_3s
10
|                               | [Tags]          | basic
11
| | ${rc}                       | ${output} =     | Run and Return RC and Output | ${APPLICATION} wait_for_3s
12
| | Should Be Equal As Integers | ${rc}           | 0


Execute the tests from within the root of the project directory and execute all the tests containing tag basic in directory test:

Shell
 




xxxxxxxxxx
1


 
1
$ robot --include basic test


All tests pass and when we take a look at the log.html file, we notice that the elapsed time equals 18 seconds. This is as expected.

3.2 Run the Tests Using Pabot

Let’s see what happens when we run the tests with Pabot. First of all, we need to install Pabot.

Shell
 




xxxxxxxxxx
1


1
$ pip install -U robotframework-pabot


We execute the same tests as we did before. Notice that you can use the same options as for the robot command.

Shell
 




xxxxxxxxxx
1
16


1
$ pabot --include basic test
2
Storing .pabotsuitenames file
3
2020-05-23 10:17:03.469249 [PID:10404] [0] [ID:1] EXECUTING Test.Test Wait For 3S
4
2020-05-23 10:17:03.470496 [PID:10407] [2] [ID:2] EXECUTING Test.Test Wait For 5S
5
2020-05-23 10:17:03.470529 [PID:10410] [1] [ID:0] EXECUTING Test.Test Wait For 10S
6
2020-05-23 10:17:06.780431 [PID:10404] [0] [ID:1] PASSED Test.Test Wait For 3S in 3.3 seconds
7
2020-05-23 10:17:08.790471 [PID:10407] [2] [ID:2] PASSED Test.Test Wait For 5S in 5.3 seconds
8
2020-05-23 10:17:13.805348 [PID:10410] [1] [ID:0] PASSED Test.Test Wait For 10S in 10.3 seconds
9
3 critical tests, 3 passed, 0 failed
10
3 tests total, 3 passed, 0 failed
11
===================================================
12
Output: .../MyPabotPlanet/output.xml
13
Log: .../MyPabotPlanet/log.html
14
Report: .../MyPabotPlanet/report.html
15
Total testing: 18.89 seconds
16
Elapsed time: 10.46 seconds


Some things to notice here:

  • Something is being stored in a file .pabotsuitenames. We will come back to that later on.
  • The tests are executed, each in their own process (unique PID’s).
  • One log and report file are generated.
  • We do not see the test output as we do when running by means of the robot command.
  • The elapsed time is now only 10 seconds instead of 18 seconds.

The log.html file also indicates these statistics. The tests are run in parallel and the elapsed time is determined by the longest running test, 10 seconds in our case.

Some other things to know:

  • We ran the Test Suites in parallel, which is the default behavior. It is also possible to run the Test Cases in parallel. Therefore, you can add the option --testlevelsplit to the command line options. Be aware that you first add the Pabot specific options to the command line and after these the standard robot command line options.
  • A directory pabot_results has been created which contains per test a directory with the output.xml, standard error output and standard output files.
  • When adding the Pabot option --verbose you can view the console output of the tests being executed just like running the robot command.

4. Influence Execution of Tests

When you start running tests in parallel, it might be that some tests need to run in a specific order. It might be that tests are dependent on each other (intended or not intended) or some tests just need to run before other tests for some reason. It is possible to influence the execution order of the tests. This brings us back to the .pabotsuitenames file we talked about earlier. Let’s take a look at the contents of this file:

Plain Text
 




xxxxxxxxxx
1


1
datasources:3fd963ff61bab32d9113697e3e5a37fb84622e23
2
commandlineoptions:97d170e1550eee4afc0af065b78cda302a97674c
3
suitesfrom:no-suites-from-option
4
file:9314425a8c8b733b8b2316174da893e7ab2a8cee
5
--suite Test.Basic Example.Test Wait For 10S
6
--suite Test.Basic Example.Test Wait For 3S
7
--suite Test.Basic Example.Test Wait For 5S


The first four lines are not of interest for us, but the lines following contain the order of test execution.

The command line option --ordering allows us to influence the test execution. We only need a file following the syntax of the .pabotsuitenames file without the first four lines. We create a file .pabotsuitenames-ordering-wait where we ensure that the 10 seconds test is executed before running any other test. We use the #WAIT keyword for that:

Plain Text
 




xxxxxxxxxx
1


1
--suite Test.Test Wait For 10S
2
#WAIT
3
--suite Test.Test Wait For 3S
4
--suite Test.Test Wait For 5S


Run the tests again with the predefined file:

Shell
 




xxxxxxxxxx
1
15


1
$ pabot --ordering .pabotsuitenames-ordering-wait --include basic test
2
2020-05-24 16:11:30.239398 [PID:6353] [0] [ID:0] EXECUTING Test.Test Wait For 10S
3
2020-05-24 16:11:40.568166 [PID:6353] [0] [ID:0] PASSED Test.Test Wait For 10S in 10.3 seconds
4
2020-05-24 16:11:40.667778 [PID:6406] [0] [ID:1] EXECUTING Test.Test Wait For 3S
5
2020-05-24 16:11:40.668592 [PID:6408] [1] [ID:2] EXECUTING Test.Test Wait For 5S
6
2020-05-24 16:11:43.978060 [PID:6406] [0] [ID:1] PASSED Test.Test Wait For 3S in 3.3 seconds
7
2020-05-24 16:11:45.983518 [PID:6408] [1] [ID:2] PASSED Test.Test Wait For 5S in 5.3 seconds
8
3 critical tests, 3 passed, 0 failed
9
3 tests total, 3 passed, 0 failed
10
===================================================
11
Output:  .../MyPabotPlanet/output.xml
12
Log:     .../MyPabotPlanet/log.html
13
Report:  .../MyPabotPlanet/report.html
14
Total testing: 18.90 seconds
15
Elapsed time:  15.87 seconds


In the console log, we clearly notice that the 10 seconds test must be ended before running any other test. The elapsed time is about 16 seconds (10 seconds for the first test and approximately 6 seconds for the 3 seconds and 5 seconds test which ran in parallel).

Sometimes it is required that tests are not allowed to run in parallel. We therefore can add a directory to the ordering file. All tests inside that directory will run sequentially. In order to verify this behavior, we add a directory sequentially which contains copies of our three tests. We tag these tests with the sequentially tag. The directory tree of the test directory is the following:

Plain Text
 




xxxxxxxxxx
1


 
1
.
2
├── sequentially
3
   ├── test_sequentially_10s.robot
4
   ├── test_sequentially_3s.robot
5
   └── test_sequentially_5s.robot
6
├── test_wait_for_10s.robot
7
├── test_wait_for_3s.robot
8
└── test_wait_for_5s.robot


We create a file .pabotsuitenames-ordering-dir-sequentially and add this directory to the list of to be executed tests:

Plain Text
 




xxxxxxxxxx
1


1
--suite Test.Sequentially
2
--suite Test.Test Wait For 10S
3
--suite Test.Test Wait For 3S
4
--suite Test.Test Wait For 5S


Execute the test:

Shell
 




xxxxxxxxxx
1
18


 
1
$ pabot --ordering .pabotsuitenames-ordering-dir-sequentially --include basic --include sequentially test
2
2020-05-24 16:37:56.565453 [PID:7694] [0] [ID:1] EXECUTING Test.Test Wait For 10S
3
2020-05-24 16:37:56.566232 [PID:7696] [1] [ID:2] EXECUTING Test.Test Wait For 3S
4
2020-05-24 16:37:56.566397 [PID:7698] [2] [ID:3] EXECUTING Test.Test Wait For 5S
5
2020-05-24 16:37:56.566463 [PID:7703] [3] [ID:0] EXECUTING Test.Sequentially
6
2020-05-24 16:37:59.977723 [PID:7696] [1] [ID:2] PASSED Test.Test Wait For 3S in 3.4 seconds
7
2020-05-24 16:38:01.987965 [PID:7698] [2] [ID:3] PASSED Test.Test Wait For 5S in 5.4 seconds
8
2020-05-24 16:38:06.897472 [PID:7694] [0] [ID:1] PASSED Test.Test Wait For 10S in 10.3 seconds
9
2020-05-24 16:38:11.624207 [PID:7703] [3] [ID:0] still running Test.Sequentially after 15.0 seconds
10
2020-05-24 16:38:15.035548 [PID:7703] [3] [ID:0] PASSED Test.Sequentially in 18.4 seconds
11
6 critical tests, 6 passed, 0 failed
12
6 tests total, 6 passed, 0 failed
13
===================================================
14
Output: .../MyPabotPlanet/output.xml
15
Log: .../MyPabotPlanet/log.html
16
Report: .../MyPabotPlanet/report.html
17
Total testing: 37.50 seconds
18
Elapsed time: 18.58 seconds


We notice that all four tests (the three basic tests and the sequentially directory) are executed in parallel. The test in the sequentially directory lasts for 18 seconds and that is exactly the elapsed time of the complete test execution. This can also be seen in the log.html file:

Up till now, all available processes are being used, but this can also be limited. Therefore, use the --processes command line option.

5. The Pabot Library

Running tests in parallel can cause some problems when the same resources are being used by different tests. This can cause unwanted side effects and influence the tests results of the different tests. The Pabot Library contains keywords in order to be able to cope with these kind of problems. We will explore one use case where we will set a lock upon a certain resource.

We copy the contents of the test_wait_for_3s.robot file to files test_acquire_lock1.robot and test_acquire_lock2.robot. We tag them with the lock tag. Assume that the function wait_for_3s contains processing which causes problems when being executed in parallel (e.g. it writes or reads to the same file). We can solve this by using the Acquire Lock and Release Lock keywords from the Pabot library. With Acquire Lock we request a lock on this part of the test and no other test will be able to acquire the lock with an identical name until the lock has been released. Besides that, we also need to import the PabotLib library. We add the same locks to both test files.

Plain Text
 




xxxxxxxxxx
1
15


1
| *** Settings ***   |
2
| Documentation      | Test the function wait_for_3s of the wait.py Python script in combination with a Lock
3
| Library            | OperatingSystem
4
| Library            | pabot.PabotLib
5
 
6
| *** Variables ***  |
7
| ${APPLICATION}     | python3 wait.py
8
 
9
| *** Test Cases ***            |                 |
10
| Wait For 3s                   | [Documentation] | Just call the function wait_for_3s
11
|                               | [Tags]          | lock
12
| | Acquire Lock                | Lock On 3s
13
| | ${rc}                       | ${output} =     | Run and Return RC and Output | ${APPLICATION} wait_for_3s
14
| | Should Be Equal As Integers | ${rc}           | 0
15
| | Release Lock                | Lock On 3s


Run the tests including the --pabotlib command line option.

Shell
 




xxxxxxxxxx
1
15


1
$ pabot --pabotlib --include lock test
2
Robot Framework remote server at 127.0.0.1:8270 started.
3
Storing .pabotsuitenames file
4
2020-05-30 13:34:00.453909 [PID:6235] [1] [ID:0] EXECUTING Test.Test Acquire Lock1
5
2020-05-30 13:34:00.454809 [PID:6237] [0] [ID:1] EXECUTING Test.Test Acquire Lock2
6
2020-05-30 13:34:03.775381 [PID:6235] [1] [ID:0] PASSED Test.Test Acquire Lock1 in 3.3 seconds
7
2020-05-30 13:34:06.878935 [PID:6237] [0] [ID:1] PASSED Test.Test Acquire Lock2 in 6.4 seconds
8
2 critical tests, 2 passed, 0 failed
9
2 tests total, 2 passed, 0 failed
10
...
11
Stopping PabotLib process
12
Robot Framework remote server at 127.0.0.1:8270 stopped.
13
PabotLib process stopped
14
Total testing: 9.69 seconds
15
Elapsed time: 6.79 seconds


We notice that the elapsed time is almost 7 seconds, where we would have an elapsed time of about 3 seconds without any locking. The log.html shows this even more clearly: The Test Acquire Lock2 test lasts for 6 seconds whereas the Test Acquire Lock1 test lasts for 3 seconds.

6. Conclusion

Pabot is a great tool for running your tests in parallel and to speed up the overall test execution. Just like running code in parallel, it can cause some challenges when you want to run your tests in parallel. Luckily, some features are available in order to overcome these challenges. However, beware that you will need to spend some effort in changing your Test Suites in order to be able to make full use of parallel test execution.

Topics:
acceptance test, python, robot framework, testing, testing tutorial

Published at DZone with permission of Gunter Rotsaert , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}