Automated Performance Tests in Jenkins CI Environments

1. Why is Continuous Integration (CI) Important?

Continuous Integration (CI) is important for developers, as they can integrate code into a shared repository and verify each check-in with an automated build, deploy, and test process. This enables them to quickly find and remove bugs, and measure the impact of each code change to application performance.

Users who want to add performance testing should identify their goals together with business analytics. This is important for determining which types of test should be run at each stage of the CI process.

Here are the types of tests you can run:

• Load Tests are conducted to understand the behavior of the system under a specific expected load.

• Stress Tests are used to understand the upper limits of capacity within the system.

• Soak Tests determine if the system can sustain the continuous expected load.

• Spike Tests determine if the system can sustain a suddenly increasing load generated by a large number of users.

• Isolation Tests determine if a previously detected system issue has been fixed by repeating a test execution that resulted in a system problem.

So for example, for Continuous Delivery, running soak and spike tests in a pre-production environment helps understand how applications perform with continuous expected loads and spikes, with each production deployment.

To simplify inclusion of load tests into CI we suggest to use the Open Source test automation tool called Taurus. Let’s understand how Taurus works.

2. Using Taurus to Easily Run Load Tests

Taurus simplifies the automation of performance testing, is built for developers and DevOps, and relies on JMeter, Selenium, Gatling and Grinder as underlying engines. It also enables parallel testing, and its configuration format is readable and can be parsed by your version control system. It’s tool-friendly, and tests can be expressed using YAML or JSON. If you don’t have Taurus yet, install it.

Let’s look at this simple configuration in YAML. 

This test ramps-up to 5 users for 30 seconds, then sustains this load for 1 more minute. The number of hits per second (throughput) is 3, and they send requests to the url: http://aws-uut:8000/

execution:

- concurrency: 5
  throughput: 3 
  ramp-up: 30s
  hold-for: 1m
  scenario:
    requests:
    - http://aws-uut:8000/

This is a clear and simple test. The same test in JMeter would be very long, with lots of text that is hard to understand, and any changes made to it would be unclear.

You can also split the file into 2 (or more) config files which will make it easier to maintain the test code with the source code, especially if many developers are working on the same test.

(1)

scenarios:
  my_scenario:
    requests:
      - label: Home
        url: /
        method: GET
        think-time: 500ms

(2)

execution:
- concurrency: 5
  throughput: 3 
  ramp-up: 30s
  hold-for: 1m
  scenario:
    default-address: http://54.210.176.248:8000
    requests:
    - include-scenario: my_scenario

The first config file defines the scenario. The second config file defines the execution parameters.

More complex scenarios can include assertions for validating response data, group requests into transactions and pass-fail criteria.

To run the test, type bzt and the name of the YAML file. In this case: bzt simple-travel.yml scenario-home-page.yml  

Give it a few seconds to run and look at the dashboard. These include response times, percentiles, response codes, and local CPU usage.

This is the dashboard for the test after 83% of the test. All response codes are 200s, CPU is  6% and memory is 70%. The two labels that are being hit are the Home Page and the Reservation page, and it’s successful with no failures.

After the test is finished, the results are shown collectively:

Now, let’s understand how Taurus works with BlazeMeter.

3. Viewing Taurus Results On BlazeMeter

By using the -report parameter, we can send the results to BlazeMeter in real-time.

Taurus is working in the background, but we can see all the reports and analyze the results on BlazeMeter in a more sophisticated and attractive fashion.

Finally, let’s see how it all comes together with Jenkins so you can include it in your Continuous Integration process.

4. Using Jenkins With Taurus and BlazeMeter for Load Testing

To use Jenkins for load testing, first install Taurus onto the Jenkins machine.

Now, open Jenkins.

1. Click ‘Configure’ on the left.

2. Choose the Jenkins Build Triggers tab on top.

3. Scroll down to the Build section and enter the command for the test.

In this case: bzt simple-travel.yml -report

If you have a complete CI deployment process, make sure the test is kicked off with it. That’s it.

To see the test details, check out the Console Output.

Results can also be shown on BlazeMeter.

We can add details easily by changing the test config file in one place on Taurus.

The test will now run on Taurus and on Jenkins. Congratulations! Now you can see the results and analyze load testing as a part of your full development process.