Over a million developers have joined DZone.

Performance Testing In a Continuous Delivery Pipeline

DZone's Guide to

Performance Testing In a Continuous Delivery Pipeline

Functional and unit testing are essential for a good CD pipeline, but don't forget about performance tests. Here is how to create a good load test for DevOps.

· Performance Zone ·
Free Resource

Maintain Application Performance with real-time monitoring and instrumentation for any application. Learn More!

Everyone talks about unit and functional tests when adopting DevOps practices, but how do you know if one of your changes has introduced a major performance degradation? Performance testing is the missing link to having a truly Continuously Deliverable pipeline.

In this blog post, you will see how performance tests can be integrated into your Continuous Delivery pipeline. We will first go over why performance tests should be included in your Continuous Delivery pipeline. Next, we will look at how to design your tests to get the maximum benefit from them and how to run them. Once the tests built and running, we will show how to put it all together.

Why Do I Need to Do Performance Tests?

If you are following the DevOps approach of software development, you probably have unit tests and functional tests to ensure all different parts of your application work as they should. But do you know how your application will react when 10 concurrent users are trying to access the same page? 100? 1000? While your application may seem to work well with just a couple users, it is important to know how your application performs under load.

Enter load tests! Load testing an application generally means that you’re measuring performance with a specific predetermined amount of load. By incorporating load tests early on in the development cycle you can make sure you always have a performant application as well as the added benefit of being able to understand the performance of your application early on. Even introducing performance testing late in the development cycle will help protect you from future changes that can impact performance. In the event that a change has caused your application to significantly slow down, or even crash, your pipeline will automatically rollback the changes. This helps the developers get feedback earlier and iterate quicker thus closing the feedback loop. 

Creating the Test Plan

When designing the test plan, the goal is not to test every single page of the application. This would cause very long-running, as well as expensive, tests. Our goal is to identify the parts of the application that are most likely to cause problems and test those on every build. As the application grows more tests are added.

For this post, we will take a simple Ruby on Rails application that has a basic sign in page. We will be using the gem ‘ruby-jmeter’ to construct our load tests. Ruby JMeter comes with a very easy to use Domain Specific Language which makes writing tests much easier. We will piggyback on Flood.io’s API to carry out the load tests. You will need to create a free account and get an API key which they provide you.

You can find the repository with all the code examples and templates for setting up the pipeline at https://github.com/stelligent/load-testing-example.

We start by creating a test file, ‘load_test.rb’, and setting a few options.  

require 'ruby-jmeter'
test do
  defaults domain: 'load-test.elasticoperations.com' # Our domain we are pointing to
  cache clear_each_iteration: true # Keeps track of the simulated browser's cache and then clears the cache at the beginning of each iteration
  # We define a load of 50 users that ramps up over 60 secords and a test that lasts for 120 seconds 
  threads 50, {ramp_time: 60, duration: 120, continue_forever: true} do
  # Time of 1-3 seconds to space out user requests and simulate real life situation 
  random_timer 1000, 3000
  # Extract csrf-token from the response body 
  extract name: 'authenticity_token',
              regex: 'meta content="(.+?)" name="csrf-token"'

The test is then broken up into 2 separate transactions that simulate how a user would use the site.

  • The first ‘transaction’ simply issues a GET request to the home page and using the ‘assert’ method, checks that the home page contains certain text.
  • The second transaction uses the ‘submit’ method to send a form to the login page via a POST request using the ‘fill_in’ method to enter the requisite field

transaction '01_my_site_visit_home_page' do
        visit '/' do
          assert contains: 'Welcome to my site'
  transaction '02_my_site_login' do
        submit '/login', {
          fill_in: {
            'utf8'                            => '%E2%9C%93',
            'authenticity_token'              => '${authenticity_token}',
            'user[name]'                      => 'Timmy',
            'user[password]'                  => 'Tables',
            'commit'                          => 'Sign In'

We then run ‘ruby load_test.rb’ to convert the file to a jmx file that can be processed by flood.io

Integrating Into a Continuous Delivery Pipeline

When integrating performance tests into a pipeline, the goal is to ensure that any issues are caught before they get released into a production environment. Since our tests are designed to determine if our application is able to handle requests under load we can’t test our live production environment. Our solution then is to create a staging environment that is a copy of our production environment. This way we can test under the exact same conditions without adversely affecting production.

To accomplish this, we will add a new phase to our ‘CodePipeline’ called ‘Staging’. Within this phase we add a CodeBuild step and provision it to match production with the added step of running our load test.

We then add a shell script that runs on each build that programmatically runs the load test and either passes or fails the build based on predefined criteria.

We launch the load test and take note of the flood uuid:

set -e

echo "[$(date +%FT%T)+00:00] Launching flood"
flood_uuid=$(curl --silent -u $FLOOD_API_TOKEN: -X POST https://api.flood.io/floods \
  -F "flood[privacy]=public" \
  -F "flood_files[]=@$WORKSPACE/tests/ruby-jmeter.jmx" )

We then gather the test status and we wait for it to finish :

while [ $(curl --silent --user $FLOOD_API_TOKEN: https://api.flood.io/floods/$flood_uuid -r '.status == "finished"') = "false" ]; do
  sleep 3

After the test completes we measure key metrics and can fail the build if the error rate or response time is above a certain threshold :

flood_report=$(curl --silent --user $FLOOD_API_TOKEN: https://api.flood.io/floods/$flood_uuid/report -r ".summary")

error_rate=$(curl --silent --user $FLOOD_API_TOKEN: https://api.flood.io/floods/$flood_uuid -r .error_rate)
response_time=$(curl --silent --user $FLOOD_API_TOKEN: https://api.flood.io/floods/$flood_uuid -r .response_time)

echo "[$(date +%FT%T)+00:00] Detailed results at https://flood.io/$flood_uuid"
echo "---"
echo "$flood_report"

if [ "$error_rate" -gt "0" ]; then
  echo "Flood test failed with error rate $error_rate%"
  exit 1

if [ "$response_time" -gt "3000" ]; then
  echo "Flood test failed with response time $response_time > 3000ms"
  exit 2

With this pipeline, we were able to create an exact copy of our production environment, call an external API to simulate load to our application, and either promote the release candidate to production or fail the build based on whether or not it passed the load test.

Gotchas and Other Things to Consider

One of the main reasons that performance testing is so often neglected is due to the fact that performance tests can take a long time and be resource intensive. Instead of testing every single part of an application, we test the most important parts. What we lose in completeness, we make up for in practicality and speed. One of the most common ways to speed up your performance tests is to run them in parallel with your unit and integration tests. This way you will not be forced to wait for the other tests to finish before you start your performance tests.


As we have shown, performance testing doesn’t need to be hard or expensive. Integrating these tests early on and on every build, we are able to catch problems earlier and improve the reliability and stability of our releases.

Collect, analyze, and visualize performance data from mobile to mainframe with AutoPilot APM. Get a Demo!

performance ,devops performance management ,performance testing ,ruby ,tutorial ,load testing

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}