Jenkins vs. Other Open Source Continuous Integration Servers

Continuous integration is now a vital part of the modern software development life cycle. While this blog has covered its importance and benefits before, a short review is beneficial for our look at open source CI servers.

When developers create new features or fix defects, it’s critical that new code changes won’t harm or have a negative impact on previously existing software elements. Key objectives to keep in mind include:

1. The whole process of software development starting from a single commit, compilation, running tests, etc. and ending up with packaging and delivery needs to be automated to make the process as efficient as possible
2. Performance testing should be an inherent part of the continuous delivery process starting from early development stages
3. All associated actions need to be automated
4. If everything goes well, the resulting software package should be packaged for distribution or even deployed into production
5. If there is an error, the relevant team members should be notified

As software development and testing techniques are becoming more and more complex, there is a need for a special software to orchestrate the aforementioned processes, as well as to ensure that the steps required for software delivery are being executed in the correct order, nothing is missing, and failures are correctly handled.

That is what a Continuous Integration Server is. In this post, we will cover the most prominent free and open source solutions and review them from the perspective of integrating performance testing into the continuous integration and delivery process.

Jenkins

Jenkins is the leading open source continuous integration tool. For the last few years, it has held the top position as the DevOps toolchain of choice due to being free, open source, and modular (although its baseline functionality is very limited, there are more than 1,000 plugins which extend Jenkins capabilities and provide extra integrations).

Jenkins is a Java-based tool so you will need Java Runtime Environment to run it. All the extensions and plugins are also to be developed in Java. Given the underlying Java Runtime, Jenkins can be installed on any operating system where Java runs.

Out of the box (or with default plugins, if you want) Jenkins supports the following:

• Version Control Systems:
  • Subversion
  • Git
• Builders
  • Command-line
  • Maven
  • Ant
  • Gradle
• Notifications
  • Git publisher
  • Email Notification
  • Set status for GitHub commit

You need to remember that Jenkins functionality is not limited to the above features, it can be extended via Jenkins Plugins even for very unusual scenarios. For instance, you can even kick off a BlazeMeter test and display results in Jenkins dashboard using the BlazeMeter Jenkins Plugin.

Installation of Jenkins and configuration for running JMeter tests is covered in Continuous Integration 101: How to Run JMeter With Jenkins, which contains comprehensive information on getting Jenkins and JMeter up and running.

Jenkins 2.0, the first major release of the software in years, was released in April 2016.

Here is an example of the Jenkins dashboard. In the bottom left corner, you can see that a build is being executed at the moment as well as the history of recent builds.

Buildbot

Buildbot is another continuous integration tool which is trusted by the Mozilla Foundation for bringing excellence into their products, and is now used at Chromium, WebKit, and many other projects.

Buildbot started as a lightweight alternative to Mozilla Tinderbox and currently it’s a fully functional continuous development and build server supporting a lot of integrations, distributed builds, custom (and often unique) build steps, notifications via multiple channels, and much more.

Buildbot is a Python-based tool, and as such, it runs everywhere Python runs. You can deploy it onto any POSIX-compliant operating system. Moreover, Buildbot config files are basically Python scripts, meaning that the configuration is version-control friendly and you have all the power of the Python language while defining your build plan. It means extreme flexibility and capabilities.

As for July 2016, Buildbot supports out of the box:

• Version Control Systems:
  • Mercurial
  • Subversion
  • Git
  • CVS
  • Perforce
  • Bzr
  • Repo
  • Gerrit
  • Monotone
  • Darcs
• Builders:
  • Command-line
  • Configure (autoconf-style projects)
  • Compile (C projects)
  • VC++ (Microsoft compilers)
  • Robocopy
  • Test (analogue of make test)
  • TreeSize - check repo size
  • Perl Module Test
  • Mysql-test-run
• Notifications
  • Email Notification
  • IRC Bot
  • PBListener
  • HTTP Status Push
  • Gerrit Status Push
  • Github Status Push
  • Stash Status Push

Buildbot installation and configuration is a separate large topic as the instructions may differ depending on each operating system and architecture, so it is better to refer to Buildbot documentation. However, if you’re looking for quick and dirty way to get it running and see if it fits your needs, here are minimal instructions:

1. Install Python. Most Linux and MacOSX distributions come with Python, on Windows, you’ll have to install it manually. Make sure you have at least the 2.7.X version.
2. Make sure you have the following binaries in your PATH (on Windows they should be located under the “Scripts” folder of your Python installation):
   1. Python itself
   2. pip - a Python package management system
   3. Easy install - a script which simplifies installation of Python packages
3. Run the following commands:
   1. pip install buildbot- installs the Buildbot master (or server)
   2. easy_install buildbot-slave - installs the so called “Buildbot slave,” the agent which will execute all the actual work
   3. buildbot create-master /path/to/buildbot/working/folder - sets up some of the initial configuration for the buildbot server
   4. buildslave create-slave slave localhost:9989 example-slave pass - creates the Buildbot agent with default parameters
4. Copy master.cfg.sample to master.cfg. The file is located under the previously specified /path/to/buildbot/working/folder. This is the main Buildbot config file in the Python language. Change the main settings according to your project needs.
5. Start the Buildbot server: buildbot start /path/to/buildbot/working/folder .
6. Start the Buildbot slave: buildslave start slave .
7. Open the following url in your browser: http://localhost:8010 (unless you changed it to something different in the master.cfg file).
8. If you plan to run a JMeter test, add a Builder such as:

 factory.addStep(steps.ShellCommand(command=["c:/jmeter/bin/jmeter.bat -n -t c:/buildbot/Test.jmx -l C:/buildbot/test.jtl"])) to your master.cfg file.

Here is the Buildbot dashboard with one build currently running along with the history of several previous test executions:

CruiseControl

CruiseControl is the most mature continuous integration server out of today’s scope and probably one of the oldest continuous integration solutions in existence. The first release was 15+ years ago. CruiseControl was originally developed as a framework to orchestrate build, testing, and other software development lifecycle for one particular internal project at ThoughtWorks. Later, CruiseControl became a standalone continuous integration server.

CruiseControl is based on Java, so it’s cross-platform by design. There are also implementations in .NET and Ruby.

The latest CruiseControl release provides the following integrations out-of-the-box:

• Version Control Systems:
  • Accurev
  • Alienbrain
  • Clearcase
  • CVS
  • Darks
  • Git
  • Mercurial
  • PTC Integrity
  • Perforce
  • Plastic SCM
  • Starteam
  • Surround SCM
  • Subversion
  • TFS
  • VSS
• Builders
  • Command-line
  • Ant
  • Nant
  • Maven
  • Rake
  • Phing
• Post-build
  • Email
  • CC-tray (desktop application)

CruiseControl configuration is XML-based. Check out the configuration reference for all possible settings. There is also a Java-based GUI frontend, CC-config, and perhaps it would be easier to use it instead of manually editing the XML config file.

Here is an example of the minimal configuration for JMeter execution in the command-line non-GUI mode is:

<cruisecontrol>
   <dashboard />
   <project name="Performance Test">
       <schedule>
           <exec Command="c:\jmeter\bin\jmeter.bat" Args="-n -t c:\jmeter\extras\Test.jmx -l c:\jmeter\extras\Test.jtl" />
       </schedule>
       <log />
   </project>
</cruisecontrol>

For your reference, below is the CruiseControl project dashboard containing a history of performance tests execution and statistics on one of the builds:

If you want to integrate results into the “Test” dashboard tab, you need to merge your result files with CruiseControl log file, through the following:

<cruisecontrol>
   <dashboard />
   <project name="Performance Test">
       <schedule>
           <!--execute performance test -->
       </schedule>
       <log>
           <!-- Add performance test results to CruiseControl -->
           <merge dir="c:\PerformanceTestResults\" Pattern="*.xml" />
       </log>
   </project>
</cruisecontrol>

Integrating Performance Testing Tools

Running a performance test as a part of the build is fine, however, it doesn’t resolve the main goal—automated results analysis. The whole idea of the continuous integration process is making software build and associated processes absolutely unattended. As a result, it is important to have some form of mechanism to fail the build when the performance test produces an error or the average response time exceeds a certain threshold. In some cases, it can be resolved via plugins like the Performance Plugin for Jenkins. But in more advanced scenarios, there might be not enough time or budget to develop the plugin or to change the load testing script.

That’s why I would recommend running your performance tests via Taurus, an automation-friendly framework for continuous testing which supports a lot of underlying load generation tools including but not limited to JMeter, Gatling, Grinder, Tsung, Locust, and more.

The key Taurus features from the continuous integration perspective are:

• You don’t need to change your existing tests scripts, Taurus can “consume” everything
• You can easily define fail criteria using flexible Pass/Fail Service service. The most important bit is that Taurus will return non-zero exit code so that any continuous integration server will treat the Taurus execution step as failed
• Taurus can produce results in JUnit format via the JUnit XML Reporter. The absolute majority of continuous integration servers can “understand” the JUnit output, and, as a result, can display and/or plot results on the build dashboard

Hopefully, this guide sheds some light on the available open source CI servers, as well as the ways of adding performance tests to your continuous integration workflow.

If you have any questions, feel free to use the comments section below, any form of feedback will be appreciated.   

Want to Know More?

Here are a few further steps that will help you to further examine the topic of performance testing and continuous integration:

1. Register for our upcoming webinar Automated Performance Tests in Jenkins CI Environments
2. Read more about Jenkins 2.0 and a comprehensive list of Jenkins Resources
3. Read more about how to run JMeter tests in Jenkins, Bamboo, and TeamCity