DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

How does AI transform chaos engineering from an experiment into a critical capability? Learn how to effectively operationalize the chaos.

Data quality isn't just a technical issue: It impacts an organization's compliance, operational efficiency, and customer satisfaction.

Are you a front-end or full-stack developer frustrated by front-end distractions? Learn to move forward with tooling and clear boundaries.

Developer Experience: Demand to support engineering teams has risen, and there is a shift from traditional DevOps to workflow improvements.

Related

  • Integrating Lighthouse Test Automation Into Your CI/CD Pipeline
  • Pipeline as a Service: How To Test Pipelines in GitLab
  • Best Practices for Building the Data Pipelines
  • CI/CD Metrics You Should Be Monitoring

Trending

  • RAG vs. CAG: A Deep Dive into Context-Aware AI Generation Techniques
  • Modernizing Financial Systems: The Critical Role of Cloud-Based Microservices Optimization
  • Understanding the Fundamentals of Cryptography
  • Multi-Cluster Networking With Kubernetes and Docker: Connecting Your Containerized Environment
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. Faster Pipelines With the Parallel Test Executor Plugin

Faster Pipelines With the Parallel Test Executor Plugin

One problem we always encounter is how to run all the tests needed to ensure high-quality changes while still keeping pipeline times reasonable.

By 
Liam Newman user avatar
Liam Newman
·
Oct. 10, 16 · Opinion
Likes (3)
Comment
Save
Tweet
Share
7.2K Views

Join the DZone community and get the full member experience.

Join For Free

This is the first is a series of posts showing ways to use Jenkins Pipeline. In this post, I'll show you how to speed up your pipeline by using the Parallel Test Executor Plugin.

So Much to Do, So Little Time

In my career, I've helped many teams move to continuous integration and delivery. One problem we always encounter is how to run all the tests needed to ensure high-quality changes while still keeping pipeline times reasonable and changes flowing smoothly. More tests mean greater confidence, but also longer wait times. Build systems may or may not support running tests in parallel, but they still only use one machine even while other lab machines sit idle. In these cases, parallelizing test execution across multiple machines is a great way to speed up pipelines. The Parallel Test Executor plugin lets us leverage Jenkins do just that with no disruption to the rest of the build system.

Serial Test Execution

For this post, I’ll be running a pipeline based on the Jenkins Git Plugin. I've modified the Jenkinsfile from that project to allow us to compare execution times to our later changes, and I've truncated the "mvn" utility method since it remains unchanged. You can find the original file here.

node {
  stage 'Checkout'
  checkout scm

  stage 'Build'

  /* Call the Maven build without tests. */
  mvn "clean install -DskipTests"

  stage 'Test'
  runTests()

  /* Save Results. */
  stage 'Results'

  /* Archive the build artifacts */
  archive includes: 'target/*.hpi,target/*.jpi'
}

void runTests(def args) {
  /* Call the Maven build with tests. */
  mvn "install -Dmaven.test.failure.ignore=true"

  /* Archive the test results */
  step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])
}

/* Run Maven */
void mvn(def args) { /* ... */ }

NOTE: This pipeline expects to be run from a Jenkinsfile in SCM. To copy and paste it directly into a Jenkins Pipeline job, replace the checkout scm step with git 'https://github.com/jenkinsci/git-plugin.git'.

This is a Maven project, so the Jenkinsfile is pretty simple. I’ve split the Maven build into separate “Build” and “Test” stages. Maven doesn’t support this split very well, it wants to run all the steps of the lifecycle in order every time. So, I have to call Maven twice: first using the “skipTests” property to do only build steps in the first call, and then a second time with out that property to run tests.

On my quad-core machine, executing this pipeline takes about 13 minutes and 30 seconds. Of that time, it takes 13 minutes to run about 2.7 thousand tests in serial.

Serial Test Pipeline

Parallel Test Execution

This looks like an ideal project for parallel test execution: a short build followed by a large number of serially executed tests that consume the most of the pipeline time. There are a number of things I could try to speed this up. For example, I could modify test harness to look for ways to parallelize the test execution on this single machine. Or I could try to speed up the tests themselves. Both of those can be time-consuming and both risk destabilizing the tests. I'd need to know more about the project to do it well.

I'll avoid that risk by using Jenkins and the Parallel Test Executor Plugin to parallelize the tests across multiple nodes instead. This will isolate the tests from each other, while still giving us speed gains from parallel execution.

The plugin reads the list of tests from the results archived in the previous execution of this job and splits that list into a specified number of sublists. I can then use those sublists to execute the tests in parallel, passing a different sublist to each node.

Let’s look at how this changes the pipeline:

node { /* ...unchanged... */ }

void runTests(def args) {
  /* Request the test groupings.  Based on previous test results. */
  /* see https://wiki.jenkins-ci.org/display/JENKINS/Parallel+Test+Executor+Plugin and demo on github
  /* Using arbitrary parallelism of 4 and "generateInclusions" feature added in v1.8. */
  def splits = splitTests parallelism: [$class: 'CountDrivenParallelism', size: 4], generateInclusions: true

  /* Create dictionary to hold set of parallel test executions. */
  def testGroups = [:]

  for (int i = 0; i  }. */
    /*     includes = whether list specifies tests to include (true) or tests to exclude (false). */
    /*     list = list of tests for inclusion or exclusion. */
    /* The list of inclusions is constructed based on results gathered from */
    /* the previous successfully completed job. One additional record will exclude */
    /* all known tests to run any tests not seen during the previous run.  */
    testGroups["split-${i}"] = {  // example, "split3"
      node {
        checkout scm

        /* Clean each test node to start. */
        mvn 'clean'

        def mavenInstall = 'install -DMaven.test.failure.ignore=true'

        /* Write includesFile or excludesFile for tests.  Split record provided by splitTests. */
        /* Tell Maven to read the appropriate file. */
        if (split.includes) {
          writeFile file: "target/parallel-test-includes-${i}.txt", text: split.list.join("\n")
          mavenInstall += " -Dsurefire.includesFile=target/parallel-test-includes-${i}.txt"
        } else {
          writeFile file: "target/parallel-test-excludes-${i}.txt", text: split.list.join("\n")
          mavenInstall += " -Dsurefire.excludesFile=target/parallel-test-excludes-${i}.txt"
        }

        /* Call the Maven build with tests. */
        mvn mavenInstall

        /* Archive the test results */
        step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])
      }
    }
  }
  parallel testGroups
}

/* Run Maven */
void mvn(def args) { /* ... */ }

That’s it! The change is significant but it is all encapsulated in this one method in the Jenkinsfile.

Great (ish) Success!

Here are the results for the new pipeline with parallel test execution:

Pipeline Duration Comparison

The tests ran almost twice as fast, without changes outside pipeline. Great!

However, I used 4 test executors, so why am I not seeing a 4x? improvement. A quick review of the logs shows the problem: A small number of tests are taking up to 5 minutes each to complete! This is actually good news. It means that I should be able to see further improvement in pipeline throughput just by refactoring those few long running tests into smaller parts.

Conclusion

While I would like to have seen closer to a 4x improvement to match to the number of executors, 2x is still perfectly respectable. If I were working on a group of projects with similar pipelines, I'd be completely comfortable reusing these same changes on my other project and I'd expect to similar improvement without any disruption to other tools or processes.

Links

  • Parallel Test Executor Plugin
Testing Pipeline (software) Executor (software)

Published at DZone with permission of Liam Newman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Integrating Lighthouse Test Automation Into Your CI/CD Pipeline
  • Pipeline as a Service: How To Test Pipelines in GitLab
  • Best Practices for Building the Data Pipelines
  • CI/CD Metrics You Should Be Monitoring

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: