DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • An Explanation of Jenkins Architecture
  • Travis CI vs Jenkins: Which CI/CD Tool Is Right For You?
  • Recipe To Implement the Jenkins Pipeline For MuleSoft Application [Videos]
  • Integrating Jenkins With Playwright TypeScript: A Complete Guide

Trending

  • Advancing Your Software Engineering Career in 2025
  • Navigating and Modernizing Legacy Codebases: A Developer's Guide to AI-Assisted Code Understanding
  • Navigating Change Management: A Guide for Engineers
  • Building an AI/ML Data Lake With Apache Iceberg
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Deployment
  4. Running Your Load Tests With PassFail Criteria - A Guide

Running Your Load Tests With PassFail Criteria - A Guide

Learn about two ways to automatically check if a specific test breaches your thresholds: using either Taurus in PassFail mode, or using Jenkins with Taurus or JMeter.

By 
Guy Salton user avatar
Guy Salton
·
Jul. 12, 17 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
6.9K Views

Join the DZone community and get the full member experience.

Join For Free

PassFail criteria for tests metrics and KPIs are an inherent part of performance tests, since they ensure your product is performing according to the requirements. For example, “The average response time for the users needs to be no more than 600 milliseconds,” or “I expect the average latency of my servers to be no more than 200 milliseconds.”

How can you check if a specific test breached your thresholds? In this blog post, I will elaborate about two ways: by using Taurus in PassFail mode, and by using Jenkins with Taurus or JMeter.

You can also test manually by checking if the metrics you wanted to measure reached values that are above your thresholds. But this is a thorough but very slow process, and it can be a real pain! You can also use the Apache JMeter™ AutoStop Listener (see this blog post).

Running Load Tests With PassFail Criteria in Taurus

Taurus is an open-source test automation framework. It uses simple YAML syntax to run open-source load and performance tests, for tools like JMeter, Selenium, Gatling, and Locust.

Taurus provides a module called PassFail, which helps you automate the threshold testing process. In Taurus, you can set pass/fail criteria in a YML file, and add this YML file to your Taurus execution. The Taurus pass/fail module supports many different load metrics, such as Response Time, Latency, Error Rate, and more. You can also assign the criteria to all the labels in your test, or only to specific labels. Learn about all the PassFail options in Taurus from this link.

Let's look at an example. Here is a Taurus execution configuration:

 --- execution: - concurrency: 10 hold-for: 2m ramp-up: 40s scenario: Thread Group scenarios: Thread Group: requests: - label: blazedemo method: GET url: http://blazedemo.com/ - body: fromPort: Paris toPort: Buenos Aires label: reserve method: POST url: http://blazedemo.com/reserve.php 

Save this file as execution.yml. Now, let's add pass/fail criteria.

 --- services: - module: passfail criteria: - avg-rt>10ms for 7s, stop as failed - hits of reserve >10 for 13s, continue as failed 

The above configuration sets two pass/fail criteria:

  • If the average response time (for all the labels in the test) goes over 10 milliseconds for 7 seconds in a row, stop the test as failed.
  • If the number of hits (number of responses) for a specific label called ‘reserve’ is more than 10 for 13 seconds in a row, continue with the test, but mark it as failed.

Save this pass/fail configuration in a different YML file and called it passfail_config.yml.

Now that we have finished with the configuration, let’s run the test.

Taurus tests are run directly from the terminal, with the ‘bzt’ command. When we have multiple YML files that we want to use in our test we simply add them one after the other:

 Bzt execution.yml passfail_config.yml 

pass fail load tests

After a few seconds, the Taurus console report will be displayed automatically.

As the average response time for the test was more than 10 milliseconds for 1 second, we get an alert. The alert appears even though we haven’t reached the threshold yet. The purpose is to let us know we might have a problem soon.

pass fail continuous integration After 7 seconds from the beginning of the test, the average response time was more than 10 milliseconds for 7 seconds in a row. This means that our first criterion was met, so the test will be stopped and marked as failed as expected.

Since the first condition was met, we didn't have a chance to check out the second condition. The two are unrelated and the test checks each one separately.

pass fail taurus

pass fail peformance testing

By using the Taurus passfail module, you don’t have to manually examine tests results anymore. If one of your tests met the criteria you set, it will fail the test automatically and you will get a notification for this in the report and logs.

Taurus is a very good tool for test automation. Now let’s see how we can schedule Taurus tests by using open-source CI tool Jenkins.

Running Load Tests With PassFail Criteria in Jenkins

We recommend running load tests as part of your continuous integration process. This ensures you are constantly ensuring your code changes or product updates don’t crash your website or app. One of the most popular open-source tools for continuous integration is Jenkins.

Both JMeter and Taurus can be integrated into Jenkins, and Jenkins can then schedule automated tests. We will explain how to run Taurus with its PassFail configuration with Jenkins. You will see it is quite easy. To learn how to run Jenkins with JMeter, click here.

As Taurus is a command line tool, you can simply use the ‘Execute Shell’ build step in your Jenkins project, to run the same execution command we ran locally:

passfail jenkins

However, you should add  --option=modules.console.disable=true, as we don’t need the Taurus console report when running from Jenkins.

After saving our Jenkins project, we can go ahead and build it by clicking on the ‘Build Now’ button (or use a cron job for a scheduled execution).

pass fail assertions schedule tests

Once the build has started, open the Console Output:

pass fail kpis jenkins

In the Console Output, you can see the results of your Taurus load test, and in this case, the failure of the test due to the pass/fail criteria being met.

pass fail thresholds jenkins

As the test failed, it will fail the Jenkins build.

pass fail jenkins

To schedule jobs in Jenkins, you can use the ‘Build Triggers’ section in your Jenkins project. There are a number of ways to schedule jobs: periodically by using CRON (see more details here), by clicking "Build when a change is pushed to GitHub," and more. This way, you can easily integrate your performance tests into your CI process, and see which builds failed due to a pass/fail criteria being met.

Congratulations! You now know how to run tests with thresholds by using Taurus and Jenkins.

Testing Continuous Integration/Deployment Open source Jenkins (software)

Published at DZone with permission of Guy Salton, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • An Explanation of Jenkins Architecture
  • Travis CI vs Jenkins: Which CI/CD Tool Is Right For You?
  • Recipe To Implement the Jenkins Pipeline For MuleSoft Application [Videos]
  • Integrating Jenkins With Playwright TypeScript: A Complete Guide

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!