DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Solid Testing Strategies for Salesforce Releases
  • Apex Testing: Tips for Writing Robust Salesforce Test Methods
  • Load Testing Essentials for High-Traffic Applications
  • Motivations for Creating Filter and Merge Plugins for Apache JMeter With Use Cases

Trending

  • Intro to RAG: Foundations of Retrieval Augmented Generation, Part 1
  • It’s Not About Control — It’s About Collaboration Between Architecture and Security
  • How the Go Runtime Preempts Goroutines for Efficient Concurrency
  • Integration Isn’t a Task — It’s an Architectural Discipline
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Modes and Modality in Performance Testing

Modes and Modality in Performance Testing

This article explores why it's crucial to check for modality of performance test results, how to detect multimodal distributions, and how to handle them.

By 
Robert Bourgault user avatar
Robert Bourgault
·
Sep. 11, 24 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
3.3K Views

Join the DZone community and get the full member experience.

Join For Free

When reporting summary statistics for performance test results, there's a common assumption that the data follows a central tendency. But is this always true? What if the data does not exhibit a single central tendency? In such cases, the traditional metrics like averages might be misleading.

This article explores why it's crucial to check for modality of performance test results, how to detect multimodal distributions, and how to handle them. For users of JMeter, a new plugin is introduced to help automate this process.

Unimodal vs. Multimodal Distributions

In performance testing, data is often assumed to be unimodal; i.e., where the results cluster around a single peak. In such cases, summary statistics like the mean (average) are meaningful because they reflect the central tendency. Here are examples of unimodal distributions under different skewness:

Examples of unimodal distributions under different skewness

However, in many scenarios, the response times can show a multimodal distribution, which means there are two or more distinct peaks in the data. Most commonly, this is bimodal:

Example of response times showing a multimodal distribution, which means there are two or more distinct peaks in the data

In a bimodal distribution, the data reflects two separate patterns, each with its own average. If you rely on the overall mean, you might miss this and end up with misleading conclusions. The summary statistics become less useful, and you must separate the data into distinct modes to understand what's happening in the system.

Why Multimodality May Happen in Performance Testing

There are several reasons for multimodal distributions in software performance testing, to name a few:

  • Varying test data: Some inputs might take longer to process than others.
  • Server-side events: Background tasks on the server under test could intermittently slow down performance.
  • Load balancing issues: A load balancer might route traffic to the slower legs of a distributed system.
  • Test script bugs: Invalid parameters could cause "fast failures" that the script doesn't catch, skewing the results.

Detecting Multimodality Using Visual Inspections

Common methods for detecting multimodal distributions are to visualize the data with two types of graphs:

  1. Histograms
  2. Percentiles graphs

Histograms

You can plot a histogram using tools like Excel, or if you're using JMeter, the Response Times Distribution plugin can generate histograms for you. Visually inspect the histogram to see if modes look genuine or if they might be artifacts of small bin size.

Percentiles

Another way to detect multimodality is through the percentiles graph. If the graph has noticeable "steps" in the middle percentiles (beyond the extremes), this could indicate multimodal behavior. As an example, the following percentile graph shows the curves for both bimodal (red) and unimodal (green) samplers:

Percentile graph showing the curves for both bimodal (red) and unimodal (green) samplers

Limitations of Manual Visualizations

Manual visualizations can be time-consuming to generate and interpret histograms, and choosing the correct bin size (the width of each bar in the histogram) can be tricky. Too large a bin size might mask subtle differences, while too small a bin size can overemphasize minor variations. This becomes especially difficult when you have multiple samplers to analyze.

Automating Multimodal Detection

To simplify this process, a new JMeter plugin automatically detects multimodal distributions in performance test data. It uses an indirect approach based on the mvalue-based modal test as suggested by Brendan Gregg.

The mvalue measures the variation in a histogram, normalized by the height of its peaks. By comparing the mvalue to a predefined threshold, the tool can decide whether the data is multimodal. Adjusting the threshold helps balance false positives and false negatives.

Brendan Gregg recommends an mvalue of 2.4 as a good starting point for investigating multimodality, while Andrey Akinshin suggests 2.8 to reduce false positives. The best threshold for your system may vary, so experimentation is key.

The Modality Plugin in a Nutshell

The plugin calculates the mvalue using two different bin sizes:

  1. Scott's normal reference rule, which is optimal for normally distributed data
  2. Freedman-Diaconis rule, which works better with skewed data

The plugin chooses the bin size that produces the highest mvalue and presents the results in a grid:

The plugin chooses the bin size that produces the highest mvalue and presents the results in a grid

Once multimodality is flagged, you can quickly select the suspicious samplers in the Rows tab and inspect their histograms in the Chart tab to confirm whether the data truly contains multiple modes. For instance, if the bin size is too narrow, the tool may inadvertently report multimodality due to "noise" or minor fluctuations in the data (false positives):

Response times distribution graph

Addressing Multimodal Results

Once you've confirmed multimodality in your test results, there are two main approaches to addres it:

  • If the issue is environmental (e.g., network latency): Engage the responsible teams to help resolve the underlying issue.
  • If the issue is within your test script: Refactor your test to account for the different modes. For example, if you detect bimodal behavior in a Payment API, you might separate the responses into two different samplers "Payment-Fast" and "Payment-Slow." This allows the statistical results to reflect the system's behavior more accurately.

By separating these modes, you get a clearer understanding of each behavior and can better communicate issues to stakeholders.

Conclusion

Understanding and handling multimodal distributions is critical for accurate performance analysis. When multiple modes exist, the average (mean) may not accurately represent the system's behavior, leading to skewed interpretations of your results.

The new JMeter Modality plugin can automate the detection of multimodality, making it easier to catch issues that might otherwise go unnoticed. By combining its results with a visual inspection, you can mitigate the risk of false positives and arrive at a more reliable assessment of multimodality. Just remember to trim outliers from your data before using the plugin to ensure it can effectively detect multimodal patterns and avoid false negatives.

Percentile Test data Testing Performance Apache JMeter

Opinions expressed by DZone contributors are their own.

Related

  • Solid Testing Strategies for Salesforce Releases
  • Apex Testing: Tips for Writing Robust Salesforce Test Methods
  • Load Testing Essentials for High-Traffic Applications
  • Motivations for Creating Filter and Merge Plugins for Apache JMeter With Use Cases

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!