How to Compose the Most Effective Apache JMeter Test Plan
How to Compose the Most Effective Apache JMeter Test Plan
Apache JMeter as a flexible and smart tool for performance testing of both static and dynamic web resources.
Join the DZone community and get the full member experience.Join For Free
Sensu is an open source monitoring event pipeline. Try it today.
Apache JMeter as a flexible and smart tool for performance testing of both static and dynamic web resources. To make the most use of it, you should keep the goal of testing at the head, compose a wise and realistic test plan with both native and third-party JMeter tools and adjust it in the process. This would help you discover system bottlenecks and potential stress points to further eliminate them.
We actively use JMeter for performance testing of our web applications, so we wanted to share a few tips on how to make a JMeter test plan even more effective and realistic.
How to Choose the Number of Threads for a Test Plan
The number of JMeter virtual users (threads) executing the test would depend on multiple factors, such as your goal while testing, parameters of your local PC and the network, and script complexity. Before you decide on the optimal number of virtual users for your JMeter test plan, consider the following tips.
1. Examine Server Logs
Server logs contain the history of web-requests with users’ IP-addresses, date, and time. So, a graph showing the server load over a week would give you an idea about the average number of real users and their requests (hits) for low, normal, and high server loads. Target the peak number of requests as opposed to average one to be replicated in your JMeter test plan.
Though JMeter does not offer a native listener to identify the number of active virtual users for a thread group in every given point of time, in Logicify, we use a third-party plug-in called Active Threads Over Time. It presents this number on simple yet laconic graphs.
Approaches to Server Loading
While load testing, you could build up various loads on the server by adjusting JMeter Thread group properties.
- Linear load — one of the typical server loads while performance testing.
The linear load is good for the majority of cases, yet test results would not be indicative of the server crashes during the load rise. In this case, it makes sense to examine the point where the system failed and start from there to determine its maximum load.
- Step load — allows to let virtual users out in chunks, for example, 25, then 25 more, and 25 more. This approach is more reliable and indicative of system performance under different loads, with a varied number of virtual users executing the test in every given time interval.
Native JMeter functionality does not support such a complex load scenario, so at Logicify, we use a third-party plug-in called Ultimate Thread Group to set Thread group parameters and examine the system behavior after every new chunk of virtual users is added. This plug-in is also good for stress testing and determining the so-called “system saturation point” — a load point where a further increase in the number of users leads to longer response time or system instability.
- Peak load — short-time extremely intensive load on the server. It allows examining the reaction of the system on such extreme load jumps and the way it recovers back to its initial performance indicator after the peak load is over.
2. Investigate Google Analytics for a Website
If the website under testing is available publicly, Google Analytics reports on audience and behavior would also be helpful in determining the optimal number of virtual users in Thread groups. When testing a company website, the total count of requests (hits) could be predicted based on the number of employees, their working hours, and schedule, plus a certain number of request added on top of these as a reserve.
3. Check System Requirements
Technical requirements for a system performance should be documented in the specification or performance test plan and discussed with the client in advance. They aid in finding the optimal number of virtual users for testing.
If you connect the server with the webpage and JMeter client via a local network, with a more stable connection as compared to the Internet — this could be figuratively called “lab” testing — you minimize the number of factors affecting response time and focus more on the system itself to find potential bottlenecks. On the other hand, connecting to the server via the Internet is a more true-to-life scenario. Yet, it makes root cause analysis and issues isolation harder.
4. Anticipate Traffic Splashes
Find out if the website may see a rapid splash in visitors while you plan performance testing. For instance, when testing an e-commerce application, be ready for seasonal sales or Black Fridays, which attract more visitors. You should be sure that the website is ready for such hit-and-run raids and execute load testing with a large number of virtual users in advance.
Choosing the optimal number of virtual users in JMeter makes testing results true-to-life and more indicative.
An Effective JMeter Test Plan Should...
1. Be Realistic
The test plan should replicate the actual journey of a user on the website. The sum of experiences in customer journey could be taken from Google Analytics Behavior Flow (if the website is available publicly) or drawn based on typical user behavior on a certain website type. For e-commerce websites, for instance, a typical customer journey would be landing on the Home page, searching for a product, reading through product description, adding it to cart, checking out.
Mind the Funnel
The funnels you use while testing should be realistic, too. As a rule, the number of users who visit a Home or Landing page is higher than the number of users actually buying a product. To take this into account while testing with JMeter, you could use Throughput Controller.
It allows defining a ratio of users to have every possible experience on the website. For instance, you could let 100 percent of your virtual users visit the Home page, 90 percent of this amount would look for a product. 60 percent of these users would add a product to their cart, and only 35 percent would proceed to checkout to actually buy the product.
Add “Think Time”
When real users visit a website, they usually need the so-called “think time” to read the content through, enter a search keyword, and analyze search results. These pauses between requests, called Timers in JMeter, should also be added into the test case to make it even more true-to-life. Do not just add 1 - 2 seconds delay between every new request; use an additional, random one in-between.
Here is an example of a Uniform Random Timer in JMeter, with a constant (200 milliseconds) and a random (0 - 750 milliseconds) delays.
2. Be Flexible
In the majority of cases, JMeter test scripts are launched via Command Line — easily and quickly. However, for every change in Thread group parameters, you need to open JMeter script in GUI mode. This is time-consuming and not convenient. To avoid this, you could use variables and property functions for some Thread group parameters:
This way, property functions could be specified directly in the Command Line:
jmeter -t TestPlan.jmx -Jusers=10 -Jcount=50
This saves a great deal of time and efforts and makes test scripts far more flexible.
Moreover, while stability testing, when a system is being tested for a long period of time (5 - 10 hours), you could watch the testing progress and regulate server load in real time using Constant Throughput Timer. With this timer, server load is adjusted with pauses, so you need to target throughput does not exceed the set parameter (in samples per minute):
You could make the target throughput a variable, too. This would allow you to adjust it via Command Line in real time, as the test script is being executed.
3. Use Unique Data for Virtual Users
To make your virtual JMeter users behave more like real website users, make sure they use unique logins and passwords, search for the different product(s) with various keywords, and add different contact details while checkout. Unique user data, such as logins and passwords, could be stored in CSV-files and imported into JMeter while test execution.
4. Target Website Demo Data
There is no sense in using performance testing for a website with 0 content. So before you start executing the test plan in JMeter, make sure there is demo data on the website, such as catalog, product pages with description, forum. Otherwise, the outcome of your testing would not be objective.
How to Analyze Previous Test Reports in JMeter
Test reports are provided by JMeter listeners, which capture responses coming back from the server while test execution and showcase them in tables, graphs, or log files. Some listeners include Summary Report, Aggregate Report, Aggregate Graph. Let’s have a closer look at Summary Report columns:
- Label — with the name/URL of a specific request
- Samples — with the number of virtual users per request
- Average — showing the average time between the moment when a request was sent and the moment it received a response
- Min — showing minimal response time
- Max — showing maximum response time
- Std. Dev. — showing standard deviation from the average value of sample response time
- Error % — with the percentage of failed requests
- Throughput — with the number of requests processed by the server per a time unit (seconds, minutes, hours)
- Kb/sec — with the throughput measured in Kilobytes per second
- Avg. Bytes — average response measured in bytes
Though the report is informative, you should not judge the test outcomes solely on these tables columns. Combine table values with graphs drawn by Listeners.
Visual representation of test reports is especially helpful in analyzing results of stability testing, i.e. when the server was subjected to a high load for a few hours. Let’s say the average response time for a transaction “Product Search” was around five seconds, which is an expected value. However, during the last 20 minutes of test execution, the server response time for the transaction increased to 20 seconds, which is way longer than the expected value. This points out to a potential bottleneck causing system degradation, but the average value in the Summary Report would be around 5,99 seconds. This way, the degradation would go unnoticed. It would be seen only on a graph drawn by the plug-in Response Times over Time, which allows to track every single transaction and locate the problematic ones to further address the bottleneck.
It is important to find the system component that causes degradation and improve it.
Opinions expressed by DZone contributors are their own.