Overview of Performance and Load Testing With Apache JMeter

Apache JMeter is one of the most popular open-source tools for performance and load testing. At Logicify, QA specialists widely use it too. In this article, I'd like to share a few tips and lifehacks to make testing with JMeter even more effective and indicative of a web app performance.

Performance Testing Types

Performance testing helps to determine responsiveness and stability of a software application under a simulated workload that is equal to or higher than expected. No other QA practice gives an idea about the number of concurrent users a website could handle without performance degradation.

Writing an effective test plan for performance testing is challenging; in most cases, reason for that being a lack of specific testing goal. Quality assurance engineer performing the testing should determine what he/she is doing this for:

• To check/prove that the system is capable of handling the expected load;

• To determine a system component that causes performance degradation, which could later be optimized;

• To compare capacity of two or more systems or environments.

• Depending on the testing goal, different types of performance testing should be practised.

• Load testing: to analyze the system’s behavior under expected load. While an expected number of virtual users simultaneously “attack” the system, it is easy to record the maximum response time of its major components and discover potential bottlenecks.

• Stress testing: to determine the system’s throughput limits and overall behavior for a potential disproportional or extremely high load. This gives an idea about the maximum number of users the system could handle and how long this would take the system to recover after it fails.

• Peak testing: to subject the system to rapid intensive load peaks followed by regular load. Such load profile can be seen in the real world, consider Black Friday and consumer electronics e-commerce solution: it might receive 10x normal load on the day.

• Stability testing: to ensure the system handles the expected load during a long period of time. This type of testing helps to determine memory leaks and performance degradation, i.e. longer response time after the system has been loaded for a while.

• Volume testing, to see how the system reacts while the quantity of stored and used data is being gradually increased.

• Ideally, performance requirements for a system should be outlined in the specification, performance test plan, or any other documentation on the project’s requirements engineering stage. These requirements are based on the company’s business plan and long-term goals. However, more often than not, on the specification stage, there is no defined maximum response time for a system under a certain server load.

JMeter as a Tool for Load Testing

• Apache JMeter is a powerful and widely used desktop tool for performance testing. Compared to other similar tools, it has the following advantages:

• JMeter is an open-source application.

• It does not require complex installation. Download the installation package, install the tool and simply click *.bat file to run it.

• It can be used both on static (HTML, JavaScript) and dynamic (AJAX, JSP) resources.

• JMeter’s basic functionality can be extended by additional plug-ins.

• It can be integrated with BeanShell, a Java-like scripting language, and Selenium, a framework for web applications test automation.

• The tool has a nice logging and visualization of testing results in graphs, diagrams, tables, etc.

• It supports distributed testing — heavy load could be tested for a single, multiple, or a network of client machines controlled from a single node.

• JMeter supports basic protocols for the test subject: LDAP, SOAP, FTP, HTTP among others.

However, test results you receive with JMeter are (sometimes and to a given extend) a black-box. This concerns the cases when the bottleneck is not in the app or server under testing but local machines’ hardware limitations and/or network configuration. If you fail to create a certain load on server because of these factors, you could perform distributed testing. It is supported by JMeter out-of-the-box functionality and allows to simultaneously launch tests on multiple nodes joined with the server by the same network.

JMeter Glossary

A test plan in JMeter contains the settings and steps to be executed. Every test plan begins with setting a Thread group, whose properties control the following:

• Number of Threads (users) — the number of virtual users that simultaneously execute the given test case

• Ramp-up Period — the period (in seconds) that tells JMeter how long it takes to start the full number of threads chosen. For example, with 30 threads and 180 seconds ramp-up period, a new thread would start every 6 seconds.

• Loop Count — the number of times each user would execute the test.

• Each test thread will execute the test plan independently. Multiple threads simulate concurrent connections to the server. The server load is adjusted as you tune Thread group parameters.

• Most frequently used controllers in JMeter are Samplers and Logical Controllers; they drive any testing process. All Controllers and Samplers are placed under a Thread group. Samplers tell JMeter to send requests to the server (via FTP, HTTP, LDAP etc) and wait for its response. They are processed in the order they appear in the tree. To modify the number of samplers or customize any other JMeter logic, you could use and combine different Logical Controllers.

• JMeter thread executes Samplers without pausing, which does not resemble the behavior of real website users. To add a certain delay between samplers, you could use available Timers. Timers are added hierarchically before Samplers or Controllers to restrict them.

• The outcome and information about any test case gathered by JMeter is accessed through Listeners, which allow to view test results — in HTML/XML files or visuals (graph, tables), and save them for further use.

To find an optimal combination of JMeter elements for every specific test case, it is usually required to run the test script multiple times.

10 Requests vs 10 Requests

The Thread group properties should be setup wisely and follow specific testing goals. Let’s consider two Thread groups examples:

Example 1:

Example 2:

Though in both cases, there would be 10 requests sent to the server (if the test plan contains just 1 request), the loads they’d create would differ. In Example 1, 10 users would start executing the test case almost simultaneously and send 10 individual requests. In Example 2, one user would execute the test case 10 times sequentially. So, the load on server created per Thread group 1 would be higher.

The ramp-up period should be set wisely. With 0-second ramp-up, JMeter would immediately create all virtual users, causing undesirably high load on server. On the contrary, with a long ramp-up period, some virtual users may already finish their job and terminate, while others have not yet started. Then the actual number of users in the system would be lower than expected. This is undesirable too.

The number of virtual users (threads) executing the test would depend on multiple factors, such as your goal while testing, parameters of your local PC and the network, and script complexity.