The user ramp up is all too often overlooked in load testing.
When launching a test, we usually divide the concurrent users by the ramp up value that was set before the test.
Let’s take a look at an example of such a scenario. The screenshot below from creating a test on BlazeMeter shows that it takes 20 seconds for all 100 users to be active and concurrent on the site.
If you do the math, you can see that this means five users become active every second.
Now for many users, this linear ramp-up works for them since the maximum concurrency is what matters most the most. But this is not the best way to approach to take.
What’s Wrong With a Linear Ramp Up?
Let’s say you want to test 100k concurrent users. If you run the load test with a linear ramp up and it crashes, you only know that your site or app can’t handle 100K users. You have no idea how much it can handle before that point.
For example: if you split it into four steps or stages, with each step at 25K users, and then you run the test, you’ll get a much more accurate view of where the bottleneck is. If your site or app crashes at 50K, you’ll know that it can handle 25K, if it crashes at 75K, you’ll know it can handle 50K etc.
How to Ramp Up in Steps
At BlazeMeter, we recently had the opportunity to run a test case with a client that needed to load test for a BIG event.
The engagement was very professional, and the engineers and architects knew exactly what their goals were: To run a 200k user JMeter test from multiple geographical locations and simultaneously run many scenarios. There was just one missing piece of the puzzle - understanding the ramp up of the concurrent users.
They shared the test plan with us, and from there we recommended using the Ultimate Thread Group, a popular JMeter plugin.
The Ultimate Thread Group does a great job in allowing multiple thread scenarios with separate ramp up values. Plus it gives the user full flexibility in controlling how they want their load scenario to be.
This test case involved:
- A ramp up of 4K users every minute
- A 120K load burst after the 10th minute
- A 40K load burst after the 15th minute
- Holding the 200K load until the 20th minute
As you can tell, there are multiple load scenarios at play here, with burst loads happening at several different times.
We then leveraged the Ultimate Thread Group to create this scenario.
As you can see, there is a slight ramp up of users every 1 minute, and there is a big spike after the 10th minute.
Note: The values are under 600 because in the Ultimate Thread Group in BlazeMeter the threads are multiplied by the number of engines.
Since we needed to hit 200K users, we knew we needed around 330 Engines in BlazeMeter itself. At the peak of the Ultimate Thread Group, we hit 600 users (600 x 330 = 198,000 users).
Testing from Multiple Geographical Locations
Once we knew how many engines we needed in BlazeMeter, the customer also wanted to test from multiple geographical locations as well.
With BlazeMeter, we can run something called a Multi-Test, which allows us to run multiple combinations of tests at the same time.
We uploaded the JMX script with the Ultimate Thread Group, and then added the tests several times in the multi-test window while overriding the geo-locations.
We were able to effectively mimic the Ultimate Thread Group and the customer’s needs by simulating multiple load bursts and thread ramp up.
What We Learned From This Test Case
JMeter has a vast number of plugins that allow you to test specific needs. The Ultimate Thread Group does a great job of giving users flexibility in their load testing scenarios.
With BlazeMeter, it was easy to upload the JMX and run a 200K user test in order to satisfy the customer’s needs. We are starting to see the Ultimate Threat Group being used more often to fit the specific test plan requirements of JMeter users.