How to Count Retries as a Single Request in JMeter

When testing applications over the network, failures are a part of the game. Apache JMeter™ provides various mechanisms for retrying failed samples. One of these is logic controllers. Logic controllers enable implementing complex testing scenarios with conditional test execution and loops. The While Controller, for example, repeats actions until the looping condition is met (for example, that the test is successful).

There is a problem, though. With each retry, reporting components like the Aggregate Graph or Summary Report show a new sample in the results. So when your test has eventually succeeded after N failed attempts, the reporting will show N failed and 1 succeeded requests. Although this could be the desired result, in most cases testers would like to see just the one sample that succeeded. In other words, all retries should be counted as a single request.

Let's try to fix that. I will show how to do that with the Transaction Controller and some Groovy.

The service used in the tutorial is called BookshelfFaulty. It's a simple service that returns a filtered books collection, but there is a code in it to randomly send back a Bad Request error. This was done on purpose to demonstrate random errors and retry logic.

Note: We don't recommend using random errors in a test, this is only done for demo purposes.

To run this service we need .Net Core SDK 2.1 or higher. Please download and install it from the .Net Core official website if you don't have it already.

After cloning the project locally, open the project folder in a command line tool and run dotnet run to start the corresponding service. Those who are interested in more information about the dotnet CLI tool can find more details here.

The "dotnet run" result should look similar to this:

After the application has started we are ready to start testing!

The failed samples retry can be implemented with one of the numerous looping controllers available in JMeter. In this example, we will be using the While Controller.

The problem is that the While Controller (and other loop controllers as well) generate a result for each sample they create, even if it is a retry request. The question is: how to hide the possible failed test samples generated in the loop and provide a single result?

There is a component in JMeter called the Transaction Controller, which can solve this problem.

To quote the official documentation,

There are two modes of operation:

• additional sample is added after the nested samples
• additional sample is added as a parent of the nested samples

We are interested in the second mode.

If you configure the Transaction Controller to add the parent sample, things will become much simpler. The test is considered as failed or passed based on that sample's result, regardless of the samples it creates internally. The problem is that the Transaction Controller sets the parent result as failed, if at least one of its sub-samples is failed. But we will fix that later on.

Let's create a test project and include the Transaction Controller in it.

1. First, add a Thread Group.

2. Add a Transaction Controller sampler to the newly created Thread Group and configure it.

Thread Group -> Add -> Login Controller -> Transaction Controller

Please pay attention to the "Generate parent sample" checkbox. It should be checked so that the Transaction Controller will create the extra sample and wrap the other samples in it.

3. A While Controller will be used to resend the failed requests. Let's create it.

Transaction Controller -> Add -> Logic Controller -> While Controller

You can learn more about the While Controller from this blog post "Using the While Controller in JMeter."

4. Add the loop ending condition to the While Controller.

The While cycle will run as long as the code in the Condition section is true. In this case, we want to make sure that one of the following is true: either the required data is found (let it be the book's author) or that the failed attempts count is less or equal than the max allowed count (3 in this case). So the correct code for this will be:

The variables "author" and "counter" will be initialized later on.

5, Add the HTTP Request Sampler to the While Controller so that it will be sending the requests in a cycle.

While Controller -> Add-> Sampler -> HTTP Request

6. Configure the HTTP Request to send requests to the service mentioned in the Preparation Steps.

Set the hostname, port and path of the service under the test. If you use default settings from the service method above, the HTTP Request configuration should look like similar to this screenshot:

Now let's configure something to capture the HTTP Request result.

7. The Boundary Extractor can help with that. Let's add and configure it to capture the author from the HTTP response:

While Controller -> Add -> Post Processors -> Boundary Extractor

The Boundary Extractor will store the author data from the response, for determining whether the test succeeded or not.

Basically what the Boundary Extractor does is take the value from the response bounded by the values from "Left Boundary" and "Right Boundary" and puts them in the variable specified in the "Name of create variable." Please pay attention that the variable name is "author", the same variable that is used in the While loop.

Details about how to configure the Boundary Extractor can be found in this article.

8. We have a While Cycle, but how can we update the counter? There is an element in JMeter that allows to increment counters. Its name, surprisingly, is a Counter. Let's add one and configure it.

While Controller -> Add -> Config Element -> Counter

Let's go into a bit of detail here. The Starting value and Increment are pretty self-descriptive. They are the starting value and the increment value of the counter. There is a Reference name section that allows us to specify the variable name to be accessed later.

Looking back at the While Controller you'll see the condition ${counter} < some_max_value. The Counter element is responsible for incrementing the {counter} value.

Let's think about what was achieved so far. We've built the test project, created the Transaction Controller, the While Controller and the Request/Extractor in the cycle.

What seems to be missing is the initialization of the variables for the while cycle. There are multiple ways to create variables in JMeter, including all types of scripts and predefined elements.

9. Here the User Defined Variables element will be used. Add it to the Transaction Controller and add the variables "counter" and "author":

This will create and initialize a pair of variables for the counter and the author so that they can be used in the While loop.

Now our test is ready.

10. Let's set up the reporting to check the results. It is the View Result in Table listener used here, but feel free to check other listeners as well.

After running the test multiple times, results will look more or less like the screenshot below:

It looks like we are having too many failed requests. Even if we take into consideration that we set up the service to fail randomly, it looks suspicious.

Let's check the earlier mentioned article about the Transaction Controller. It says: "Any sample that fails will cause the entire Transaction Controller to fail". So, even if only one sample fails, the entire transaction is considered as failed. Let's fix that.

JMeter supports different types of assertions, including custom script based assertions. There already is a great article about Groovy Assertions so I won't describe its work here. Let's just configure one to change the Transaction Controller assertion logic.

11. Add a JSR223 Assertion element to the Thread Group

12. Add the following script to the Groovy Assertion

The script sets the test result as successful if the author is found. Now, the parent sample will show the successful sample.

13. Let's run the test now and check the results.

There are many more green samples.

14. The final step is to make sure that the Transaction Controller actually sends back the correct result when some of its samples fail.

15. Let's add the View Result Tree and check its results after several tests.

The View Result Tree shows all the samples generated by the Transaction Controller and their children. The screenshot shows three types of samples:

• Samples that succeeded immediately after the first attempt.
• Samples that succeeded after a few failed attempts.
• Samples that failed, after sending the request three times (which was the maximum number of allowed retries in the While loop).

If you run the same script in BlazeMeter, you will also see the retries as a single, successful request in the results.