Details
How reusable performance tests help teams catch API slowdowns before AI-generated code reaches production.
Functional tests confirm that an API returns the correct results and handles expected data conditions. They do not necessarily show whether the API will remain responsive under production-like demand.
In the demonstration, approximately 245 functional test steps passed on the AI-generated feature branch. Performance testing then revealed response times that exceeded the established thresholds.
The feature branch added validation, logging, and security-related changes without breaking its functional behavior. However, its added computations slowed an endpoint that needed to remain within a strict performance range.
This illustrates an important risk: AI-generated code can be functionally correct while still degrading application performance.
ReadyAPI allows teams to convert an existing functional test into a load test without rebuilding it.
The converted test retains its API calls, data sources, generated test data, assertions, loops, and chained responses. Changes to the underlying test can also flow into the performance test, reducing maintenance.
Teams can define limits for average response time, 95th-percentile response time, error volume, and tolerance.
These assertions provide a measurable pass-or-fail standard. Performance testing can therefore act as a release gateway instead of relying on subjective judgments about whether an API seems fast enough.
A single performance result has limited value without historical context. Teams need a baseline from a known, stable version of the application.
ReadyAPI’s Compare Runs capability showed that the feature branch increased response time for the tested POST request. This gives developers evidence they can use to investigate the specific code, database, logging, or security changes responsible.
Application integrity requires more than proving that software works once. Teams need continuous evidence that APIs behave and perform as intended as code changes.
Combining functional validation, performance thresholds, historical comparisons, and shareable reports helps teams catch regressions before users experience them.
Practical implication: Run important API workflows under realistic load before approving a release.
Practical implication: Test every AI-assisted feature branch against established performance thresholds.
Practical implication: Reuse existing test logic, data, assertions, and workflows instead of maintaining separate test suites.
Practical implication: Compare each feature branch against a known, stable version to identify measurable regressions.
Practical implication: Make functional and performance assurance part of the regular development and release process.
Functional tests confirm that an API produces the expected results. An API can pass those tests and still respond too slowly under load, creating timeouts or poor user experiences in production.
Yes. The demonstration shows an existing functional test being converted into a load test. Its test steps, data sources, generated data, assertions, loops, and chained API calls are retained.
The session demonstrates assertions for average response time, 95th-percentile response time, maximum errors, and permitted tolerance during a test.
The feature branch passed its functional tests, but its performance test exceeded the defined thresholds. Some calls reached a 95th-percentile response time of 334 milliseconds and an average response time of 254 milliseconds.
A baseline provides a stable reference point. Teams can compare new feature branches against it to determine whether a specific release introduced a slowdown.
Compare Runs places current and historical performance results side by side. In the demonstration, it showed that the tested POST request had slowed by approximately 15 to 25 milliseconds per command.
It provides continuous, measurable evidence that an application remains functional and performs within acceptable limits as new code is introduced.
Hi everyone, very nice to meet you.
My name is Dom and I am a solutions engineer here on the SmartBear team, specifically focused on APIs and API testing.
Just going to share my screen here for a moment so we can all make sure that we can see the ReadyAPI solution.
Let me just pull this up here for a second.
Bit of background on me.
I’m originally a .NET developer, so I kind of come from a programming background.
And what we’re going to focus on today, as Maureen said, is showing the reusability of some of our functional tests and how quickly we can bring them into a performance test to act as that kind of final gateway or that final kind of measuring stick when we are using agentically programmed features or new updates to our code bases.
Where we want to say not only does this maintain the integrity of our application, does it still do what we’ve asked it to do, but are we introducing slow degradation of our services without really realizing due to the nature of some of these maybe lower LLMs which might add too much per request into a new feature.
What we’re going to go through today is building out a logical functional test for this.
It’s not often that a user is going to land on your website, your mobile application, your backend environment and simply just perform one or two API calls and then leave.
In reality, what they’re much more likely to do is to chain these different API calls together in what we’re going to call our first functional test that we have here today.
I’m going to open up the functional test just so you kind of get a look at what that looks like.
You’ll see there’s a large number of test steps there.
Don’t be too frightened.
Really, all we’ve done is we’ve taken these base functional tests from the RESTful APIs that we’ve imported here to ReadyAPI.
We’ve added a number of steps from the ReadyAPI test module, things like a data source.
I’m pulling a list of countries from a JDBC data source connection.
As soon as I’ve pulled those countries in, I’m using some data generation tools.
So, for example, to actually just generate some scores rather than pulling random numbers, random text files, I’m actually able to use our own data generation step to build a date for the match, building a number of away goals or team goals.
And after that, what we’re really doing is just chaining some API calls together.
So, making sure that when we do post a new score that we update that within our table, that we make sure that I’ve ran some validation tests here to ensure that missing fields, negative scores, empty team names—all of these data cases have been validated here in my test.
And finally, what we’ll mention is I have an iteration loop.
So based off the fact that we do have a data source available, rather than just running this test once, a smoke test or a very happy path case if you will, and marking as complete, what I’d like to do is I’m going to repeat this loop until my data source is fully used up, whether that be a loop of 1,000 or 100.
And essentially, what I’m trying to do with this functional test is mimic how a real-life user might arrive on my own website or my own backend service to validate that my API is handling that.
Once we have a functional test that we’re relatively comfortable with, I’ll just execute this now actually so we can kind of see that action and process.
And at the moment, this is just running off my own local master branch.
So we’ll dive into that in a moment.
I can see in the transaction log the rough information there on what test steps have passed, what has failed, the start times for those, but we can see at the moment that we have a relatively happy case path for our functional tests.
Now, we’re not just validating that they are in fact running.
We are actually adding assertions to those as well.
So if I open up my post call here, for example, I can see I have a number of different assertions that are attached to that post call.
A, I’m validating that I get a 201 back.
I’m validating that when I send in some particular JSON files that I’m getting the relevant data returned as part of those assertions.
So there is logic here to validate my own business case.
So at the moment, what we have is a relatively successful and happy case path kind of functional test.
Now, if we wanted to build these for a performance test, rather than having to add some scripting, rather than having to set up 100 virtual users in the background and funnel them through this functional test case that we have, I simply need to right-click on the functional test that we’ve designed and select “Create Load Test” here at the bottom.
Going to call this one “demo”.
And as soon as I’ve finished that, I’m just going to scroll down a little bit here on my screen.
You’ll see that a new load test has been added under my performance test.
This load test is not a blank canvas.
It is not something we now need to go and start from scratch.
Instantly, what we have here is, if I expand this window a little bit, you will see all of the logic that we had built within our functional test.
The data sources, the data generation, the chaining of command to command, passing responses and results from one post call into the next have instantly been pulled into this step here.
And I now have a fully functional test that has been instantly brought over.
If I needed to make a change here back at my core APIs again, that would be mirrored through my functional tests straight into my performance tests, making these much more maintainable.
Now, what we’re going to do is we’re actually going to jump out of ReadyAPI for just a moment, and I’m going to jump into my Visual Studio Code branch where I actually have this local host demo running.
Let me just drag and drop this over onto our screen just for a moment.
Hopefully that is visible to you guys.
Nothing incredibly complex here: a basic .NET application that’s running a controller for our World Cup API in the background.
You can see all of those scores that we simulated here have been logged within my own file.
What I’m going to do now is I’m going to kill this master branch, for example, that I’m running and I am going to run a git checkout command of a new feature branch that a developer has shared with me on my team.
Now I can see here as soon as I’ve run that that there’s some logger information that have been added.
I can see a couple of the controllers as maybe some new code.
Now, as many modern developers might run through, they are, you know, agentic first.
They are used to building these tools using kind of the things you might see in the right-hand side here like a Claude Code, for example.
What I’m going to do is I’m going to ask Claude Code to run a git diff on my master versus my new feature branch.
This is just to get a comparison of what changes maybe this developer has added, what features they have built.
I know from my own simulations here that I have asked it to harden some of our endpoints to add some maybe additional data validation, some kind of logging based off some security concern that our DevOps department came back with.
I’ll let that run in the moment in the background.
It’s actually flagged it for me that there are some potential complex computations going on here.
Now, what I would say is if I was a developer and I wanted to test these as a new branch of code, maybe I’ll run my set of functional tests off them.
I know that there’s been changes that have been added to this new feature branch.
So, let’s dive in and see if we can catch those potential issues within our existing set of functional tests.
This is the exact same functional test here.
We haven’t kind of changed anything.
Nothing new, I suppose, has been added or changed in that instance.
But what we will do is we’ll hit run on these and we will trigger the exact same approach where we are going through our data source.
We are looping these data sources together and we are validating that all of our functional test cases have passed.
Once we jump into the transaction log, I can see here again that the roughly kind of 245 steps or so have passed within my test case.
And I know that relatively it was a happy path test case, but my functional test have a number of assertions as we showed there with on our post call.
Or I can jump into one of these as well.
I can see that there are assertions for each of these post calls.
Checking for a 404 for a malformed UUID.
In this case, checking for a 404 response if a non-existent match is found.
All of these have passed with my new branch of kind of AI generated code that a developer has shared with me.
At the moment, I would say, well, this branch should be ready to go live.
There’s been no massive loss.
I can see here that my functional tests are still passing.
Okay, but let’s dive in and see what we can do with our performance tests.
The reason why we ran that initial one based off the master branch, as you can see here on the statistics page, is to build those baseline numbers, to build those baseline statistics.
The only way for us to really validate whether there is any degradation to a performance or potential slowdown is if we have a measuring stick to go off.
What I would often ask here is a QA member on the call or if there’s anyone from an active development team: if it got to Q3 or Q4 towards the end of a year and you started to notice some timeout errors, for example, on a particular API endpoint, or the product owner came to you and said we’re noticing a slowdown and we’re seeing some complaints, for example, from specific users—what happened?
What changed in that API?
Why are we seeing a slowdown?
If all you have is a list of git commits or pull requests or feature branches that have been merged into that API endpoint, it’s extremely difficult to differentiate.
Well, was the slowdown caused by a new feature?
Was the slowdown caused by a security update?
Was the slowdown caused by a link command that had changed or maybe a DB schema that was updated in the background?
By running these types of performance tests and basing them maybe off a baseline metric, we can start to really build out a use case where we can measure per feature release, per execution of a staging branch, what has happened that has changed our initial API.
What we’re going to do is we have the same load tests.
I’m going to open up some of the assertions that we have.
I believe in our initial reporting metric, we had a time taken average.
I’m going to open this one up here for a second.
I can see that that’s set at 85 milliseconds.
I’m going to increase that.
We had some on our initial one there; I think that was all the way up to kind of 135.
So, I’m going to set that as 110, for example, per millisecond.
I can also add, for example, tolerance.
If we would say we would allow certain users to maybe be a little bit slower or the average temp test case maybe on the initial run or on the last run as the database starts to fill up with data.
I can be very prescriptive here about what tolerance I will allow per second or, for example, the max number of errors we will allow before we potentially stop that test case.
This is an assertion that I have added at the overall performance case level.
And if I wanted to be more prescriptive than that and actually jump into an individual scenario here, I can click on the functional test that I have generated and I can see that, for example, the time taken average, the 95th percentile—can see by that X mark there.
This actually initially failed within our baseline model.
So let’s have a look at that.
We’ll say the 95th percentile is 130 milliseconds.
Let’s increase that out to maybe 145 for this new feature branch that we have.
We know there’s been some changes to this model.
We want to make sure that they’re allowed.
And we’re going to be very, very strict on this one.
We’re going to say that the max errors is zero within 5 seconds.
We’re not allowing anything here because I know that this endpoint, perhaps it’s tied to payment processing service, tied to something in our backend.
I simply cannot allow this endpoint to go over the 150 millisecond mark or else there are issues and potential timeout issues down the line.
Now that we have kind of that baseline set up, let’s execute our test case again.
I’ll just jump in here for a second and I’ll just clear these existing match scores, if you will, so that we have a relatively clean run.
And what I’m going to do is, based off this new feature branch, I’m going to hit “Run Test”.
And we’re going to trigger to see how our 10 seconds go.
Just hit... take a second just to get those virtual users loaded up.
I can see that I’m allowing kind of a warm-up time of about 10 seconds.
This is going to execute again for about a duration of 10 seconds.
And I can see the failures have started to spike up quite a bit there.
I can either jump in to review the errors at an individual level here using my log files.
So, for example, within the ReadyAPI screen, I can see that the kind of the 95th percentile that we set up a moment ago, I’m double, if not triple exceeding that at certain calls with a 334.
Similarly, the time taken average, which was 0 to 110, I’m hitting up to 254 milliseconds on that.
Perhaps these logging files can be helpful for the QA member who’s actually jumping into the tool.
But if we needed to build these out again using a report so that I can share them with members outside of the team, I’m going to generate this report.
But what I’m going to ask is I’m going to ask it to include the statistics history.
Now what we’re starting to build is a baseline performance test for our master branch compared to this new AI-generated code new feature branch.
And once this is generated, I can jump in—the exact same overview charts that we have a moment ago.
I can see them compared based off my baseline model or I can generate these and compare specific metrics based off the individual new feature branch that we have running here a second ago.
What steps were the fastest to execute?
What steps were the slowest to execute?
I can see that that post call that I had triggered, for example, as a bit of a canary in the coal mine, has now suddenly jumped up to about 156 milliseconds on certain occasions of my test execution.
I’m going to return finally just to our statistics page.
I mentioned how important it could be.
For example, you can see now that those times, those red lines that we had a moment ago, have altered.
They have changed.
But without comparison data, it’s really invaluable to me.
What I simply can do here is, using the “Compare Runs” tool, I can select a historic run.
Going to select the one from just a couple of moments ago, the baseline model that we ran in our demo.
And I can see instantly here, when I start to compare things like the average time taken or if we just focus on... clear these out for just a moment and we’ll just focus on that post scenario.
I can see that that post scenario has instantly jumped by about 15 to maybe 25 milliseconds per command.
Now it got faster as we got running, but instantly I would have some concerns about what features have been added to slow down, for example, that particular post command.
I can go back to the developer and I can say: “Has something changed in the link query? Has something updated in our DB?”
Because when I start to compare these statistics again, if I start to look at run one versus run two or if I jump into some of the minimum commands for example here, I can see that the run time was much, much slower to get started with.
These are the type of kind of baseline statistics that we can build and, as I mentioned, share these out immediately as a Jira report if needs be.
Sorry.
Now, again, I’ll just resize this so we can all take a look.
I can attach exactly what project files, what log files that I need to and instantly have those reports maybe sitting on Maureen’s desk tomorrow morning to explain, for example, why that new feature branch wasn’t quite ready to go.
Presenters:
Dom Crosbie
Solutions Engineer at Smartbear
Join Now for More Content & Events
For event and sponsorship inquiries, please email: [email protected]
