Is Endurance Testing Dead?
Is Endurance Testing Dead?
Endurance testing, or soak testing, has somewhat fallen out of grace with the rise of DevOps and continuous testing. Is it time to bury it for good?
Join the DZone community and get the full member experience.Join For Free
Learn how error monitoring with Sentry closes the gap between the product team and your customers. With Sentry, you can focus on what you do best: building and scaling software that makes your users’ lives better.
Recently I posted a question on the LinkedIn “Performance Specialists” group regarding soak (or, endurance) testing and whether it is required if the team wants to achieve daily deployments into production. I have to say, it was an interesting discussion with different views. That discussion led me to write this post about my own thoughts on the topic. Before we get to that, let’s first define what endurance testing is and its objectives.
What Is Endurance Testing?
Endurance or soak testing is a type of load test that normally runs for an extended period of time (a few hours to days) while the system under test is subjected to a production-like load (or anticipated load) to understand/validate its performance characteristics.
Why Conduct Endurance Tests?
Endurance tests are conducted to answer different kinds of questions such as:
- Does my application performance remain consistent or degrade over time? For example, your application might have a resource leak such as memory or connection leak which manifests slowly over time and impacts your system.
- Is there any interference that was not accounted for and could potentially impact system performance? Example application backups/VM backups/batch jobs/third party jobs running at different times of day that might not have been accounted for in other tests but do impact system performance.
- Are there any slow manifesting problems that have not been detected by other types of test? Other then the resource leaks, you could also detect problems which are due to sheer volume of data. Example of such an issue is a full table scan on the database. As the data grows, the database queries start to slow down because they are doing a full table scan. Similarly, running out disk space because too much data is being written to a log file which in turn causes the application to crash.
Now let’s get back to the question of whether endurance testing is required or not if you want to achieve daily deployment.
My personal view is that there is no simple answer to conducting endurance test and for how long to achieve daily deployments. Would you want to conduct an endurance test that lasts for 12 hours, if you are making a text change? Probably not. What about if there is a rewrite of a function that makes calls to a database and third-party API? Probably so.
In the end, it will all come down to the level of risk that team/stakeholders are willing to take to deploy code into production and stand by their decision, and how quickly they can mitigate performance issues in production without impacting brand image, sales, user experience and so forth.
However, I don’t believe endurance testing is dead. There is a place for it in the continuous deployment world. It just needs a little bit more thinking and planning (different approach may be), if you want to achieve daily deployments. You can run soak tests overnight, analyze and share results in the morning with rest of the team or conduct it over the weekend. Another approach could be that the team reviews which changes they believe are of low risk and therefore are the best candidates for deployment during the week without requiring an endurance test. While high-risk changes undergo endurance testing over the weekend before deploying into production the following week.
Finally, there need to be monitoring and alerting tools in place to identify performance-related issues (be it in production or non-prod). Any issue identified in production also need to be fed back to performance engineering team. This will help them improve their performance testing process.
Published at DZone with permission of Harinder Seera . See the original article here.
Opinions expressed by DZone contributors are their own.