The Future of Performance Testing and Tuning

To gather insights on the current and future state of Performance Testing and Tuning, we talked to 14 executives involved in performance testing and tuning. We asked them,"What’s the future of performance testing and tuning from your point of view - where do the greatest opportunities lie?" Here's what they told us: 

Automation

• Shift left with development, coding, and building. More than 50% think about quality but few are thinking about performance metrics in the SDLC, that will come next – single use performance tracking. Automation of vitals analysis. Automate performance testing in the SDLC.
• Increase optimization with automation versus manually, especially in the cloud since that’s where the greatest gains can be made. Better self-tuning.
• Use AI to increase automation and explore applications. Three worlds: 1) Testing – explore apps pre-release; 2) Production – explore an app while it’s running; and, 3) code static analysis. Predict the impact on the user. Propose remedies to improve the UX.
• Self-healing performance engineering and platform architecture. Smarter remediation actions based on better data and better automation scripts. The system knows how to deal with problems. We help build more resilient systems that are smart, self-healing, smart scaling, with failover and redundancy.

DevOps

• I believe there is a place for performance testing integration as a part of DevOps strategy. Coming back to an earlier point, while some new feature is neat and working as expected, how well does it scale? So, while DevOps focuses more on the fact that “our code is functional”, performance testing might give an insight into the scalability factor. 
• Adopt cloud-friendly APM tools which are established. Shift to DevOps team functions applications to get the operations team closer in with their data. Break down silos. Operations team data is relevant and accessible to developers. A strong set of shared data. 
• More toward DevOps and cloud. Must measure stress limits – that’s the future. Learning the stress limits is very important. Everything is becoming microservices based. Must know things are going to work end-to-end. Identify the metrics upon which you measure and stress test.

Integrated Testing

• An integrated, single pane view of the infrastructure and the application so you’re able to automatically able to optimize the performance of the application and the infrastructure. 
• Things like unit tests are almost synonymous with quality code, appearing alongside code in application repositories. I would expect performance tests to be alongside those in future. Performance testing in isolation is a thing of the past, we see more and more load tests happening during continuous integration or deployment pipelines, so I expect that to grow.

Other

• The cloud is dramatically changing the way organizations test today. With more applications moving to the cloud whether SaaS, PaaS (Platform as a Service), or IaaS (Infrastructure as a Service) the testing methods of the past need to be re-examined. New testing tools and methods will emerge to provide more visibility for cloud applications and services.

• It’s always a numbers game – a demand to improve performance. Customers always expect good performance.

• The future of performance testing lies in taking a user-centric approach. It’s not about CPU usage or disk IOPs, it’s about what the user experiences and perceives as performance — whether it’s on their mobile or desktop device.
• The future for performance testing and tuning is always going to be eternal vigilance. Even if there are no software changes, an upgrade in hardware can result in sub-optimal performance if the same parameters are used. In rare cases, modern hardware can even be slower than new hardware so it must be periodically rechecked.  I think the most obvious change in the near future that will attract the most attention is going to be related to the speed of storage. Much of the software stack is designed to expect slow storage and steps are taken to manage that as best as possible. With ultra-fast storage, some old decisions will be sub-optimal. For example, typically an IO request is dispatched and queued with the requesting process being woken up at some point in the future. This requires multiple switches between different parts of the system. Storage now can be fast enough where the cost of switching exceeds the cost of the IO request itself. While this has been expected for quite some time and a lot of the necessary support is already in place, it's still going to take time to adjust to this reality while still being able to cope properly with slower storage.
• Performance testing is everyone's responsibility, more than it ever has been in the past. Platforms that enable customers with different technical skills to start with providing significant opportunities for platform vendors. Giving customers more ways to create and express tests in a load testing context is a great opportunity.

Here’s who we spoke to:

• Dawn Parzych, Director of Product and Solution Marketing, Catchpoint Systems Inc.
• Andreas Grabner, DevOps Activist, Dynatrace
• Amol Dalvi, Senior Director of Product, Nerdio
• Peter Zaitsev, CEO, Percona
• Amir Rosenberg, Director of Product Management, Perfecto
• Edan Evantal, VP, Engineering, Quali
• Mel Forman, Performance Team Lead, SUSE
• Sarah Lahav, CEO, SysAid
• Antony Edwards, CTO and Gareth Smith, V.P. Products and Solutions, TestPlant
• Alex Henthorn-Iwane, V.P. Product Marketing, ThousandEyes
• Tim Koopmans, Flood IO Co-founder & Flood Product Owner, Tricentis
• Tim Van Ash, S.V.P. Products, Virtual Instruments
• Deepa Guna, Senior QA Architect, xMatters