To gather insights on the current and future state of Performance Testing and Tuning, we talked to 14 executives involved in performance testing and tuning. We asked them,"What are some real-world problems being solved by performance testing and tuning?" Here's what they told us:
- "We’ve helped some of the world’s largest companies improve the performance of their infrastructure while understanding what they’re trying to accomplish with their apps. This includes companies such as: Cisco, Go Daddy, GE, T-Mobile, MetLife, AT&T, Boeing, Dell EMC, Hewlett Packard, LinkedIn, New York Life, Oracle, and Unilever."
- "I'm unable to get into the specifics as there are confidentiality concerns. However, a recent example was a scenario where it was desirable to run a particular workload virtualized that had always been run on bare metal before. The application in question was complex with multiple tiers and often ran in a distributed fashion. The intent was to allow it to be hosted for external entities as well as the usual ability to scale up or down as demand required. Unfortunately, the application in question also had a 50-80% slowdown versus bare metal when run virtualized. There were multiple root causes for the slowdown affecting many aspects of the system. Through analysis, each was identified in turn and mitigated or eliminated where possible. In some cases, it was possible to tune for it but in others it was necessary to modify the virtualization tooling, the machine emulator and the kernel's support for virtual machines. Ultimately, it was possible to get all the test cases running within 5% of bare metal. Another example involved a deployment of an application that had extreme low-latency requirements and had a SLA for maximum latency. The challenge was to tune the system for both low latency and to reduce outside interference to acceptable levels. That had to be done without using as real-time kernels prioritize deterministic performance which is not necessarily the same as high performance."
- "We had a social media client that wanted to launch in a week, had a big budget for advertising, and anticipated having 500,000 users in two months. However, they had only tested their application with 10 users. We created a test database to see how the application would function with 500,000 users and it took the home page three hours to load. The client delayed the launch and we re-architected the features and queries to accelerate response. Benchmarking and capacity planning is very important."
- "DNS testing can reveal attacks or incidents with DNS resolution. We recently helped a company identify a DNS attack on their authoritative DNS. The connection errors reported by Catchpoint’s DNS tests were researched and the attack was uncovered. Being able to identify attacks earlier reduces the impact they have on end users and the organization as a whole. The faster attacks can be identified, the quicker mitigation strategies can be put in place. Today, applications use a variety of third-party elements and tags. Visitors to a site don’t care whether a slow down or failure is due to a third party or content controlled by the site owner. Synthetic testing can help organizations identify poor performing tags and quickly resolve performance issues. On Thanksgiving Day some retailers experienced performance slowdowns due to poor performance of a third-party tag. This tag slowed down performance of the site overall. With monitoring, once a poor performing tag is identified organizations can take steps to remove the tag to improve site performance."
"1) Retail has more hybrid apps while they still host their own ERP and POS systems in internal data centers. Everything else (CDNs, DDOS) need tuning and optimization. 2) Online businesses and educational video delivery on-demand application experience. 3) Modern video delivery with movies streamed to movie theaters, must understand all component communications."
- "We provide full-stack monitoring across the entire development and production process. [With] auto-log monitoring, we are able to see full specification data and are able to detect more problems related to connections and architecture. We have automated instrumentation of all processes, containers, and networks. We use AI to identify problems and root causes. Performance engineering is able to leverage this data. We can see more problems in the cloud and in microservices and we’re able to integrate with legacy systems."
- "We’re a DevOps shop so we’re making changes every two weeks. Since we cannot foresee the impact of the changes we are making we need to check production, understand if the environment has changed, analyze quickly, and restore to normal or stop production. A lot of trial and error."
Here’s who we spoke to:
- Dawn Parzych, Director of Product and Solution Marketing, Catchpoint Systems Inc.
- Andreas Grabner, DevOps Activist, Dynatrace
- Amol Dalvi, Senior Director of Product, Nerdio
- Peter Zaitsev, CEO, Percona
- Amir Rosenberg, Director of Product Management, Perfecto
- Edan Evantal, VP, Engineering, Quali
- Mel Forman, Performance Team Lead, SUSE
- Sarah Lahav, CEO, SysAid
- Antony Edwards, CTO and Gareth Smith, V.P. Products and Solutions, TestPlant
- Alex Henthorn-Iwane, V.P. Product Marketing, ThousandEyes
- Tim Koopmans, Flood IO Co-founder & Flood Product Owner, Tricentis
- Tim Van Ash, S.V.P. Products, Virtual Instruments
- Deepa Guna, Senior QA Architect, xMatters