To gather insights on the state of performance optimization and monitoring today, we spoke to 12 executives from 11 companies that provide performance optimization and monitoring solutions for their clients.

Here's what they told us when we asked, "What skills do developers need to have to optimize the performance and ease of monitoring their applications?" 

Design for degradation. Educate yourself upfront with regards to the operating environment, the end-user environment, and how the application will be used — LAN versus WAN versus internet versus mobile. Think ahead about the consumption environment.

Don’t assume that the model you have in your head is correct. Measure and see where the bottlenecks are. Don’t guess where performance problems will be. Explore global latency. Matrix and evaluate under real-world loads. Use as a baseline to push and improve your own systems. “The backup never fails. It’s the restore that does.”

Care the correct amount about performance. It’s not economically feasible to replicate the full environment when there are plenty of other tests using a representative sample and scenarios. You’re going to get it wrong. Design in breakable themes. You’re going to get pieces wrong. Make it easy to make changes to elements of your code.

Set a reliable benchmark or baseline for your true performance goal (i.e., access in 60 to 100 milliseconds). Figure out which performance metrics are most important to the business. Know how you are performing over time relative to the benchmark. Know what the end user is experiencing. How to tie together what you are learning into something you can use as a developer. Spend time to determine how to get alerts to determine value.

You can’t solve every problem with design. See more. Empower the developer with the right knowledge and ability to make design decisions securely. Knowledge of how design impacts and find creative solutions. Stay in a creative mode.

A lot of companies are not following best practices for app development. This is amplified in big data because multi-tenancy can bring the system down. Do perform tuning and analytics up front to ensure you are not affecting anyone else. Collect data across the stack. Always have full stack visibility. This enables you to have a more processed way of getting apps to production. Understand security and data governance issues. Understand data use and resource use.

Have common sense and experience in creating well-performing applications. Have the ourage to limit functionality and convenience of APIs to have better performance.

Application developers and DevOps engineers need to eliminate guesswork and have a precise understanding of how their unique applications will perform on their underlying infrastructure. They need a predictive understanding of how every single change (even minor ones) will impact performance. They need the accuracy provided by a real-time feedback platform that persistently analyzes all changes and updates, and the resultant impacts on application response times. The key new skill is to have an authoritative understanding of the underlying IT infrastructure and the expertise to keep it running smoothly in the face of constant change, independent of vendors or whether it is on-premise or in the cloud.

They need to have some experience, that’s for sure. I do not think somebody that never saw the lights of production would handle optimization well. I would prefer someone with DevOps experience, although a developer that had access to monitoring tools would also be adequate. Developers need to have knowledge of the production environment. That means they should be able to pull out stats from a production server, be it an app server or a DB server. Understand logging and use it to signal problems in production. If monitoring tools are in place they need to be able to understand their output. When encountering problems while monitoring load testing or production environment, those involved in these processes would require the right skills to address the problems. For example, if there is a problem with a database, you should know how to add an index, move files around or add another DB server. Should the problem be a memory leak they would have to know how to do a heap dump and investigate what component is leaking.

The scientific principles (o-notation, etc.) to optimizing a program still apply today, of course. So, any programmer should know them. The biggest challenge today is to be able to see and analyze the bigger picture. Optimization no longer lies in the code alone. Developers need to be able to understand the architecture of the system, how services talk to each other, how the database is accessed, how messages are read by concurrent consumers, etc. This requires a broad way of looking at things, accumulated experience, and an analytical mind.

Think like a customer and think about what parts of the application would most impact you as an end user. Start from the user persona and build from there.

By the way, here's who we spoke to!

• Josh Gray, Chief Architect, Cedexis.
• Jeff Bishop, General Manager, ConnectWise Control.
• Bryan Jenks, CEO and Co-Founder, DropLit.io.
• Doru Paraschiv, Co-Founder, IRON Sheep TECH.
• Yoav Landman, Co-Founder and CTO, JFrog.
• Jim Frey, V.P. Strategic Alliances, Kentik.
• Eric Sigler, Head of DevOps, PagerDuty.
• Nick Kephart, Senior Director Product Marketing, ThousandEyes.
• Kunal Agarwal, CEO, Unravel Data.
• Len Rosenthal, CMO, Virtual Instruments.
• Alex Rysenko, Lead Software Engineer, and Eugene Abramchuk, Senior Performance Engineer, Waverley Software.