The other day I was speaking with someone who asked me a few times how we find and hire engineers who are capable of building performance/monitoring tools. I didn't have a great answer in the moment, but after the conversation was over, I realized it's because I fundamentally believe we all are capable of understanding and solving systems performance problems. We're all brainiacs, so to speak.
This has been a theme of my career. If you look back at my pre-Percona days, read the 2nd and then the 3rd edition of my book High Performance MySQL, examine my open source software, look at the ebooks and white papers I've written, or listen to the recordings of any of my dozens of conference presentations, it's a constant refrain.
Solving performance problems isn't rocket science, it's not a dark art, it's not the exclusive domain of an exceptional few. It's something we all can do.
It's a simple, almost axiomatic, set of principles. First, you must understand the goal and a clear definition of performance, then what causes systems to fall short of that, then how to measure those things, then specific tools and processes to fix it. As a consultant, I solved the "hard" performance problems again and again in a matter of minutes or hours by just framing the question in the right way, deciding what to measure, and showing the output to the clients. The results were predictable: every single time, one or more of the engineers would say "I know what that is!" and start fixing it. All I did was figure out what and how to measure and how to analyze it. I rarely, if ever, even diagnosed the actual problem.
I didn't invent any of this. I've just listened and read a lot from other people who are smart and already did the deep thinking. I'm talking about Peter Zaitsev, Cary Millsap, Karen Morton, Brendan Gregg, Adrian Cockcroft, Neil Gunther, Robyn Sands and so many others.
My founding vision for VividCortex was to turn everyone into a performance superhero by showing them the superpowers they didn't know they already possessed. (Our official mission and vision are on our careers page.)
The hard part is the systematic approach, measuring what matters, and floating the most important stuff to the top for humans to make sense of. This is what VividCortex does. I have unshakable faith that the rest will take care of itself, because time and time again, that's what I saw from consulting clients. And that's what I've seen with hiring engineers to help build those tools, too.