Redefining Incident Response

If you think your org doesn’t have any incidents, it’s time to change your definition of an incident.

This week we’re joined by Nora Jones, Jeli's founder, and CEO, to simplify incident analysis and explain why so many incidents go underreported. Before beginning her journey as a founder, Nora helped pioneer chaos engineering at companies like Netflix and Slack, where she developed a passion for understanding the intersection of software and people.

A stellar engineer, manager, and founder, we caught up with Nora on the heels of her keynote address at the LeadDev conference in New York.

Episode Highlights

• (1:29) How to lead based on incidents
• (6:50) Learning from incidents
• (14:50) Chaos engineering at Netflix
• (21:34) Why orgs should call more things incidents
• (30:54) Founding Jeli
• (37:01) Nora's advice to eng leaders

Episode Excerpt

Nora Jones: Incident analysis and Incident Management is actually an evolution of my thinking around chaos engineering. You know, I started my career always in risk and always in helping understand incidents and the quality of the situation, depending on the product need at the time, right? We're always making trade-offs to hit deadlines. And I think there could be more beneficial organizations if people were being proactive about learning from incidents and maybe purposefully injecting failure. But my thoughts have evolved over time, like, rather than injecting failure into your systems. You really should be looking at the incidents that you've already had. You know, I saw a lot of people taking that advice I gave, like, and I think it was in 2017, to do more chaos engineering. But I saw it in the absence of looking at the incidents they already had. So they were like injecting chaos. So like, learn from things and see how things reacted. But the hard part was, it's like, there are a million ways you could inject chaos in your organization. You could do 1,000 different things. Are any of those actually going to be useful? Right? Is it useful to fail a data center and see what happens? Is it useful to maybe take one of your devs off the on-call rotation that everyone relies on? Is it, you know, those, there are a million different things that could happen? But the question is, is it useful? And how do you know what kind of experiments run in the way you can learn what kind of experiments are run if it is useful to do that on-call thing if it is useful to take down a data center is to look at your previous incidents, and to look at and not just incidents or to look at your surprises. So look at the things that made you go fewer, like those are good situations to maybe inject chaos, but what I was seeing the industry do was to take that advice and just do any random thing right without actually — it was just more based on a gut feeling which those conversations can be useful, but they're really useful after an investment you already made after something you already spent all this time paying for which was which is an incident, you know?

Click here for the full episode.