Details
How engineering leaders can assess readiness, build trust, and adopt AI-led incident response at the right pace.
AI coding tools are increasing development velocity and system complexity. More code, services, and integrations can also mean more incidents and longer investigations.
While software development now moves at AI speed, many incident-response practices still operate at human speed. AI-assisted investigation is becoming necessary to help on-call teams keep pace without increasing burnout.
AI excels at machine-scale analysis, system-wide correlation, and continuous coverage. Engineers provide the business context, judgment, and accountability AI lacks.
The goal is not autonomous incident response. It is a partnership in which AI handles time-consuming analysis while engineers retain control over consequential decisions.
The session divides incident response into four stages:
Investigation: AI can lead by gathering evidence, testing hypotheses, and identifying likely causes.
Coordination: Engineers should lead stakeholder communication, delegation, and response decisions while AI documents progress.
Action: AI can prepare a fix, but an engineer should approve, modify, or reject it.
Closing the loop: AI can draft objective details, while engineers own the lessons and follow-up actions.
This model gives leaders a practical way to decide where automation is appropriate and where human oversight remains essential.
Datadog’s SRE team found that adding AI to noisy monitors and poor-quality signals only amplified existing problems.
Before introducing AI, teams need clean observability data, customer-impact-based alerts, clear ownership, maintained runbooks, and mechanically enforced monitoring policies.
Datadog began with lower-risk uses such as summarizing incident channels and drafting postmortem timelines. It later expanded AI into investigations and proposed code fixes.
This phased approach allows teams to validate accuracy, understand failure modes, and build confidence before using AI in more consequential workflows.
AI can gather telemetry, brief late responders, document timelines, summarize impact, and prepare potential fixes.
However, engineers should continue to own decisions that shape reliability outcomes. That includes approving production changes, explaining why an incident happened, determining what the team learned, and assigning follow-up actions.
Practical implication: Use AI to analyze telemetry and correlate system changes at the same pace that AI-assisted development creates them.
Practical implication: Let AI handle large-scale analysis while engineers retain responsibility for context, judgment, and accountability.
Practical implication: Give AI more autonomy during investigation and documentation, but require human approval for consequential actions.
Practical implication: Improve observability hygiene, alert quality, ownership, and runbooks before introducing AI.
Practical implication: Start with safe, repeatable workflows and expand only after the system earns the team’s trust.
AI-assisted development is increasing code volume, deployment frequency, and system complexity. AI can analyze logs, traces, services, and deployments faster than an individual engineer, helping response teams keep pace.
AI is best suited to initial investigation, signal correlation, hypothesis testing, responder briefings, timeline capture, and first drafts of incident documentation.
Engineers should lead coordination, stakeholder communication, production-change approval, novel failure analysis, postmortem learning, and decisions that require business context or accountability.
Teams should improve observability hygiene, connect alerts to customer-impacting SLOs, assign clear monitor ownership, maintain runbooks, and enforce monitoring policies mechanically.
Introduce AI one workflow at a time. Begin with lower-risk tasks such as summaries and documentation, validate its performance, and then expand into investigations and proposed fixes.
A purpose-built agent can access live telemetry, runbooks, previous incidents, and organizational context without responders manually copying information into a chat tool. It can also retain incident learnings and operate inside established response workflows.
According to the session, Datadog integrates with PagerDuty and also offers Datadog On-Call as an alternative. Its incident-response workflows integrate with Slack, Microsoft Teams, and Google Chat.
Dominique Roller: Hello everyone and welcome to today’s DZone webinar, “Pathways for AI Adoption and Incident Response”. I’m Dominique Roller and I will be your host for today’s session. Before we get started, we have a few housekeeping notes. Today’s session is being recorded and we’ll share the on-demand recording with all registrants after the event. If a question pops up during the presentation, please send it through our Q&A tab and we’ll save time at the end and get to as many questions as we can.
Now, let’s get into today’s session. AI is becoming a bigger part of how engineering teams think about incident response. It can help reduce noise, speed up investigations, and take some pressure off on-call teams. But when an incident is high-stakes, leaders need more than automation. They need to know where AI can be trusted, where human judgment still matters, and how to introduce these capabilities without creating more uncertainty during critical moments. That’s what today’s session is focused on.
Curtis Maher from Datadog will walk through a path towards AI incident response, drawing on lessons from Datadog’s reliability engineering team. By the end of this session, you should have a clearer way to think about where your organization stands today, what real-world AI use cases can look like in incident response, and how to adopt AI at a pace that makes sense for your team, your systems, and your risk tolerance.
With that, I’m pleased to introduce today’s speaker. Curtis Maher is a product marketing manager at Datadog, where he leads go-to-market strategy for incident response. He works with customers to improve how their teams detect, respond to, and learn from incidents. And he brings those real-world lessons back into product strategy to help teams move faster under pressure. Curtis, thank you so much for joining us today, and I will hand it over to you.
Curtis Maher: Awesome. Thank you, Dominique, and thank you everybody for joining. I’m really excited for the session today. So, let’s get into it.
All right, so here’s where things are headed. We’ll start with what’s changed about incident response and why AI has now become necessary. Then I’ll walk through how AI and engineers split the work across the incident life cycle. And we’ll close with what our own SREs have learned running this in production and how other engineers have found value along the way too. Let’s get into how AI is shaping engineering today.
So let’s set the scene. Your team isn’t using AI to autocomplete a line of code anymore. They’re running Cloud Code, Cursor, Copilot, CodeX, and other coding tools, handing off whole tasks, and reviewing the diff. And the scale of those tasks is quite significant.
One report from SemiAnalysis estimates Cloud Code alone is writing around 4% of all public GitHub commits as of March this year, and they’re actually projecting 20% by the end of the year. So that’s just one tool and on public repos, so the number inside your private codebase could actually be higher.
Undoubtedly code is shipping at a different pace. PRs that took days are now landing in hours. Velocity is up, throughput is up, but that velocity has a cost, and the cost shows up in production.
We’re seeing that cost play out in the data. In the 2025 Google Cloud DORA report, we found estimates that if teams fully adopted AI in development, they’d see a 29% increase in system instability.
The quote on the slide here says it best: “While teams are adapting for speed, their underlying systems have not yet evolved to safely manage AI-accelerated development”.
In other words, code development is moving at this AI speed, but our responses are still moving at human speed, and that gap is actually widening.
So what does that gap actually mean for on-call engineers? Well, it means more incidents. So naturally, as the volume of code shipping goes up, the surface area for things to go wrong goes up with it.
It means more complexity because coding agents are helping teams build more services, more integrations, and more moving parts than ever before—more than one engineer can hold in their head.
And those lead to the worst part here, which is more stress. There are more incidents and there’s more complexity, which means higher incident volume and higher incident length. Engineers are increasingly burnt out by the pressure of managing reliability in systems that are evolving faster than they can keep up with.
So what do we do about that? Let’s talk about what the future of incident response in the AI era looks like.
The answer isn’t just hire more engineers. AI created this pace of development, and really only AI can match it in production. Some of the hardest tasks about being on call are actually some of the best use cases for AI.
So if you think about what the first hour of an incident looks like, you’re sifting through hundreds of services and a million log lines trying to find the right signal. No human is reading all of that the way that AI can today.
You’re also connecting dots across the system. So a latency spike here, a config change there, a deploy from two hours ago. AI does that faster than you can even open the three tabs you need to reach through.
And importantly, AI is always on. It doesn’t sleep, it doesn’t have to context switch, and it doesn’t have a backlog of other work that it’s waiting to do that it probably would rather do. So that’s really the case for AI.
But AI alone can’t own reliability because for everything that AI is good at, there’s a set of things it structurally just can’t do. And those happen to be the exact things that your engineers are best at. Let’s walk through this.
So AI lacks business context. AI will see a degraded service, but it doesn’t see that it’s degraded for your biggest renewal, or that the team that owns it is in the middle of a launch and that a rollback would tank their whole week. Engineers can bring that context.
Also, AI can’t exercise judgment. AI is strong when the pattern looks familiar, but the novel failure, the call between two fixes where one makes things worse, or the moment in a war room when you decide to slow down or push through—human engineers need to make that call.
Then lastly here, AI can’t carry accountability. Someone has to authorize taking a risky decision. They have to have their name at the top of the postmortem and they need to be the voice on a customer call. Human engineers have to carry that weight.
So the real question doesn’t come down to AI or engineers, but it’s how to put them together so they each play to their strengths. And that’s what the rest of this session is really going to be about.
This is the partnership that we believe in. Not AI replacing engineers, not engineers trying to outrun AI, but both working together, each doing what they’re best at.
So AI brings things that engineers can’t: machine-scale analysis, system-wide correlation, always-on coverage. It’s really able to do the heavy lifting that no human is able to match individually.
Engineers also bring things that AI can’t: that business context we discussed, sound judgment, real accountability. They are the ones owning decisions that actually shape the engineering outcomes.
When you put them together in one workflow, you get the best of both. You have speed without losing control, scale without losing context. That’s what we’re going to spend the rest of this session showing you.
We think of incidents in four phases, and in each one, you’ll see a similar pattern. One side takes the lead and the other comes in where they add the most value. And we’ll show it through the lens of how Datadog’s own teams work.
What makes all of this possible for our teams at Datadog is our incident response toolkit. This combines observability, alerting, and incident coordination into a connected workflow.
So when an issue occurs, you can automatically page a responder using Datadog On-Call. Then manage the incident by automating response steps like initiating that war room or creating the Slack channel, and then recording every action and learning along the way in a single source of truth.
This gives teams visibility during chaotic incidents, and it helps them learn from incidents to make proactive reliability investments.
Having this single source of truth is invaluable for the human engineers responsible for owning incident outcomes, but it also makes it possible for AI agents to work alongside your response team by sharing that live incident context that it needs and recording its actions alongside the rest of the team.
Now specifically, our AI agent, Bits Investigation, is what we’ll focus on in this webinar. Bits Investigation is an AI teammate that does the work of an on-call engineer. It investigates alerts as they fire. It pulls from your telemetry and runbooks, and it reasons through the problem like a senior SRE would.
What makes Bits Investigations powerful is that it gets you to the root cause in just minutes. Let’s talk through what this looks like in practice.
Now to get concrete, I’m going to walk you through four stages of the incident life cycle: Investigation, Coordination, Action, and Closing the Loop. For each one, I’ll show you who leads, who supports, and how that handoff works in Datadog.
The first stage is the initial investigation. So, the moment that alert fires, this is really where AI should lead. And here’s why. The first minutes of an incident are mostly information gathering.
You get paged. The first thing you’re doing isn’t immediately fixing anything. You’re reading dashboards, checking deploys, scrolling through logs, and maybe asking yourself, “Is this related to that thing that pushed yesterday?”. That work is essential, but it’s actually exactly what AI is good at.
Bits Investigation, our AI agent, kicks in the moment the monitor fires. It investigates the alert. It correlates signals across services, traces, logs, and deploys. It runs through hypotheses. It rules things out. So by the time that the engineer gets to their laptop, there’s already a conclusion waiting.
Now on the slide, you can see Bits investigated a high-latency alert on a flight search API. It traced the latency through the query engine, found CPU saturation on the flight database service, and identified a slow SQL query as the root cause.
And that was six minutes end-to-end without any human involvement. So as an SRE, instead of starting from an empty search box, you start with the root cause already identified.
Bits tells you, “Here’s what happened, here’s my evidence, here’s what I would do next”. Now, the engineer does what engineers do best. They apply their judgment.
They might ask follow-ups: “What’s the source of the traffic surge? Is this tied to a customer that we know about?”. They add context that Bits just simply doesn’t have. And then they validate the findings against what they know about the system.
The shift is that the engineer here isn’t doing the grunt work; they’re doing the part that they are best at.
Now, the second stage is coordination. So, you’ve opened up your laptop, you can see the root cause. Now, the work shifts. How do you loop in the right people? How do you make sure stakeholders are aware of what’s going on? And how do you make sure that the right fix actually gets put into place?
This is where engineers have to lead. Coordination is a people job. And to be clear, this doesn’t mean that nothing is automated. Plenty is, but an engineer is the one that is building that structure. It’s setting up automations and then it’s ultimately driving the response during an incident.
So you have an incident commander delegating tasks, deciding when to page another team, owning that communication to leadership and to customers. None of that should be on autopilot.
The incident commander needs to be somebody who understands the stakes, who knows that the service throwing errors is maybe mid-migration and half the team is already heads-down on that cutover, or that knows that the responder for another team is in a different time zone and might not see the incident for another few hours.
With Datadog, the moment you declare an incident, we can automatically spin up the structure for that engineer-led response—creating a dedicated Slack channel and video bridge, paging the right responders that are available at the right time, and building status pages with a click when it’s needed.
But underneath that, AI is doing a lot of work that used to fall on the incident commander or the incident scribe.
It’s briefing new responders. So when somebody joins a Slack channel 20 minutes in, they can get an AI-generated catch-up telling them what’s happened, what’s been tried, and what’s coming next. The incident commander doesn’t have to stop what they’re doing to re-explain to a new participant in the incident.
It can help by capturing a live timeline. So the tool can capture every status change, every role assignment, every action taken automatically, with the graphs and the relevant context from the incident.
With that, Zoom meetings can get transcribed and summarized and built into the same timeline. So, you’re no longer relying on a human to write down everything that’s going on throughout the incident.
And then lastly, summarizing live progress. So, in Datadog, the incident overview continuously updates itself with an AI-generated summary of what’s happened, what the impact is, what the affected services are, and what the responsible team is doing to address this issue.
This is the kind of work that sometimes gets dropped under pressure when humans are responsible for it. But in this case, AI does it every time without question.
The third stage is actually fixing the issue. Action is interesting. The right pattern here is a first-pass model: AI prepares the fix, the engineer approves it.
On the slide, you can see Bits found an unbounded cache memory leak in an inventory metadata service. It didn’t just say, “Go fix the cache.” It generated a code fix, a real pull request scoped at the specific function with the exact lines to change.
And that fix is then staged for review. There’s a “Create PR” button right there in the investigation.
The same pattern applies for operational fixes: scale up a deployment, roll back a config, restart a service. Here, Bits prepares it, names it, points it at the right cluster, but waits for the human that’s in charge to hit run.
The engineer’s job here is the final call. They review what Bits is proposing. They step in for novel cases where the decision is lower confidence or the situation is unusual. They own the decision ultimately, because deploying a fix during an incident is exactly the moment where you want a human accountable.
What changes is the engineer’s starting point. Instead of figuring out what to do, it becomes approve, modify, or override. That’s a faster decision, and it’s the right place for human judgment.
Now the last stage is closing the loop. You’ve taken action to resolve the incident, now you have to learn from it.
Postmortems and reporting are actually one of the hardest places to draw the line between AI and human engineers. That’s because they’re so tedious that everyone wants to automate them, but they’re also the most important step if you want the team to get better the next time.
That said, there are absolutely parts where AI should take the first pass—the timeline, the impact summary, the chronology of actions.
As you manage the incident, Bits has been capturing all of your team’s actions the entire time—every Slack message, every graph, every resolution step. So, pulling that into a draft is exactly the kind of work that can be easily offloaded.
But when it comes to shaping the narrative, this is where the line matters. At Datadog, we rarely, if ever, let AI generate the full postmortem on its own.
The sections that drive learning—what we learned, what we would do differently, the follow-ups configured with names attached—those have to be written net-new by the SRE running the investigation.
And that’s deliberate. We don’t want an engineer opening a doc, seeing it’s already written, and then clicking submit, because if that happens, you’ve automated away the part of the process that actually changes how teams operate.
So AI handles the objective information and engineers handle the critical subjective details.
So our own SREs use Datadog for incident response. They’re the first ones using these new AI features as they get deployed to production, and we’ve picked up a few lessons along the way.
So that’s what we want to share here in this session, partly so that you can skip some of the mistakes that were made and the learnings along the way, and partly so you know that what we’re shipping is something that we trust ourselves at Datadog.
Let’s get started here. The first one is build the foundation first. Early on, we tried layering AI onto the monitors and signals that we already had, and it didn’t fix issues of noise; it made existing noise even louder.
So AI should not be expected to repair bad data where you have it, because it’s only going to amplify that bad data. Before you turn anything on, make sure your observability hygiene is clean.
So number two here: anchor it to customer impact. We used to page on infrastructure symptoms like CPU, memory, queue depth, and then force engineers to figure out whether any of it actually mattered.
That interpretation burden was a real source of fatigue. So what you should be doing is tying your paging to SLOs on real user journeys instead. If customers can’t feel it, no one in your organization should be getting woken up for it.
Number three here: enforce policy mechanically. Monitor hygiene through good intentions doesn’t work—we tried. Runbooks went stale, ownership got murky, and priorities drifted.
What you need to do is put the rules in code. Every monitor needs an owner, a runbook, a priority, all enforced in Terraform, not just in convention. Once that baseline exists, AI is great at helping keep it clean.
Number four: respect what makes reliability hard. We have supported over 30,000 customers. We’d love to tell you that AI is going to fix every bug and prevent every outage, but right now it can’t.
Reliability is too high-stakes a job to hand over completely. Keep humans in the loop for the decisions that matter. Maybe we’ll revisit this claim in the future, but right now we have to respect what makes reliability hard. The difficult calls that have real stakes need human intervention.
Number five: deploy AI for context. Where AI really earns its keep is in the legwork—summarizing channels for the late joiner, surfacing patterns across past incidents, drafting the postmortem. So, it’s important that you aim the AI at that toil that you’re experiencing in your day-to-day.
And then last here: earn trust per use case. We didn’t turn AI loose on everything all at once. We started where it was clearly safe and clearly useful—things like summarizing incident channels and drafting postmortems.
Now we’ve expanded it to incident investigations and code fixes. So the lesson is roll AI out one workflow at a time, because that trust isn’t just granted, it’s something that you’ll earn and your organization will realize over time.
And we’re not the only ones running it this way. These are some of the teams using Bits investigations and incident response on their production systems today across retail, finance, healthcare, media, and more.
There are two quotes worth calling out here. Our engineering leader at Uber Freight talks about a 70% MTTR cut from day one, and that it felt like adding a senior engineer to their team.
Then Jason at Arc XP talks about how incident response has enabled their team to reduce incident volumes by a tremendous 86%, letting them focus on that higher-value engineering work.
All right, thanks everybody. Dominique, I’m going to pass things back to you.
Dominique Roller: Thanks so much, Curtis. That was a really helpful way to think about AI adoption and incident response, especially the difference between AI for routine work and building trust for higher-stakes moments.
So we’ll go ahead and move into our Q&A portion of the session. As a reminder, you can submit questions through the Q&A tab and we’ll get to as many as we can. All right, Curtis, let’s see what we have.
All right, first question: “We already use PagerDuty. Does Datadog replace that or work with it?”
Curtis Maher: Yeah, that’s a great question. You know, the honest answer is a little bit of both here.
So, Datadog does integrate with PagerDuty if you’re looking to keep using that tool. But it’s also true that we have our own paging product called On-Call, and it can replace PagerDuty entirely.
Whether you’re using PagerDuty or Opsgenie or any of the other paging tools out there, we have our own product here. So if you need a paging tool, On-Call is going to do exactly what you need.
You can connect to any monitoring system you use, whether that’s Datadog or other monitoring tools. You can manage your on-call schedules and rotations with, you know, customizable escalation policies as you would expect. And then you can page responders via phone, email, SMS, push alerts, etc.
I think what’s cool here as well is our team has been incorporating more AI functionality into our On-Call product, too. It, for instance, can help with things like grouping related pages, and that’s been kind of proved early as a great way to help cut down on some of the alert noise that our customers deal with.
And I guess the last thing I would say here, you know, what’s great about On-Call is that it is often actually cheaper than PagerDuty. So for some organizations, this can be a way to save money.
With our product, we don’t have any add-ons, so you get the full feature set as advertised with On-Call when you decide to switch over.
So that’s what I would say. You know, we’re a very flexible platform, so no matter where you are and what you’re using, there’s usually a solution that we can work out.
Dominique Roller: And we love affordability.
Curtis Maher: That’s true.
Dominique Roller: To the second question: “How is this different from using Claude on our incidents?”
Curtis Maher: That is also a very interesting question. I would say I use Claude all the time. I’m not an engineer by trade; I work with a lot of engineers. I think internally at Datadog, we love Claude.
The big difference I would point to is with Bits Investigation. You know, we’ve done all the hard work for you. Our team has built the whole agent—not just kind of that API, but the harness and everything else under the hood that makes it work.
So you know, if you as a customer are kind of evaluating building this yourself or using our agent, it’s important to note you wouldn’t have to stitch all this together yourself.
Our team has run thousands of evaluations to tune how it goes about investigations, how it behaves, which models it uses at which juncture throughout the investigation, all of that stuff.
And you know, unlike just a model API, Bits Investigations are actually meant to remember your organization, remember past incidents.
It’ll hold onto learnings from past incidents so that when an alert that you’ve seen before shows up, it already knows kind of the playbook that worked, and it can kind of handle it the same way whenever there are similar incidents reoccurring.
But honestly, I think the biggest thing here again, as I kind of alluded to before, is friction. So, you know, with a chat tool, you may be copying and pasting the context in the middle of an incident in these pretty high-stress moments.
Because really, you need to feed a lot of context to allow agents to be useful, especially in complex situations like an engineering incident.
And what’s great is that our Bits Investigation just natively has access to all of that data. So it’s able to be reasoning with that full context, all of the observability telemetry immediately when you need it, instead of once you’ve spent the time to feed it the information that it’s going to rely on to be useful itself. I hope that helps.
Dominique Roller: Definitely, it sounds like a really great resource. It looks like we’re running short on time, but we have a little bit more time for one more question: “Does it plug into Slack or do people have to go into Datadog?”
Curtis Maher: Yeah. No, I actually think we probably briefly touched on this earlier in the presentation. You know, we find that the short answer is yes. We have a great integration with Slack.
We actually find that many of our customers manage incidents directly from these chat platforms like Slack, like Microsoft Teams, like Google Chat.
And so we have a pretty thoroughly built-out integration that does a number of things that I think helps bridge these chat platforms with the incident record that the teams rely on as a source of truth.
This means it can save messages from your channel into the incident record. It means that the key messages in communication, the actions that were taken in Slack, they all show up in one place in Datadog.
And I think, as I alluded to before, this actually is really helpful when you have to generate that postmortem. Because we’re centralizing everything, including the core decision-making that is captured in Slack, you have even greater context, I think, to get an accurate representation of how the incident went.
And I think also just to tie this into our agent, Bits Investigation, we have that agent capability integrated into Slack as well.
So when you run an investigation, you’re going to see its live progress delivered in Slack. You’re going to see the progress surfaced in Slack, including the steps it’s taking, the learnings along the way, and then also the ultimate result.
And that really allows your team to like discuss the findings where they want to discuss the findings, and even ask follow-up questions directly to the agent itself.
And so, the ultimate goal here in how the team has built this out is to make it really feel like the AI agent is part of your team.
It’s going to be doing and showing its work in Slack with your team, and it can actually collaborate there in Slack just as a human engineer would.
So we’re really excited about that. And Slack and, you know, Google Chat, Microsoft Teams—this is an area of continued investment. We’ve definitely heard it loud and clear from customers that they want more deeper, richer integrations there.
Dominique Roller: Amazing. Amazing. Well, with that, we are officially running out of time. So, if we didn’t get a chance to get to your question today, the Datadog team will follow up after the session.
Thank you again, Curtis, for walking us through this measured approach to AI in incident response.
One of the biggest takeaways from today’s conversation is that adopting AI is not just about adding more automation. It’s about understanding where your team is ready, where trust still needs to be built, and how to bring AI into incident workflows in a way that supports better decisions under pressure.
We also heard how leaders can take a more practical path forward, from identifying the right use cases to building confidence over time with a phased adoption roadmap.
All of it was really, really useful information, Curtis. Before we wrap up, I’d love to hand it back to you to share how attendees can continue the conversation and learn more.
Curtis Maher: Sounds good. I just want to thank everybody once again for spending their time listening to me speak. I hope you found the content interesting.
Here on the screen, I have a QR code that just goes to the product page for incident response, but there you can click to start a free trial and you’ll be paired up with a rep who can talk to you about your business, your needs, and how incident response and Bits AI can come and support.
So, if you’re curious, please access this link. And I just want to once again thank you—thank you everybody for your attention. We appreciate it.
Dominique Roller: Thanks so much. So my thanks to everyone that joined us today. Please keep an eye on your inbox for the on-demand recording and we truly hope to see you at future DZone webinars.
Presenters:
Curtis Maher
Product Marketing Manager, Datadog
Join Now for More Content & Events
For event and sponsorship inquiries, please email: [email protected]
