Details
As AI-assisted development increases the speed and volume of software delivery, production environments are becoming more complex than individual engineers can reasonably track. This session explains why incident response must evolve alongside development—and why fully autonomous response is not yet the answer.
Datadog presents a shared-responsibility model across four stages of incident response: investigation, coordination, action, and closing the loop. AI performs machine-scale analysis, correlation, summarization, and first-pass recommendations, while engineers supply business context, validate findings, authorize changes, and remain accountable for the outcome.
Coding agents can complete substantial development tasks and increase delivery speed, but more code also creates more services, integrations, dependencies, and potential failure points.
Incident-response processes must evolve to handle production systems that are changing faster and becoming harder for any one engineer to understand.
AI is well suited to processing large volumes of telemetry, correlating signals across distributed systems, recognizing patterns, and remaining available around the clock.
Engineers remain essential because they understand business impact, can exercise judgment during unfamiliar situations, and accept responsibility for consequential decisions.
The first phase of an incident involves gathering and correlating information from monitors, logs, infrastructure metrics, traces, deployments, dashboards, and runbooks.
Datadog’s Bits Investigation can begin this work when a monitor triggers, test multiple hypotheses, identify a potential root cause, and recommend next steps. Engineers then validate the evidence and determine how to proceed.
Incident coordination is fundamentally a people-centered responsibility. The incident commander brings in the appropriate responders, delegates work, communicates with executives and customers, and manages the response.
AI can support this work by briefing late-arriving responders, maintaining a timeline, summarizing the incident, and capturing information from collaboration channels.
After identifying a likely root cause, AI can suggest a rollback, operational change, or code correction. If a source-code repository is connected, it may also prepare a proposed code fix.
The engineer should review, modify, approve, or reject the recommendation. Production changes carry consequences, so accountability cannot be delegated entirely to an automated system.
AI can assemble objective information such as the incident timeline, affected services, supporting evidence, and actions taken. This removes the burden of beginning with a blank document.
However, engineers should write the conclusions that drive organizational learning, including what the team learned, what should change, and who owns each follow-up action.
Datadog recommends beginning with lower-risk uses such as summarizing signals, timelines, and incident channels. Teams can then expand into autonomous investigations and proposed remediation steps as confidence grows.
Reliable AI-assisted response also depends on clean monitors, accurate ownership information, service tags, current runbooks, relevant dashboards, and mechanically enforced policies.
AI cannot correct a weak observability foundation
Practical implication: Clean up monitors, ownership metadata, service tags, dashboards, and runbooks before introducing AI-led investigations.
Alerts should reflect customer impact
Practical implication: Prioritize high-severity monitors tied to service-level objectives so AI and responders focus on conditions that materially affect users.
Investigation and authorization should remain separate
Practical implication: Allow AI to analyze evidence and prepare a recommendation while requiring an engineer to authorize state-changing actions.
Coordination remains a human responsibility
Practical implication: Keep an incident commander in control of delegation, escalation, stakeholder communication, and customer messaging.
AI is a strong starting point for context work
Practical implication: Introduce AI first for summarization, timeline creation, responder briefings, and evidence collection.
Postmortems should preserve active learning
Practical implication: Automate the factual draft, but require engineers to write lessons learned and assign follow-up work.
AI systems need feedback and evaluation
Practical implication: Give responders a way to flag incorrect investigations, provide better evidence, and evaluate different models against known incident outcomes.
Q: Why is AI becoming more important for on-call teams?
A: AI-assisted development is increasing the amount and complexity of code entering production. AI can help incident-response teams analyze the resulting volume of telemetry and dependencies more quickly than a purely manual process.
Q: What work is AI best suited to during an incident?
A: AI is particularly useful for collecting evidence, correlating signals, testing hypotheses, summarizing activity, maintaining timelines, and preparing possible fixes.
Q: What responsibilities should remain with engineers?
A: Engineers should provide business context, validate AI findings, coordinate responders, weigh trade-offs, authorize production changes, communicate with stakeholders, and own postmortem conclusions.
Q: What is Bits Investigation?
A: Bits Investigation is Datadog’s AI-assisted investigation capability. It can begin investigating when an associated monitor triggers, correlate observability data, evaluate possible causes, produce a conclusion, and suggest next steps.
Q: How quickly can Bits Investigation produce a conclusion?
A: The session states that an investigation conclusion can be produced within approximately three to five minutes of an alert.
Q: Can Bits Investigation recommend a code fix?
A: Yes. When the appropriate source-code repository is connected, it can propose a code change. An engineer should still review and authorize the change before it is applied.
Q: How can teams reduce the likelihood of an incorrect investigation?
A: Teams can improve monitor messages, maintain useful runbooks and dashboards, apply accurate service tags, provide feedback on investigation results, and direct the system toward more relevant traces or logs.
Q: Why should engineers still write parts of the postmortem?
A: Writing lessons learned and follow-up actions requires judgment and reflection. Automating those sections entirely could remove the part of the process that helps teams improve.
Q: Where should a team begin introducing AI into incident response?
The session recommends beginning with context-oriented work such as summarizing signals, timelines, and incident channels before moving into autonomous investigation or remediation recommendations.
Dominique Pugh: Hello, everyone, and welcome to today’s DZone webinar, The Future of On-Call: How Engineers and AI Play to Each Other’s Strengths.
I’m Dominique Pugh, and I’ll be your host for today’s session.
For developers and engineering teams, being on call has always required a difficult combination of speed and judgment. When something breaks in production, teams must work through logs, traces, metrics, dependencies, recent changes, and alerts while trying to understand what matters and restore service quickly.
AI is beginning to change that workflow. It can help surface patterns, connect signals, and take on repetitive investigative work while engineers focus on context, judgment, and the decisions that matter most. That is the balance we’ll explore today.
Before we get started, I have a couple of quick housekeeping notes. Today’s session is being recorded, and we’ll send the on-demand recording to registrants after the event. If you have questions during the presentation, please submit them through the Q&A tab.
Here is a quick look at where we’re headed. Brianne will explore how AI is changing the on-call experience and how Datadog is applying it throughout the incident lifecycle.
From investigation and root-cause analysis to coordination and postmortem work, you’ll gain practical insights into how AI can accelerate investigations across complex systems, where engineers still play a critical role in validating findings and making decisions, and how AI and automation can support the broader incident lifecycle without taking engineers out of the loop.
To walk us through it, I’m pleased to introduce Brianne Bujnowski, Senior Product Marketing Manager for AI at Datadog.
Brianne, thanks for joining us. I’ll hand it over to you.
Brianne Bujnowski: Thank you so much for the introduction. I’m excited about this topic.
Today, I’m going to talk about the future of on-call and what we envision this process looking like as engineering undergoes these changes. We’ll discuss how engineers play to their strengths, how AI plays to its strengths, and how we can reshape this process.
I’m a Senior Product Marketing Manager at Datadog. I spend most of my time talking to engineers about how they use AI for observability, particularly for reliability. Those conversations help me route feedback to the people building these capabilities at Datadog.
Much of my time is spent taking your reliability and AI use cases back to our teams so that Datadog can continue innovating while upholding strong observability practices.
We’ll begin with how AI is shaping development and then discuss what that means for production and the on-call process.
Next, we’ll look at how to divide the work between engineers and AI so that each can play to its strengths. I’ll then walk through the workflow used by Datadog’s SRE team, combining several Datadog products with our recent AI SRE capabilities.
Lastly, this process wasn’t perfect from the beginning. I’ll share some of what we learned while implementing this new on-call process and what the future may look like for Datadog and your teams.
AI is undoubtedly shaping engineering today. Coding agents are writing and shipping more code than ever. AI is no longer limited to autocompleting a line of code. Engineers are handing off full-scale tasks to agents such as Claude Code, Cursor, Copilot, Codex, and others.
SemiAnalysis reported that Claude Code was responsible for approximately 4% of public Git commits and projected that figure could reach 20% by the end of the year. Those are only public commits. Consider how much higher the number could be within privately managed environments.
Pull requests that once took one or two days can now be completed in hours. Velocity and throughput are increasing, but that velocity has a cost to the business.
The increased speed of development can create a substantial reliability burden in production. Google Cloud has observed that while teams are adapting for speed, their underlying systems have not yet evolved to manage AI-accelerated development safely.
Companies that have fully adopted AI in development have reportedly experienced a 29% increase in system instability.
What does more code development through coding agents mean? Unfortunately, many customers are seeing more incidents. More code creates a larger surface area to maintain, which often means more opportunities for things to break.
It also means greater complexity. Agents are building services and integrations that require more coverage. The environment can become too complex for any one person to keep entirely in their head.
Previously, a senior SRE might have understood almost the entire environment. With coding agents accelerating development, it is becoming nearly impossible for one person to know everything. That complexity creates an extreme challenge when an engineer is trying to find the needle in the haystack during an incident.
More incidents, longer incidents, and more complex incidents all fall on the shoulders of the SRE or engineer running the on-call shift.
The downstream effect is tangible. It leads to burnout and confusion. It can also make it difficult for newer SREs to learn how to be effective on call because they cannot contain all the relevant incident and system complexity.
Now that AI is expanding software development, Datadog believes the on-call process must also evolve through AI.
The answer is not simply to hire more engineers. AI in development requires a corresponding use of AI in production.
Many of the hardest on-call tasks are also strong applications for AI. Consider how often an SRE team must sift through thousands or millions of signals when an alert occurs. AI can analyze that information in minutes.
A latency spike, configuration change, and recent deployment may all happen around the same time. AI can correlate those separate signals in a single investigation, whereas an SRE may need 30 or 40 minutes to determine that two events in different parts of a distributed environment are affecting each other.
AI also does not sleep. When you are on call, you are responsible for the reliability of your services around the clock. That can be difficult for a person.
AI can extend around-the-clock coverage so that an incident occurring at 3:00 a.m. is already being investigated before the engineer begins responding.
However, AI alone cannot own reliability. Full autonomy may be a future objective, but we are not close to that point today.
Customers often tell me that they want both autonomy and control. There are several reasons why control remains important.
First, AI often lacks business context. It may not know that another team is in the middle of a launch or that rolling back a deployment would seriously disrupt that work.
Second, AI lacks some forms of judgment. It is very good at pattern recognition, but what happens when an incident is novel and no one has seen it before? How does it know what to do when the circumstances are ambiguous?
Third, there is accountability. AI cannot be the person named on the postmortem. It cannot independently authorize a consequential fix or act as the responsible voice on a customer call.
At Datadog, we believe the future of on-call brings AI and engineers into the same process, with each playing to its strengths.
AI contributes machine-scale analysis, system-wide correlation, and always-on coverage—even at 3:00 a.m.
Engineers provide business context, exercise sound judgment when new issues arise, and remain accountable for the results. If someone has been affected, they should ultimately be speaking with a responsible person rather than a bot.
We’ll discuss several AI capabilities that Datadog’s SRE team uses throughout the incident lifecycle, but I want to highlight one central capability: Datadog’s AI SRE, Bits Investigation.
Bits can be associated with a monitor. When an alert triggers, it runs an investigation across the ecosystem.
It begins by pulling information from the monitor alert, including services, tags, and anything linked in the monitor message, such as a Confluence runbook or relevant dashboard. That context is collected at the time of the alert.
Bits then runs several hypotheses in parallel, reasoning in a way similar to an experienced SRE. It evaluates possible root causes and validates or invalidates those hypotheses until it identifies the underlying issue.
The investigation conclusion can be provided within approximately three to five minutes of the alert. The result can also be sent to Slack or Microsoft Teams so that an on-call engineer responding at 3:00 a.m. can begin with a potential root cause and move toward a fix more quickly.
Now let’s look at four components of an incident. I’ll walk through the workflow from alert to resolution and postmortem, showing how AI and engineers can play to their respective strengths.
We’ll also consider who should lead each stage. Does the stage require substantial human judgment, or does it require the machine-scale analysis that AI can provide?
Investigation
The first stage is investigation.
Datadog’s SREs associate Bits Investigation with SEV-1 and SEV-2 monitors. When an issue occurs, Bits immediately begins investigating.
It analyzes the alert and correlates signals across the environment, including infrastructure metrics, log lines, traces, and related information. It rules out hypotheses until it identifies a likely root cause and produces an investigation conclusion.
Bits can also propose next steps. If it has access to the relevant GitHub repositories, it may propose a code fix. It can also recommend an operational response such as rolling back a deployment.
For investigations, we recommend letting AI take the lead. Allow the AI SRE to identify a possible root cause, and then bring in engineers to exercise judgment about what should happen next.
Engineers add business context. They can also ask Bits questions, challenge its conclusions, investigate further, and validate the findings.
Within each hypothesis, engineers can inspect the associated tool calls and metrics used to reach the conclusion.
During the investigation phase, let AI perform machine-scale analysis across the environment. Then involve the SRE to confirm that the root cause and suggested next steps are appropriate before taking action.
Coordination
The next stage is coordination.
Once the monitor triggers, the team must establish the incident channel and bring the appropriate people together. That requires a person. Coordination is a people-first process.
Introducing AI into the workflow does not eliminate the need for an incident commander. The commander drives the response, brings the appropriate people into the incident channel, delegates work, and owns communication among teams, executives, customers, and status-page audiences.
Although coordination is people-first, Datadog still uses AI to support this part of the workflow.
When someone is added to an incident Slack channel connected to Datadog, the system can provide a summary of what has happened, what the incident involves, and what the suggested next steps are.
If an engineer joins two hours after the incident begins, that person can receive an AI-generated summary and get up to speed quickly without asking the team to stop and explain everything that has already happened.
Datadog also captures a live timeline. Information from the incident—including the Bits investigation, the identified root cause, suggested fixes, actions taken, and relevant activity from Slack or Microsoft Teams—can be captured in one timeline.
AI can also summarize the incident afterward. That summary may be shared in a collaboration channel, routed to executives, or used as a starting point for a status-page update.
Let the engineer lead coordination. Keep it a people-first process focused on bringing in the appropriate participants and ensuring that the right actions are taken. Use AI for summarization and context building.
Action
The third stage is action.
After Bits investigates a monitor alert, it provides an investigation conclusion and suggested next steps. If the relevant GitHub repository is connected, it may also provide a proposed code fix.
The proposed change shows what would be modified from the original pull request, and an engineer can send the approved fix back to GitHub.
The objective is to move from alert to action as quickly as possible. When the investigation finishes in three to five minutes, the team can already have a recommendation for the next step.
Let AI take the first pass at identifying how to correct the issue, but require an engineer to push the pull request, roll back the deployment, or authorize another production change.
Someone must be named on the postmortem. Someone must authorize the fix. Someone must be the responsible voice on the customer call.
Action is important, but an incorrect action can have serious consequences. A person must remain responsible for understanding what happened, approving the response, and learning from the outcome.
Closing the Loop
The final stage is closing the loop.
The team has corrected the issue by rolling back a deployment, applying a code fix, or taking another appropriate action. The last step is to learn from the incident.
We recommend allowing AI to take the first pass, although this is one of the most difficult stages in which to determine whether AI or engineers should lead.
Writing a postmortem is tedious, which makes it tempting to automate. At the same time, engineers must take ownership of the postmortem and ensure that the appropriate follow-up actions are identified.
AI can supply general context, assemble the timeline, and summarize what happened. However, an engineer must understand what occurred and determine how to prevent or handle it more effectively next time.
Let AI create an initial draft so that no one begins with a blank screen, but make engineers responsible for the outcome. Engineers should determine the follow-up tasks and add the judgment that makes a postmortem useful.
At Datadog, only certain parts of a postmortem can be generated through AI. Sections such as “What We Learned” and the follow-up actions must be completed by an engineer from the beginning.
That is the workflow from investigation to coordination, action, and closing the loop. It is how Datadog structures incident response and incorporates AI while ensuring that engineers make the final calls when necessary.
This process will not work perfectly the moment it is introduced. It took time for Datadog to refine its on-call process for the age of AI.
Here are six lessons we learned that may help teams introducing AI into their own workflows.
First, build the foundation before introducing AI.
Before implementing AI in our incident-response workflow, we cleaned up our own Datadog environment. We ensured that monitors had service tags, proper runbooks, and links to the correct dashboards.
This was a hygiene check. Before allowing AI to operate throughout the system, make sure that the data it reads and interprets is reliable. We spent time building that foundation and making the environment understandable to a machine.
Second, anchor alerts to customer impact.
We initially anchored monitors to individual infrastructure and application performance conditions. This created noisy investigations because it was unclear which alerts reflected real customer impact.
We now associate the AI SRE with SEV-1 and SEV-2 monitors tied to service-level objectives. That helps ensure that an engineer receiving a 3:00 a.m. alert is responding because customers are affected—not simply because a noisy monitor continued triggering throughout the night.
Third, enforce policy mechanically.
We tried relying on engineers to associate every monitor with the correct team and service, but this is difficult to maintain consistently in practice.
We now enforce that information through Terraform. When someone creates a monitor, they must add its associated team and service. That keeps the information Bits uses clean and consistent.
Fourth, respect what makes reliability difficult.
This applies to work such as writing postmortems. Some tasks may be tedious, but they also require human judgment and careful thinking. Those portions cannot simply be automated away.
Finding and understanding the correct fix can be difficult, but doing that work creates meaningful improvements throughout the environment.
Fifth, deploy AI for context first.
If you are deciding where to introduce AI into incident response, begin with context: summarizing signals, timelines, and incident channels.
These are tasks AI can generally perform well. Judgment-heavy decisions have more potential for error.
Lastly, earn trust one use case at a time.
We did not immediately introduce AI across every part of the workflow. We began locally with context-oriented uses, then gradually expanded into full-scale investigations and AI-proposed next steps.
Following these principles can help create an on-call process that takes advantage of both AI and engineering strengths.
More than 3,000 customers are using Bits Investigation, and more than 500,000 investigations have been run across customer and Datadog environments.
At Uber Freight, the team reported that Bits reduced mean time to resolution by 70%.
At Arc XP, the team explained that newer engineers may lack the mental model needed to navigate cross-service logs and identify the relevant traces and metrics. With Bits, they can begin with a concise summary of the most relevant observability data instead of an empty search box.
If you’re interested in using Bits Investigation, you can begin with the free trial and explore other AI capabilities available within Datadog.
With that, I’ll hand it back for questions.
Q&A
Dominique Pugh: Let’s begin with what happens when AI gets something wrong. If Bits Investigation provides a confident but incorrect root cause, what mechanism corrects it so that the same mistake is less likely to happen again?
Brianne Bujnowski: That is a very good question.
We are extremely confident in the accuracy of Bits Investigation, but this question reflects the desire for control when using AI, and I agree that control is important.
There are several ways to improve Bits Investigation within your environment.
First, refine the monitor message. Make sure you have clean runbooks, links to useful dashboards, and relevant traces that Bits can investigate when that monitor triggers.
You can also provide feedback. If an investigation is incorrect, you can indicate that through the chat feature. Bits asks whether the investigation was useful, and you can point it toward traces or log lines that were more relevant so that it can improve.
Lastly, make sure the environment is clean before implementing Bits Investigation. Monitors should have the correct service tags and on-call owners so that the right people can be brought into the response with the appropriate context.
Those practices, combined with the product’s overall accuracy, have produced strong results for customers.
Dominique Pugh: There is also a great deal happening beneath these systems that developers may be curious about. Which models are you using, and how do you give them the context needed to understand incident response and an organization’s environment?
Brianne Bujnowski: In terms of AI vendors, we work with Anthropic and OpenAI.
To help Bits Investigation understand incident response, Datadog has built a full-scale evaluation platform. It allows us to learn from previous incidents in Datadog’s own production environment and correct the system when necessary.
If Bits reaches an incorrect conclusion, we have the actual root cause from the incident and can use that information to improve the product.
The evaluation platform does much of the work needed to ensure that root-cause conclusions remain accurate in complex environments, including Datadog’s own environment.
It also allows us to understand which models perform best for different types of investigations. When vendors release new models, we can evaluate whether accuracy increases or decreases for particular incident categories.
We can then use the most effective model for the investigation being conducted in customer and Datadog production environments.
Dominique Pugh: I want to return to the relationship between engineers and AI.
As teams begin using this technology, are there areas where AI is particularly effective at surfacing deep dependencies or cross-domain correlations that might take an engineer much longer to uncover? How are engineers responding to AI’s involvement in the investigation process?
Brianne Bujnowski: There is a concern among some engineers that AI will remove them from the workflow. The central idea of this presentation is that there is space for both.
Many SREs and on-call engineers I speak with are excited about what Bits can provide and what it can remove from their workload.
A shorter incident allows the engineer to return to a lower-stress state sooner. It can also free that person to spend more time improving reliability throughout the environment.
Whether the root cause is completely accurate or not, you can ask Bits what could be improved for the next investigation. You can discuss which hypotheses it should examine next time and how to implement stronger practices within the observability environment.
I encourage people not to treat the result as only a single root-cause output. Examine each hypothesis, review the underlying traces and metrics, and talk with Bits.
Make it part of the process so that it feels more like a teammate during an incident than a system that produces one isolated answer.
The platform can also propose code fixes and help identify lines of code that may need to change. It should be viewed as a partner rather than one isolated component of the workflow.
Bits Investigation has a 14-day free trial. I recommend experimenting with the tool and seeing what it can identify even before you fully refine monitor messages and begin providing feedback.
Explore the possibilities before implementing it across the entire environment. You may be surprised by its accuracy and the amount of time it can save.
Dominique Pugh: Thank you again, Brianne.
One of the most useful lessons for the DZone community is that this is not a story about replacing engineers in the incident-response process. It is about changing where engineering time is spent.
AI can accelerate investigations, surface connections across complex systems, and reduce repetitive incident-response work. Engineers still provide the context needed to understand the impact, weigh trade-offs, and decide what happens next.
To learn more about Datadog’s approach to AI-assisted incident response, explore the resources shared with today’s session and learn how Bits Investigation can support your team’s on-call workflows.
Thanks again to Brianne for joining us, and thank you to everyone who attended today’s DZone webinar. Keep an eye on your inbox for the on-demand recording, and we hope to see you at another DZone webinar soon.
Presenters:
Brianne Bujnowski
Sr. PMM - AI, Datadog
Join Now for More Content & Events
For event and sponsorship inquiries, please email: [email protected]
