Details
Deploying an AI SRE is only the beginning. To improve over time, an agent must learn from incident outcomes, incorporate feedback from human SREs, and retain contextual knowledge about services, dependencies, deployments, and previous investigations.
In this third video of Elastic’s three-part series, Frank Swain explains how human feedback, auditable decision trees, and contextual knowledge graphs can make AI-led investigations more effective. He also identifies an important operational risk: as agents learn, their production behavior may drift from what teams originally tested. The agents therefore need to be monitored and audited as closely as the systems they investigate.
An AI SRE should capture what happened during an incident, how the system failed, which remediation steps were attempted, and which action ultimately succeeded.
Reusing this information gives future investigations a stronger starting point and prevents teams from repeatedly solving the same problem from scratch.
After an incident, human SREs can evaluate the investigation, correct faulty conclusions, and provide information the agent may have missed.
This creates a feedback layer through which incident outcomes influence future behavior. Over time, the agent’s decision-making can evolve toward more reliable results.
Decision trees provide an ordered process for evaluating possible causes. When an alert occurs, an agent might check for a recent deployment, dependency failure, or resource saturation before moving to another hypothesis.
Because the agent eliminates possibilities sequentially, the resulting investigation is easier to understand and review.
A decision tree records the branches an agent followed and the possibilities it eliminated. Engineers can review that path after an incident instead of receiving only a final answer.
This makes the agent’s reasoning more legible and aligns its workflow more closely with the investigative process used by experienced engineers.
An individual alert rarely explains why a distributed system failed. The agent also needs to understand the relationships among services, dependencies, previous incidents, runbooks, deployments, metrics, logs, and traces.
A knowledge graph structures those relationships and gives the AI SRE persistent context it can carry into future investigations.
Telemetry may show that a service failed, but context can explain why it failed under the specific conditions surrounding the incident.
By connecting a deployment, dependency, resource constraint, and service relationship, the agent can produce a more useful explanation than it could from an isolated signal.
An agent that learns will also change. Its behavior in production can eventually differ from the behavior originally tested in staging.
Teams must monitor agent decisions, audit investigation paths, evaluate outcomes, and detect behavioral drift as part of operating an AI SRE safely.
Feedback should be part of incident closure
Practical implication: Add an explicit step for engineers to rate the investigation, correct errors, and record the remediation that worked.
Incident knowledge must be reusable
Practical implication: Store investigation outcomes in a form that agents can retrieve during similar future incidents.
Reasoning should be inspectable
Practical implication: Record the hypotheses evaluated, evidence used, branches followed, and causes eliminated during each investigation.
Knowledge graphs should represent relationships
Practical implication: Connect services, owners, dependencies, deployments, runbooks, telemetry, and previous incidents rather than storing them as isolated records.
Improvement can introduce behavioral drift
Practical implication: Re-evaluate agents continuously instead of assuming that a model or workflow that passed staging tests will behave identically after learning from production outcomes.
The AI SRE itself needs observability
Practical implication: Track the agent’s tool calls, recommendations, confidence, investigation paths, outcomes, and changes in behavior over time.
Q: Why does an AI SRE need continuous improvement?
A: Production environments, dependencies, and failure patterns change. An agent that does not incorporate new incident knowledge may continue using outdated assumptions or repeating ineffective investigations.
Q: What is continuous knowledge capture?
A: Continuous knowledge capture records what happened during an investigation, why the system failed, and which remediation steps succeeded so the information can be reused later.
Q: How does human feedback improve an AI SRE?
A: Human SREs can confirm useful conclusions, correct mistakes, supply missing context, and identify the response that worked. This feedback can guide future decisions.
Q: How do decision trees support incident investigations?
A: Decision trees give agents an ordered set of questions and branches to follow. The agent can eliminate possible causes one at a time and preserve a record of its reasoning.
Q: Why are decision trees useful for auditing?
A: They show which hypotheses the agent considered, what evidence influenced each branch, and how it reached its conclusion. Reviewers can inspect the process rather than relying only on the final answer.
Q: What information belongs in an observability knowledge graph?
A: Relevant information can include services, dependencies, deployments, previous incidents, runbooks, metrics, logs, traces, remediation actions, and relationships among those elements.
Q: How is a knowledge graph different from storing telemetry?
A: Telemetry records signals. A knowledge graph represents how services, events, changes, and operational resources relate to one another, giving the agent additional context for reasoning.
Q: What is behavioral drift in an AI SRE?
A: Behavioral drift occurs when an agent’s production behavior changes over time and begins to differ from what teams previously tested or expected.
Q: How should teams monitor an AI SRE?
A: Teams should audit investigation paths, recommendations, tool use, incident outcomes, human corrections, and changes in decision-making over time.
Frank Swain: Hi, I’m Frank Swain, Principal Product Marketing Manager at Elastic.
This is the third video in a three-part series we’ve put together for you. Today, we’re going to discuss how to improve AI SREs and achieve machine-speed observability.
In the previous videos, we discussed how to set up AI SREs. Getting them running, however, is only the first step in the journey.
Once you have an AI SRE operating, you need to help it evolve and improve continuously.
One way to accomplish this is through continuous knowledge capture. Every investigation becomes a learning opportunity, capturing and reusing knowledge about how systems failed in the past and how teams successfully remediated those failures.
The key to this process is a human feedback layer. After every incident, human SREs have an opportunity to provide feedback to the agentic SRE. Over time, this allows its decision-making to evolve.
One way to improve an AI SRE’s performance is through decision trees.
A decision tree gives the agent a straightforward, sequential process to follow. When an alert fires, the agentic SRE can ask whether there was a recent deployment, a dependency failure, or resource saturation. It can follow those branches to reach the root cause.
Decision trees provide several benefits.
First, the process is sequential. It doesn’t happen all at once or entirely in parallel. The agent eliminates possible issues before moving to the next step.
This provides a consistent, auditable path. Every branch the agent follows is recorded, and the investigation can be reviewed afterward.
It also mirrors how experts reason. The decision tree reflects how an engineer might approach the problem, keeping the agent’s logic understandable to reviewers.
Another important component is the knowledge graph.
A knowledge graph helps build contextual memory for the AI SRE by giving it access to resources such as previous investigations, runbooks containing possible actions, metrics, logs, traces, and deployment history.
The deployment history helps the agent determine whether a recent change may have triggered an incident.
The important point is that this process is about relationships, not only individual signals. The graph captures how components actually relate to one another.
This gives the AI SRE persistent context. It can understand how a change in one area affected other parts of the environment.
The knowledge graph also helps explain why something happened—not only what happened.
Context allows the agent to explain not simply that a system failed, but why it failed given the surrounding conditions in the environment.
What we’ve learned is that agents stagnate when organizations treat them like tools that can be installed and then ignored.
The agents that improve are treated more like engineers. They learn from every investigation, incorporate feedback, and carry context into the next incident.
Human feedback turns outcomes into better decisions. Decision trees keep those decisions systematic and auditable. Knowledge graphs give the agent the memory required to understand not only that something broke, but why it broke in that particular environment.
None of this can operate in the dark.
An agent that learns is also an agent that changes. Its behavior in production can drift from what the team originally tested in staging.
The final component holds everything together: the ability to monitor and audit the AI SRE as closely as it monitors your systems.
For the complete picture of how to integrate and improve an AI SRE, download our guide, Transforming Observability With Agentic AI and the Autonomous AI SRE. There is a QR code on the screen and a link in the description.
Thanks again for watching. We look forward to seeing you next time.
Presenters:
Frank Swain
Principal Competitive Intelligence Manager, Elastic
Join Now for More Content & Events
For event and sponsorship inquiries, please email: [email protected]
