AI in SRE: A Practical Autonomy Model for Self-Healing Infrastructure

Most SRE teams do not need another dashboard.

They need a safer way to move from "something is wrong" to "we know what to do next." A model that detects anomalies is useful. A model that can touch production can also make a bad incident worse.

That is where most conversations about AI in SRE become too optimistic for my taste. The hard part is not only detection. It is deciding how much autonomy the system should have, under which conditions, and with what blast-radius controls.

I learned this while working on large-scale cloud services where one customer-facing symptom could turn into a flood of alerts. A degraded dependency might show up as latency in one service, retries in another, queue growth somewhere else, and CPU pressure downstream. During an on-call shift, that can look like five separate problems. Usually, it is one problem echoing through the stack.

That experience changed how I think about self-healing infrastructure. The goal is not to build a system that blindly fixes everything. The goal is to build an operational control loop that can separate routine, low-risk recovery from incidents that still need human judgment.

The model that has worked best for me is graduated autonomy: Let the system act automatically only when the action is well understood, reversible, and narrow in blast radius. For everything else, the system should collect evidence, recommend the next step, and keep humans in control.

Why Static Alerts Stop Scaling

Static alerts are not the enemy. I still want to know when disk usage is dangerous, error rates spike, or latency crosses a service-level threshold. But thresholds do not understand context.

A CPU spike during a scheduled batch job may be normal. The same spike during steady-state traffic may be a retry storm. A latency increase in one region may be harmless during a controlled deployment, but suspicious if it appears across multiple availability zones with no recent change event.

At small scale, engineers can carry that context in their heads. At enterprise scale, they cannot. Services emit hundreds of metrics across regions, dependencies, deployments, and customer paths. Eventually the team is no longer tuning alerts. It is negotiating with noise.

In one rollout I was involved with, the most useful improvement was not adding more alerts. It was grouping alerts around dependency context and suppressing repeated downstream symptoms. The on-call experience became calmer because engineers could focus on the likely failure path instead of chasing every red graph independently.

That is the kind of problem AI can help with. Not by replacing SRE judgment, but by organizing noisy signals into a more useful operational story.

Detection Is Only the First Layer

ML-based anomaly detection helps because it learns a service's normal operating shape instead of relying only on fixed thresholds. For cloud metrics, that usually means learning seasonality, traffic cycles, deployment windows, regional differences, and service-specific behavior.

An LSTM autoencoder, isolation forest, or well-tuned statistical baseline can all be useful. I care less about the model family than the quality of the telemetry around it. A simple model trained on clean, consistent data will usually beat a sophisticated model trained on messy metrics.

A practical anomaly pipeline usually looks like this:

1. Collect metrics, logs, traces, and change events.
2. Normalize them by service, region, dependency, and time window.
3. Score each signal against its learned baseline.
4. Group anomalies by dependency graph and recent changes.
5. Produce an evidence bundle for automation or human review.

Here is a simplified version of the scoring stage:

from dataclasses import dataclass
from typing import List

@dataclass
class MetricWindow:
    service: str
    region: str
    signal: str
    values: List[float]
    recent_deploy: bool = False

@dataclass
class AnomalyScore:
    service: str
    region: str
    signal: str
    score: float
    reason: str

class BaselineModel:
    def expected_range(self, service: str, region: str, signal: str):
        # In production, this may come from a trained model,
        # feature store, or rolling baseline per service and region.
        return (0.0, 1.0)

def score_window(window: MetricWindow, baseline: BaselineModel) -> AnomalyScore:
    low, high = baseline.expected_range(
        window.service,
        window.region,
        window.signal,
    )
    latest = window.values[-1]

    if latest > high:
        distance = (latest - high) / max(high, 0.001)
        reason = f"{window.signal} above learned baseline"
    elif latest < low:
        distance = (low - latest) / max(abs(low), 0.001)
        reason = f"{window.signal} below learned baseline"
    else:
        distance = 0.0
        reason = "within learned baseline"

    if window.recent_deploy and distance > 0:
        reason += " during recent deployment window"

    return AnomalyScore(
        service=window.service,
        region=window.region,
        signal=window.signal,
        score=min(distance, 1.0),
        reason=reason,
    )

The production value is not just the score. It is the metadata around it: ownership, dependency path, recent deploys, feature flag changes, customer impact, and whether the same pattern has appeared before.

A single anomalous metric should rarely trigger remediation. Sustained anomalies across correlated signals are more trustworthy than one spike in one chart.

Correlation Turns Noise Into an Incident Story

During an incident, the useful question is not "Which graph is red?" It is "What changed first, and what depends on it?"

That is where dependency-aware correlation becomes more useful than raw anomaly detection. A database issue may surface as API latency, retries, queue saturation, and CPU pressure. Without a dependency graph, every downstream service looks guilty. With one, the system can rank likely causes instead of handing the engineer a wall of symptoms.

A useful correlation engine should look at topology, timing, change context, and customer impact. Which dependency failed first? Was there a deployment or config change? Which service is closest to the customer-facing error?

The evidence bundle should be readable by a human. If the model says "root cause confidence: 0.86," that is not enough. It should also explain why.

{
  "candidate_root_cause": "identity-token-cache",
  "region": "example-region-1",
  "confidence": 0.86,
  "customer_impact": "elevated authentication latency for a subset of requests",
  "supporting_signals": [
    "p99 latency above learned baseline for multiple consecutive windows",
    "cache hit rate dropped below its recent operating range",
    "downstream services showed retry growth after the initial cache anomaly",
    "no database saturation was observed",
    "no deployment was detected in the immediate incident window"
  ],
  "recommended_action": "drain_and_restart_one_cache_node",
  "estimated_blast_radius": "single node in a redundant pool",
  "rollback_plan": "keep node out of rotation if health checks fail after restart"
}

This is more useful than another alert. It gives the on-call engineer a starting hypothesis and the reasoning behind it.

The Graduated Autonomy Model

The most important design decision in self-healing infrastructure is not which ML algorithm to use. It is which actions the system is allowed to take.

I divide remediation into three tiers.

Tier 1: Fully Automated, Low-Risk Actions

Tier 1 actions are safe, reversible, and narrow in blast radius. These are actions the system can execute without waiting for a human when confidence is high.

Examples include restarting one unhealthy instance, scaling out a stateless service, draining one bad node, flushing a bounded cache, or shifting a small amount of traffic away from a degraded zone.

The key phrase is bounded blast radius. Auto-remediation should not restart half the fleet, fail over a primary database, or disable a feature globally just because a model is confident. Confidence is not a substitute for safety.

Before I put an action in Tier 1, I expect it to pass these checks: it is reversible, affected capacity is small, redundancy is healthy, there is no active global incident, the same action has not failed recently, rollback is defined, and health checks can verify success quickly.

The first Tier 1 actions should be boring. Restarting one unhealthy node is not exciting, but it is exactly the kind of action that can be automated safely when the system has enough evidence.

Tier 2: Automated Recommendation With Human Approval

Tier 2 is where many real incidents live. The system may know what should happen, but the action still needs human approval.

Examples include rolling back a deployment, disabling a feature flag, failing over a database, increasing capacity beyond a normal band, or changing regional routing.

For Tier 2, the system should prepare the action, show the evidence, and ask for approval. The human should decide whether the action makes sense, not build the command during the incident.

One pattern I have seen repeatedly: the slowest part of remediation is not always finding a likely cause. It is gathering enough confidence to take a risky action. When the system attaches deploy timing, error movement, affected endpoints, config changes, and rollback commands into one review card, the decision becomes easier.

Tier 3: Human-Led With AI Context

Tier 3 incidents are novel, high-risk, or ambiguous. The system should not execute remediation. It should help humans reason.

This includes possible data corruption, multi-region cascading failures, security-sensitive incidents, conflicting signals across dependencies, low-confidence root-cause analysis, or any action with unclear rollback behavior.

In Tier 3, the system's job is to summarize what it knows, what changed recently, which hypotheses are most likely, and which dashboards or runbooks are relevant. That alone can save time, but it keeps production control where it belongs.

Architecture: A Control Loop, Not a Magic Button

A practical self-healing system looks like a control loop with guardrails.

The important part of this diagram is the policy gate. Detection and correlation produce a recommendation, but the policy gate decides autonomy. Without that layer, "self-healing" becomes a risky automation script with an ML label attached.

The policy gate should evaluate confidence, risk, blast radius, recent action history, service criticality, and rollback readiness. I would express that as policy-driven code:

from dataclasses import dataclass
from enum import Enum
from typing import List

class Decision(str, Enum):
    AUTO_EXECUTE = "auto_execute"
    REQUEST_APPROVAL = "request_approval"
    HUMAN_LED = "human_led"

@dataclass
class RemediationProposal:
    action: str
    confidence: float
    blast_radius_percent: float
    reversible: bool
    rollback_defined: bool
    service_tier: str
    evidence: List[str]

@dataclass
class RuntimeContext:
    active_global_incident: bool
    recent_failed_action: bool
    healthy_redundancy: bool
    minutes_since_last_same_action: int

TIER_1_ACTIONS = {
    "restart_single_instance",
    "scale_stateless_service",
    "drain_single_node",
    "flush_bounded_cache"
}

TIER_2_ACTIONS = {
    "rollback_deployment",
    "disable_feature_flag",
    "database_failover",
    "regional_traffic_shift"
}

def decide_autonomy(
    proposal: RemediationProposal,
    context: RuntimeContext
) -> Decision:
    if context.active_global_incident:
        return Decision.HUMAN_LED

    if context.recent_failed_action:
        return Decision.HUMAN_LED

    if not proposal.rollback_defined:
        return Decision.HUMAN_LED

    if proposal.action in TIER_1_ACTIONS:
        safe_enough = all([
            proposal.confidence >= 0.90,
            proposal.blast_radius_percent <= 5.0,
            proposal.reversible,
            context.healthy_redundancy,
            context.minutes_since_last_same_action >= 30,
            len(proposal.evidence) >= 3,
        ])
        return Decision.AUTO_EXECUTE if safe_enough else Decision.REQUEST_APPROVAL

    if proposal.action in TIER_2_ACTIONS and proposal.confidence >= 0.75:
        return Decision.REQUEST_APPROVAL

    return Decision.HUMAN_LED

This is not drop-in production code, but the structure is the point: actions are classified, confidence is not the only input, and safety can override the model.

In reliable systems, the model proposes; policy disposes.

What I Measure Before Expanding Autonomy

I would not start by asking, "Can we automate remediation?" I would start by asking whether the system's recommendations are trustworthy.

Before allowing Tier 1 execution, I would track root-cause precision, false positives by service, recommendation acceptance, time to useful diagnosis, remediation success, rollback frequency, and any secondary incidents caused by remediation.

The last two matter the most to me. A self-healing system that fixes one issue but creates another is not healing. It is moving the incident.

My preference is to run in shadow mode first. Let the system detect, correlate, and recommend, but do not let it execute. Compare its recommendations against what engineers actually did. Once the system repeatedly recommends the same low-risk actions humans already take, graduate those actions into Tier 1.

That is how trust gets built: not through a big launch, but through repeated correctness in narrow, well-understood situations.

Lessons Learned From Building Toward Self-Healing

The most useful lessons are not about model architecture.

Clean telemetry beats clever models. If service names are inconsistent, regions are missing, logs are unstructured, and ownership metadata is stale, the model will struggle. Before debating LSTMs versus transformers, fix the telemetry pipeline.

Change events are first-class signals. Deployments, config pushes, schema changes, and feature flag flips explain many anomalies. If the model cannot see change events, it will treat every incident like a mystery.

Alert suppression is not the same as diagnosis. Reducing noise is useful, but the system must preserve the causal path. Suppressing duplicate downstream alerts only helps if the upstream root cause remains visible.

Automation needs a memory. Every remediation should leave an audit trail: what was detected, what action was taken, what happened afterward, whether rollback was needed, and whether humans agreed with the recommendation.

Start with boring actions. Restarting one bad instance is not glamorous. Draining one node is not a research breakthrough. But these are exactly the kinds of actions that make sense for early autonomy because they are repeatable, reversible, and easy to verify.

Where LLMs Fit

Large language models are useful in SRE, but I would not put them directly in the execution path for remediation. Their best role is communication and context assembly.

An LLM can draft an incident summary, explain the evidence bundle, turn raw telemetry into a timeline, identify runbooks, and prepare a post-incident report. That saves time without giving the model direct control over production.

The safer pattern is separation of responsibilities: ML or statistical models detect anomalies, graph correlation ranks likely causes, policy gates decide autonomy, deterministic automation executes approved actions, and LLMs summarize what happened.

That separation keeps the high-risk parts deterministic and auditable while still using AI where it helps most.

Final Thought

Self-healing infrastructure is not about removing SREs from production. It is about removing the repetitive, low-risk work that slows them down during incidents.

The best version of AI in SRE is not a magic system that fixes everything. It is a careful control loop: detect early, correlate intelligently, act only within policy, and learn from every outcome.

If you are building toward self-healing, do not start with full autonomy. Start with evidence. Then recommendations. Then approval-based actions. Then, only after the system has earned trust, allow narrow automated remediation.

That path is slower than the hype cycle, but it is much closer to how reliable infrastructure actually gets built.