AI-Powered CI Test Failure Triage

Last month, a nightly regression came back with 60 failures out of roughly a thousand tests. I spent the first half of the next morning fixing none of them. I sorted them: some were the same failure under different names, some were environment noise, and some were the product actually breaking.

By lunch it was three things. A dependent service had returned errors for about a two-hour window, a data seeding job hadn't finished before the suite started, and one genuine defect. Sixty red rows, three problems.

That sorting is most of my week. It's also the part nobody automates, because it isn't a fixed sequence of steps. It's judgment applied to evidence scattered around a CI dashboard, a log group, a ticket history, and whatever got merged Friday afternoon.

So I built a triage step that does the mechanical sorting for me. Not an agent, not self-healing tests. A post-suite job that clusters failures, pulls surrounding evidence, and hands me a summary I can verify. This is what's in it, what it cost, and the two ways it went wrong before it went right.

What Automation Is Good At, and What It Isn't

Automation answers one question extremely well: did this pass or did it fail? It answers it faster than I can and the same way every time, which is the whole point.

What it doesn't tell me:

• Why did this fail?
• Are these eight failures actually one failure?
• Which of them do I care about before the release call at 11?
• Have we seen this before? (We usually have.)
• What should I look at next?

Every one of those is a reading problem, not an execution problem. The data already exists. It's just scattered around, and reassembling it by hand is slow and boring.

The split I settled on: automation executes, a model reads, and I decide. The third part isn't a formality, and most of the engineering below exists to protect it.

Step One: Cluster Without the Model

My first mistake was handing raw failures to a model and asking what happened. Sixty stack traces is a lot of tokens; the answer took thirty seconds and was vague in a way that's hard to argue with.

The fix was to do the deterministic part deterministically. Most failures in a big suite are duplicates. You can find that out with string normalization and a hash — no inference involved.

# triage/cluster.py
import hashlib
import re
from collections import defaultdict
from xml.etree import ElementTree as ET

# Order matters: strip the most specific patterns first.
NOISE = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
                r"[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"),
    (re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}\S*"), "<TS>"),
    (re.compile(r"0x[0-9a-fA-F]+"), "<ADDR>"),
    (re.compile(r"https?://[^\s'\"]+"), "<URL>"),
    (re.compile(r"'[^']{0,80}'"), "<STR>"),
    (re.compile(r"\b\d+\b"), "<N>"),
]


def signature(message: str) -> str:
    """Collapse a failure message to a stable shape."""
    if not message.strip():
        return ""
    first_line = message.strip().splitlines()[0][:300]
    for pattern, replacement in NOISE:
        first_line = pattern.sub(replacement, first_line)
    return first_line


def cluster(junit_path: str) -> dict:
    groups = defaultdict(list)
    for case in ET.parse(junit_path).getroot().iter("testcase"):
        problem = case.find("failure")
        if problem is None:
            problem = case.find("error")
        if problem is None:
            continue
        shape = signature(problem.get("message", ""))
        key = hashlib.sha1(shape.encode()).hexdigest()[:10]
        groups[key].append({
            "test": f"{case.get('classname')}.{case.get('name')}",
            "shape": shape,
            "time": float(case.get("time", 0)),
        })
    return groups

On that 60-failure run, this collapsed to six clusters. Two of them held 30 and 15 tests. That single number — 60 down to 6 — is most of the value, and no AI was involved in producing it.

It also changes the economics of everything downstream. The model never sees 60 stack traces. It sees six normalized shapes plus a slice of evidence for each shape, which is roughly 4K tokens instead of 60K. Cheaper, faster, and a much smaller surface for the thing to make something up on.

If you build one piece of this, build this piece.

Step Two: Attach Evidence, With Hard Limits

Each cluster then gets enriched from sources that are already there: the log group for the window the suite ran in, the deploys that landed since the last green run, and any tickets whose text matches the normalized shape.

# triage/evidence.py
import boto3

logs = boto3.client("logs")

PLACEHOLDERS = ("<UUID>", "<TS>", "<ADDR>", "<URL>", "<STR>", "<N>")


def log_hits(shape: str, start_ms: int, end_ms: int,
             group: str, limit: int = 5):
    """Pull a bounded sample of matching log lines for one cluster."""
    # The normalized shape has placeholders in it; search on the
    # longest literal run instead.
    stripped = shape
    for token in PLACEHOLDERS:
        stripped = stripped.replace(token, "|")
    literal = max(stripped.split("|"), key=len).strip()

    if len(literal) < 12:
        return []  # too generic to search on; don't guess

    response = logs.filter_log_events(
        logGroupName=group,
        startTime=start_ms,
        endTime=end_ms,
        filterPattern=f'"{literal}"',
        limit=limit,
    )
    return [
        {"ts": e["timestamp"], "stream": e["logStreamName"],
         "message": e["message"][:400]}
        for e in response.get("events", [])
    ]

Two constraints in there are deliberate, and both came from things going wrong.

The limit=5 is because the second version of this pulled every matching line and blew past the context window on a noisy cluster. Five representative lines with timestamps answer "is this the same error, and when did it start" just as well as five hundred.

The len(literal) < 12 guard is because a cluster whose shape normalizes down to something like Expected but was will match half the log group. Returning nothing is correct there. An empty evidence list is a fact, and the prompt below is written so the model has to say so rather than fill the gap.

Step Three: Ask for a Reading, Not a Verdict

The prompt is the part I rewrote the most. What finally worked was forcing every claim to point at a piece of evidence by ID, and constraining the service field to a list of things that actually exist.

SYSTEM = """You are triaging failures from one CI run.

Rules:
- Every claim must cite evidence by its id. No id, no claim.
- If the evidence does not support a cause, set cause to null.
  Do not guess. "Insufficient evidence" is a correct answer.
- suspected_service must be one of KNOWN_SERVICES or null.
- Do not restate the stack trace back to me.

Return only JSON matching this schema:
{"clusters":[{"cluster_id":str,"test_count":int,
  "cause":str|null,"suspected_service":str|null,
  "confidence":"low"|"medium"|"high",
  "evidence_ids":[str],"suggested_next_step":str}]}
"""

Before that enum existed, the model named a service we don't run. It sounded completely reasonable — right naming convention, plausible responsibility, adjacent to the real one. That's the failure mode that actually costs you: not obvious nonsense, but a confident answer that reads as if it came from someone who knows your architecture. An engineer who didn't know the service list would have gone looking for it.

KNOWN_SERVICES gets injected from the same dependency manifest the deployment uses, so it can't drift from reality.

Step Four: Check Its Work Before a Human Sees It

This is the piece I'd argue is non-optional, and it's about forty lines.

# triage/validate.py
def validate(model_output: dict, clusters: dict,
             evidence: dict, window: tuple) -> dict:
    start_ms, end_ms = window
    kept, dropped = [], []

    for item in model_output.get("clusters", []):
        cid = item.get("cluster_id")

        if cid not in clusters:
            dropped.append((cid, "unknown cluster id"))
            continue

        if item.get("test_count") != len(clusters[cid]):
            item["test_count"] = len(clusters[cid])  # we own this number

        cited = set(item.get("evidence_ids", []))
        real = {e["id"] for e in evidence.get(cid, [])}

        if item.get("cause") and not cited:
            dropped.append((cid, "cause with no evidence"))
            continue
        if not cited <= real:
            dropped.append((cid, f"cited missing evidence: {cited - real}"))
            continue

        for e in evidence[cid]:
            if e["id"] in cited and not (start_ms <= e["ts"] <= end_ms):
                dropped.append((cid, "evidence outside run window"))
                break
        else:
            kept.append(item)

    return {"clusters": kept, "dropped": dropped}

Counts come from the parser, never from the model — that's the one class of error I refuse to ship, because a wrong test count makes every other number in the summary suspect. Anything citing evidence that doesn't exist gets dropped rather than shown with a caveat, since a caveat still puts the idea in the reader's head.

The dropped list is not thrown away. It's the closest thing I have to a quality signal on the triage step itself, and I look at it weekly. When one cluster type keeps landing there, the problem is usually my evidence collection, not the model.

What It Actually Produces

Before, for one cluster:

30 tests failed.

After:

cluster 4a1c9f2e — 30 tests

cause: dependent service returned 503 on order lookups

service: order-api       confidence: high

evidence: log lines at 02:14:06, 02:31:44, 03:02:10 (same message,

same stream); no matching lines in the prior night's run

next: check order-api health for the 02:10–03:20 window

The second one I can disprove in about ninety seconds. That's the actual bar. Not "is it right" — "can I check it quickly." Every design decision above serves that.

Numbers

• Triage time on a large regression run: roughly 90 minutes to about 20.
• Clusters per run: 5–8, from 40–70 raw failures.
• Model cost: a few cents per run. Not a factor.
• Wall clock for the triage step: under a minute, running after the suite.
• Claims dropped by the validator: around 1 in 8 in the first month, closer to 1 in 20 once the service enum and evidence limits were in.

That last number is the one I'd watch if you build this. A rising drop rate means something upstream changed.

Where I Stopped, On Purpose

The obvious next step is letting it act. Reopen the ticket, rerun the cluster, patch the test. I did the first two. I won't do the third.

The pitch for self-healing tests goes: the agent notices a failing test, decides the locator is stale, updates it, the test goes green, nobody gets paged. Run that on a test that was failing because the product broke. The agent didn't fix anything. It deleted the evidence, faster than a human would have noticed. The suite is green, the dashboard is calm, the defect ships.

Green isn't the goal. Green with the check still intact is the goal, and nothing I've built or used reliably tells the difference between "this assertion is stale" and "this assertion just caught something."

So I tier by blast radius:

Most of the anxiety about AI in QA comes from treating it as one permission level. It isn't one.

Testing the Thing That Helps You Test

There's a part of this the industry hasn't sorted out, and I'm including it because pretending otherwise would be dishonest.

The output isn't deterministic, so pass/fail assertions don't map onto it cleanly. What I do instead is keep a fixture set of about twenty past runs where I already know the real root cause, and score new prompt or model versions against them on two things: did it name the right cause, and did it cite evidence that actually supports it. It's regression testing with a rubric instead of an assert, and it's the only reason I noticed when a model version change made the summaries longer and vaguer without making them wrong.

That's the other thing to plan for. Behavior shifts underneath you on someone else's release schedule. A prompt-based workflow that's been stable for months can quietly degrade after an upgrade nobody on your team asked for. Pin your model version, and treat a version bump as a change that needs a test run.

And you are feeding logs — possibly source, possibly customer data — into a service. Whatever your org's answer is on that, get it in writing before you build this, not after. It's a security conversation more than a QA one, but it lands on our desk anyway.

The Point

The most valuable thing here isn't generating tests. It's reading everything we already collect and can't get through fast enough.

Speed comes from the model. Repeatability comes from automation. Evidence comes from systems we already run. The call stays with me, mostly because I have to answer for the release.