Evals in CI: How to Catch Agent Regressions Before Production Does

Your deploy pipeline versions everything except the thing that matters most: what the agent actually does.

Code is versioned. Configs are versioned. The prompt is probably in a repo somewhere, hopefully with a review attached. But the agent's behavior, the mapping from inputs to actions, shifts whenever any of its inputs change: a model version bump you did not ask for, a retrieval corpus that gained three thousand documents overnight, a tool schema change from another team, a "small" prompt edit to fix one bad answer in a demo. None of those go through a behavioral regression test. They go straight to production, where your users run the eval suite for you, one incident at a time.

This is the gap evals in CI close. If you ship agents, behavior is your build artifact. Test it like one.

Unit Tests Cover the Shell, Not the Brain

A typical agent codebase has two halves. The deterministic shell: tool wrappers, parsers, retry logic, the policy gate. That half has unit tests, because it is code and code gets tested. The probabilistic core: the planner that decides which tool to call, with what arguments, in what order, and when to stop. That half runs on vibes.

The failure mode is always the same. Someone tweaks a prompt to fix a bad answer in a demo. The fix works. Three weeks later, support notices the agent stopped escalating refund requests above the threshold, or started calling the wrong search endpoint, or doubled its average tool calls per task. Nobody connects the regression to the prompt edit, because nothing measured behavior before and after. The prompt diff looked harmless. Behavior diffs are invisible until they get expensive.

Evals make the behavior diff visible. An eval is a fixed input with a recorded expectation about behavior, run the same way every time, scored the same way every time. A suite of them is a regression test for the part of your system no unit test can reach.

Build the Suite in Three Layers

A useful eval suite is not one big pile of test cases. It is three layers with different jobs.

The golden set. A curated set of representative inputs with labeled expected behavior, versioned in git next to the prompts. Keep it small enough to run on every pull request, a few hundred cases at most. The expectations should be behavioral and checkable: the agent called the right tool with valid arguments, it refused where policy requires refusal, it escalated instead of acting on an irreversible step, the final answer carried the required citation. Deterministic assertions wherever possible. This is your smoke test.

Scenario suites. Multi-turn, tool-using scenarios run against scripted mock tools so they are repeatable. Real agents fail in the middle of plans, not at the start: the third tool call returns an error, the retrieved document contradicts the user's request, the plan needs replanning halfway through. Scenario evals exercise the planner across a full trajectory and assert on the trajectory, not just the final answer. Did it recover from the tool error or loop forever? Did it notice the contradiction or bulldoze through it? These run nightly or on significant changes, because they are slower and occasionally flaky.

The adversarial set. Inputs designed to break the agent: injection attempts smuggled through tool outputs, permission-boundary probes ("just run it as admin"), conflicting instructions from untrusted documents, edge cases in tool arguments. This set encodes your threat model as executable tests. Every incident and every red-team finding becomes a new case here. If your layered defenses matter, and they do, this is where you prove they still hold after each change instead of just asserting it.

Score Without Lying to Yourself

The fastest way to waste an eval program is a single aggregate score. An average that climbed from 87 to 89 hides the three scenarios that collapsed from 100 to 40. Report per-scenario deltas against the baseline, and gate on "nothing regressed beyond tolerance," not on "the average went up."

Scoring itself needs the same rigor as the suite:

Deterministic checks first. Most agent behavior that matters is checkable without a model: right tool, valid arguments, correct refusal, correct escalation, required fields present, disallowed actions absent. Write these as code. They are fast, stable, and auditable.

Model-as-judge for the fuzzy remainder. Some things only a reader can score: answer quality, tone, whether the explanation actually addressed the question. Use a judge model with a written rubric, and calibrate it: have humans label a sample, measure judge-human agreement, and re-calibrate when you change the judge. A judge is also a model. It drifts, it has biases, it favors verbosity. Pin its version like any other dependency, and track its disagreement rate as a metric of the eval system itself.

Run the nondeterministic parts more than once. Agents are stochastic. A scenario that passes once and fails twice is telling you something different from one that fails three times. Run each scenario a few times, report the distribution, and quarantine flaky scenarios into a separate bucket instead of letting them veto every merge or, worse, getting deleted because they were annoying.

Wire It Into the Merge

Evals only matter if they can stop a bad change. The wiring looks like this:

def eval_gate(change, baseline, tolerance):
    report = run_eval_suite(change, GOLDEN_SET + ADVERSARIAL_SET)
    regressions = [s for s in report.scenarios
                   if s.score < baseline[s.id] - tolerance]
    if regressions:
        return fail(f"{len(regressions)} scenarios regressed", regressions)
    return pass_with_report(report)

Four rules make the gate real:

1. Every behavior-changing diff triggers the golden set. Prompts, tool definitions, model version pins, retrieval configuration, policy rules. Not just code changes. Prompts are code now; treat the diff that way.
2. Compare against the baseline for the base commit. A scenario that regressed beyond tolerance fails the build. The report names the scenarios, shows before and after, and links the diff.
3. Tier the slow suites. Scenario suites run nightly and on release branches. The adversarial set runs on any change touching trust boundaries: new tools, new data sources, permission changes. If the per-PR eval takes an hour, engineers will route around it. A tiered suite that actually runs beats a comprehensive suite everyone skips.
4. Baselines update deliberately. When a change intentionally alters behavior, the new baseline gets reviewed like the code change itself, with the per-scenario diffs as the review material. Silent baseline updates are how eval suites become decoration.

Measure What Agents Actually Break

Generic quality scores miss the failure modes that are specific to agents. Your eval dimensions should include:

Tool-call correctness. Right tool, valid arguments, sensible order. An agent that answers correctly after calling five wrong tools first did not behave correctly. It got lucky.

Policy compliance. Refusals and escalations where required. This is where runtime guardrails and evals meet: the policy gate enforces at runtime, the eval suite proves the planner respects the policy across scenarios instead of routing around it.

Injection resistance. The adversarial set's core metric. After every change, the agent should still refuse or neutralize injected instructions at the same rate. This is a regression metric, not a one-time audit.

Trajectory efficiency. Tool calls per task, tokens per task, wall-clock time. An agent that takes forty tool calls to do a three-call job is a regression even when the answer is right. Cost and latency are behavior.

Recovery behavior. What it does when a tool errors, returns garbage, or times out: retry with backoff, replan, escalate, or spiral. Script these in the scenario suite. Production will test them otherwise.

The Failure Modes of Eval Programs

Evals rot like any other system. Expect it and design for it.

Overfitting to the golden set. Prompts get tuned until the evals pass while wild behavior stays broken. The defense is a held-out set nobody tunes against, plus sampling real production traffic into new eval cases. If your golden set never grows, your agent is being trained to pass a test.

Judge drift. The judge model gets updated, its verdicts shift, and suddenly half the suite "regresses." Pin the judge, version the rubric, and investigate judge-side changes before agent-side ones.

Stale scenarios. New tools ship, new features launch, and the eval suite still tests last quarter's agent. Review the suite on a cadence. Every production incident should produce a new eval case. That is the cheapest way to keep the suite honest.

Eval theater. The suite runs, the dashboard is green, and nobody reads the per-scenario diffs before merging. Green without review is a ritual. Require the diff in the PR the way you require the code diff.

Close the Loop With Production

The eval suite is half of a loop. The other half is production telemetry feeding back into it. Sample live trajectories, label the failures, and promote them into the golden or adversarial sets. The incidents your users found become the regressions your CI catches next time. Over months, the suite becomes a record of every failure mode the system has ever learned, which is exactly what a regression suite is supposed to be.

This is the part teams skip, and it is the part that compounds. An eval suite containing only cases someone imagined on day one tests the agent you used to have.

The Takeaway

Agents are software with probabilistic planners and deterministic effects. Ship them like software: version the behavior, test it on every change, gate merges on the results, and feed production failures back into the suite. A prompt edit that cannot pass the same bar as a code change does not merge.

Your users are already running your eval suite. The only question is whether you run it first.

Views expressed here are my own and do not represent my employer.