Production-Grade LLM Regression Testing: Contracts, Evals, and Quality Gates

Conventional unit tests remain essential around an LLM application, but they do not answer the most important production question about whether a change preserved useful behavior. A parser can still return valid JSON while the answer becomes less grounded. A tool call can satisfy its schema while selecting the wrong tool. A retrieval pipeline can return documents successfully while omitting the evidence required for the final answer. This gap exists because an LLM application is not a deterministic function whose correctness can always be represented as actual == expected. Empirical work on code generation has shown substantial output variation across repeated calls, including at temperature zero, so single-run assertions can mistake sampling noise for either success or failure. Regression testing therefore has to evaluate behavior statistically and semantically, not only execution paths. 

Unit Tests Prove Contracts, Not Behavior

The first layer should still look familiar. Deterministic properties deserve deterministic tests like JSON must parse, required fields must exist, tool arguments must conform to a schema, forbidden operations must remain blocked, and latency or token budgets can be checked numerically. Anthropic’s evaluation guidance explicitly separates success criteria from the mechanisms used to measure them and includes exact-match, code-based, human, and model-based grading as different options rather than treating one metric as universal. 

A contract test for a support-answering endpoint can remain intentionally narrow:

def validate_contract(result):
    assert result["status"] in {"answered", "abstained"}
    assert isinstance(result["answer"], str)
    assert len(result["answer"]) <= 2_000
    assert all(c["source_id"] for c in result["citations"])

Passing this function proves that the response is consumable by downstream software. It does not prove that the answer is correct, that every material claim is supported, or that an abstention occurred when evidence was missing. Those are behavioral properties and need evaluators matched to the application’s actual failure modes. Research on LLM-based judging also shows why a single generic "quality" score is insufficient as judges can exhibit position, verbosity, and self-enhancement biases, even though strong judges can correlate well with human preferences under controlled evaluation. 

Build the Regression Corpus From Failures

A useful regression suite begins with representative tasks rather than prompts invented solely for testing. Production traces, manually verified examples, support escalations, retrieval misses, tool selection failures, malformed outputs, and previously fixed incidents should become durable evaluation cases. LangSmith’s current evaluation documentation describes this feedback pattern where production traces can be added to datasets so that a failure observed in live traffic becomes a repeatable offline test, while experiments compare application versions on the same dataset. 

Each case should preserve enough context to reproduce the behavior that matters. A question alone is often insufficient for RAG or agent systems because the retrieved documents, tool state, permissions, conversation history, and expected policy can affect the result. The expected value also should not always be a canonical sentence. A stronger case stores assertions about required facts, forbidden claims, acceptable citations, expected tool choices, and whether abstention is mandatory.

case = {
    "input": "Can an expired license be renewed online?",
    "required_facts": {"renewal_window": "30 days"},
    "forbidden_claims": {"automatic_extension"},
    "must_cite": {"policy-17"},
    "expected_action": "answer"
}

The corpus also needs slices. A global average can improve while an important category regresses. Long queries, multilingual requests, ambiguous requests, high-risk tool actions, low-retrieval-confidence cases, and specific product domains should retain explicit tags so candidate performance can be compared within those populations. Google’s production ML guidance similarly recommends monitoring real-world metrics and inspecting data slices because aggregate quality can conceal skew or deterioration in subgroups. 

Score Properties Instead of Strings

Exact matching works for classifications, IDs, tool names, and other canonical outputs. Open-ended language requires property-based evaluation. For RAG, retrieval and generation should be measured separately. Ragas formalized this decomposition with metrics targeting retrieval quality, answer relevance, and faithfulness rather than collapsing the entire pipeline into one score. That separation matters operationally because an unsupported answer and a retrieval miss require different fixes. 

A production-like evaluator can combine deterministic checks with semantic scoring:

def evaluate(case, result):
    return {
        "contract": contract_score(result),
        "groundedness": groundedness_score(
            result["answer"], result["evidence"]
        ),
        "task_quality": rubric_score(
            case["input"], result["answer"], case
        ),
        "tool_correctness": tool_score(case, result),
    }

The evaluator should expose dimensions rather than immediately averaging them. A groundedness score of zero must not be hidden by excellent style. A wrong financial action must not pass because the explanation is fluent. Hard safety and contract requirements should act as vetoes, and softer qualities such as completeness or tone can be aggregated only after those constraints pass.

LLM as a judge is useful when the desired property cannot be expressed with code, but the judge itself becomes part of the test infrastructure. G-Eval demonstrated that rubric-driven LLM evaluation can align better with human judgments than older reference-based metrics for some generation tasks, while later judge research documented systematic biases and sensitivity to evaluation setup.  The practical consequence is straightforward with the judge model, rubric, prompt, sampling settings, and parser should be versioned like any other dependency. Borderline cases should be sampled repeatedly or routed to human review rather than converted into false precision by a single score.

Gate Changes Against a Baseline, Then Close the Production Loop

Regression testing is most informative when a candidate is compared with a pinned baseline on identical cases. Absolute thresholds such as "quality must exceed 0.85" can hide a meaningful drop from 0.94 to 0.86. A gate should therefore combine non-negotiable case failures with relative movement in quality, cost, and latency.

def release_allowed(baseline, candidate):
    if candidate["critical_failures"] > 0:
        return False

    quality_delta = candidate["quality"] - baseline["quality"]
    latency_ratio = candidate["p95_ms"] / baseline["p95_ms"]

    return quality_delta >= -0.01 and latency_ratio <= 1.10

The tolerances in this example are application policy, not universal constants. More important is the comparison model where the baseline and candidate run against the same versioned corpus, results remain inspectable by case and slice, and repeated samples are used when output variance is material. Recent research on judge reproducibility reinforces this requirement, finding that temperature control can reduce variability without eliminating it and arguing that grader disagreement should be treated as an evaluation signal rather than ignored. 

CI can then apply evaluation at different depths. Pull requests can execute a compact, high-signal corpus covering critical behaviors, while scheduled or pre-release runs can execute broader datasets and repeated samples. Official LangSmith guidance supports offline evaluation for comparing versions before deployment and provides CI/CD integration patterns around evaluation runs.  The important design principle is not the specific platform, as evaluation results must be release evidence rather than a dashboard consulted only after a failure.

Offline evaluation still cannot fully represent production traffic. Query distributions change, documents change, tools return new states, and upstream services evolve. Production monitoring therefore completes the regression loop. Google’s production ML guidance emphasizes live quality measurement, drift detection, and ongoing monitoring rather than treating validation as a one-time pre-deployment event.  Sampling real traces, reviewing low-confidence or high-impact outcomes, and promoting confirmed failures back into the regression corpus turns operational incidents into permanent test coverage.

AI regression testing becomes reliable when deterministic software tests and behavioral evaluations are treated as complementary rather than interchangeable. Unit tests should enforce contracts and invariants, curated datasets should preserve real failure modes, evaluators should measure task-specific properties, baseline comparisons should detect meaningful degradation, and production traces should continuously refresh the suite. An LLM application is ready for release not when every generated sentence matches a fixture, but when measured behavior remains within explicit quality, safety, cost, and latency tolerances under representative conditions. That shift turns evaluation from an ad hoc prompt-checking exercise into a disciplined software engineering control for probabilistic systems.