Building an AI Agent That Converts Production Failures Into Regression Tests

Production failures often contain enough evidence to explain what went wrong, but not enough structure to become an executable test. A trace may expose the failing request path, a log may contain the exception, and downstream spans may reveal the dependency response that triggered the defect. The useful engineering step is to transform that evidence into a deterministic regression test rather than another incident summary. 

Recent bug-reproduction systems follow the same principle that a useful reproducer should fail on the buggy revision for the reported reason and become passing evidence after the defect is fixed. Issue2Test and ReProAgent both use execution feedback instead of treating test generation as a single prompt-and-response operation. 

Start From the Incident Evidence

The agent should begin from a machine-readable incident envelope, not a copied stack trace. OpenTelemetry’s stable log data model includes TraceId and SpanId, while its exception conventions associate exception records with the corresponding span context. W3C Trace Context standardizes traceparent for propagating trace identity across service boundaries. Those identifiers allow the failing execution path to be reconstructed without forwarding an entire observability dataset to a model. 

A small adapter can convert an alert into the minimum evidence required by the agent:

FailureContext buildContext(Incident incident) {
    Trace trace = telemetry.getTrace(incident.traceId());
    Span failed = trace.failedSpan();

    return new FailureContext(
        failed.operation(),
        failed.exception(),
        trace.parentPath(failed),
        trace.downstreamCalls(failed),
        repository.revision(incident.deploymentId()));
}

The deployed revision is essential. A regression test generated against current source can target code that has already moved away from the production state. The incident should therefore resolve to the commit, image digest, or equivalent immutable revision that produced the telemetry. The trace supplies runtime evidence, and the repository supplies the code that interpreted it.

Telemetry also requires reduction before model access. Request bodies, authorization headers, customer identifiers, and database values are rarely necessary to reproduce control flow. OpenTelemetry documents Collector processors to remove attributes, filter records, redact attributes, and transform values before export. Those controls should run before failure context reaches the agent rather than relying on a model to ignore sensitive fields. 

Reduce the Failure to Executable Context

Raw traces are too broad for test generation. The agent needs a compact slice containing failing application frames, the request shape, relevant downstream interactions, and nearby tests that define local conventions. ReProAgent’s 2026 design separates bug localization, root-cause analysis, test planning, and test generation, combining repository retrieval with runtime interaction. Its results support treating reproduction as a staged, tool-using process rather than direct code completion. 

For a checkout failure, an error span may show InventoryClient.reserve() followed by a NullPointerException after the inventory service returned HTTP 503. Retrieval should locate InventoryClient, the calling checkout path, exception mapping, and existing checkout tests. Unrelated controllers, persistence code, and complete trace payloads add noise without strengthening the reproducer.

The resulting agent input can be expressed as an explicit contract:

TestRequest request = new TestRequest(
    context.failureFingerprint(),
    context.relevantSource(),
    context.relatedTests(),
    context.downstreamResponses(),
    "Generate one deterministic JUnit regression test. " +
    "Do not modify production code. Do not assert the observed bug as correct behavior."
);

That final constraint is critical. A model can produce a test that asserts NullPointerException simply because production emitted it. Such a test would pass on the buggy implementation and preserve the defect. Bug-reproduction benchmarks instead use fail-to-pass behavior where the test fails on the pre-fix revision and passes after the correcting patch. Recent research on LLM repair validation also finds that passing executions can provide little bug-discriminating evidence, making differential validation important. 

Generate the Test Against the Intended Contract

The oracle should come from repository evidence rather than model invention. Existing tests, API specifications, exception policies, sibling implementations, and documented response contracts can establish intended behavior. When those sources conflict, the candidate should remain unresolved instead of receiving a fabricated assertion.

Consider a production failure where inventory returned 503 and checkout converted a missing response body into an internal NullPointerException. Existing endpoint tests may establish that unavailable dependencies map to a stable 503 response with an INVENTORY_UNAVAILABLE code. The generated regression test can encode that contract while reproducing the recorded dependency behavior:

stubFor(post(urlEqualTo("/inventory/reservations"))
    .willReturn(aResponse()
        .withStatus(503)
        .withBody("{\"code\":\"overloaded\"}")));

mockMvc.perform(post("/orders")
        .contentType("application/json")
        .content(failureRequest))
    .andExpect(status().isServiceUnavailable())
    .andExpect(jsonPath("$.code").value("INVENTORY_UNAVAILABLE"));

WireMock can match HTTP requests and return predefined responses, and it supports fixed or randomized delays and lower-level fault simulation. That allows a recorded external condition to become a deterministic test setup rather than a dependency on a live production service. 

Close the Loop With Execution Feedback

Generation should be treated as the first candidate, not the final artifact. Issue2Test refines tests using compilation and runtime feedback, while ReProAgent includes runtime interaction throughout reproduction. A practical agent should compile and execute every candidate in an isolated checkout of the incident revision. 

TestCandidate refine(TestCandidate candidate, FailureContext context) {
    for (int attempt = 0; attempt < 4; attempt++) {
        TestRun run = sandbox.run(context.revision(), candidate);

        if (run.compiles() && reproduces(run, context))
            return candidate;

        candidate = model.revise(candidate, run.diagnostics(), context);
    }
    return TestCandidate.rejected();
}

The reproduces check should be stricter than “test failed.” It can verify that the expected application path was reached, the recorded downstream condition was exercised, and the observed exception or response fingerprint overlaps the incident. Compilation failures feed back into correction, a test that fails before reaching the target path is rejected and a test that passes on the buggy revision is not a reproducer.

Once a fix exists, the same test should run against both revisions. ReProAgent defines fail-to-pass rate around exactly this distinction: failure on the buggy state and success after the issue-resolving patch. Differential execution is stronger evidence than asking a model whether generated code appears correct. 

Make the Test the Durable Artifact

After deterministic replay, the reproducer can enter the normal test suite. JUnit treats failed assertions and uncaught exceptions as test failures, so ordinary CI can enforce the regression once the test is valid. Normal execution should require neither production telemetry nor another model call, and incident secrets should never be embedded in the generated fixture. 

A practical CI handoff can also preserve provenance without preserving raw incident data. A small metadata record can contain the incident identifier, source revision, generated test path, reproduction fingerprint, and validation command. That record makes regeneration and review easier while keeping the committed test independent of the observability backend. The test itself remains the executable source of truth.

In practice, the generated test is verified under strict CI controls before ever reaching the main suite. The agent’s changes (adding the new test) occur on an isolated branch or worktree, and the CI pipeline runs git diff to confirm that only test files were created or modified, any application code changes cause an immediate failure. The test is then run against the original codebase to confirm it reproduces the production failure, and again against the patched build to ensure it now passes. Any anomaly (for example, the test accidentally passing on the buggy code or still failing after the fix) triggers a manual review. 

Meanwhile, any necessary fixtures from the incident (such as specific database records or request parameters) are set up in the test so it precisely mirrors the failure scenario. Metadata from the failure (stack trace, error message, etc.) is included in the commit or PR for traceability. This enforces that each generated test is precise and verifiable in CI before the developer ever sees it.

Production observability becomes substantially more valuable when failures can be converted into executable evidence. The reliable pattern is to correlate telemetry to the deployed revision, reduce that evidence to the failing path, derive assertions from existing contracts, generate a deterministic test, and repeatedly execute it until the production failure is faithfully reproduced. The final acceptance criterion is demanding but clear: the test must fail for the real bug, pass after the real fix, and remain safe enough to run on every future change. That turns an AI debugging agent from a code generator into a controlled mechanism for converting operational failures into permanent regression protection.