When a Coding Agent Fixes the Test Instead of the Bug

A coding agent opens a pull request for a background job that keeps retrying after the user cancels it. The test that caught the bug now passes, and the agent reports that all tests are green. But the diff changes two things: the retry handler and the test itself. Which change eliminated the failure, and would the test still catch the bug if it came back?

I use coding agents to design integration tests, triage failures, and prepare fixes for back-end services, so this is a practical review question for me. Sometimes the test really is wrong, and changing it is the right fix. Sometimes the agent has just found the shortest way to a green build. In continuous integration, both look the same.

Benchmarks show how far it can go. ImpossibleBench makes the tests contradict the task, so any passing solution has to cheat, and GPT-5 still passed 54% of its conflicting SWE-bench tasks. But tests aren't always right, either: OpenAI found issues with the tests or the task description in 59.4% of the 138 SWE-bench Verified problems it examined.

So a test edit proves nothing by itself. Below, I'll show how to get evidence either way with a small Python example: one bug, three PRs that all pass CI, and a check that tells them apart in seconds.

A Green Build Is a Statement About the Checks

I find it useful to look at a test run as a pipeline. Tests are selected and set up, the application runs, the test observes and asserts, the harness reports the outcome, and a CI gate turns that outcome into a merge decision.

Figure 1: A test run as a pipeline. A change to any stage other than the application changes what a green result means.

Only the application stage can repair a defect. If a PR touches only the application, going from red to green tells you something about the code. If it also touches another stage, the green result may say more about the evidence than about the bug.

One Bug, Three Pull Requests

The Failing Test

Here's the contract for the example. Once cancel() returns, no new attempt may start for that job. An attempt that's already running may still report its result, and a job that wasn't canceled must still be retried after a transient failure.

The model below starts with one attempt already running and handles callbacks one at a time. run_retry() stands in for the scheduler starting a queued retry; there are no threads, network calls, or sleeps. Save it as retry_job.py:

class RetryJob:

    def __init__(self):
        self.attempts = 1
        self.canceled = False
        self.retry_pending = False

    def cancel(self):
        self.canceled = True
        self.retry_pending = False

    def on_transient_failure(self):
        self.retry_pending = True

    def run_retry(self):
        if self.retry_pending:
            self.retry_pending = False
            self.attempts += 1

The bug is about ordering. cancel() clears any queued retry, but the running attempt can report a transient failure afterward, and on_transient_failure() then queues another retry without checking whether the job was canceled. These two tests go in tests/test_retry_job.py:

from retry_job import RetryJob


def test_cancel_blocks_retry_after_late_failure():
    job = RetryJob()
    job.cancel()
    job.on_transient_failure()
    job.run_retry()
    assert job.canceled is True
    assert job.attempts == 1


def test_uncanceled_job_retries():
    job = RetryJob()
    job.on_transient_failure()
    job.run_retry()
    assert job.attempts == 2

Run python -m pytest -q from the repository root. The cancellation test fails because attempts is 2, and the ordinary retry test passes.

Three Ways to Turn It Green

The first PR leaves the application alone and loosens one assertion:

-    assert job.attempts == 1
+    assert job.attempts >= 1

Now the test accepts a second attempt, and a third and a fourth. Its name still promises that cancellation blocks a retry, but the assertion no longer checks that promise.

The second PR fixes the handler and leaves the tests alone:

     def on_transient_failure(self):
+        if self.canceled:
+            return
         self.retry_pending = True

The cancellation test retains its meaning, and the ordinary retry test shows that the guard hasn't disabled retries for every job.

The third PR does both. It's the easiest to miss: the application diff looks right because the application is right now, but the test can no longer catch the bug if a later change brings it back. Ordinary CI reports 2 passed for all three PRs.

Run the Swap Check

Ordinary CI runs exactly one combination: the PR's tests against the PR's code. The two combinations it skips answer the questions a reviewer actually has:

• Base tests on head code. Does the change still meet the original expectations? If an original test fails here, the PR changed a promise, and that needs a reason.
• Head tests on base code. Do the new tests still detect the bug? For a bug fix, at least one of them should fail here. If they all pass, they can't tell the buggy code from the fixed code.

Figure 2 shows all four combinations for the three PRs:

Figure 2: Swap matrices for the three pull requests. CI only shows the head code with head tests, which is green in all three.

In an agent's PR, the tests are part of the change, so you have to swap them on purpose. Git worktrees make that cheap. These commands build the cell that matters most for a bug fix, the base code with the head's test files:

base=$(git rev-parse main)            # commit IDs, because inside a worktree
head=$(git rev-parse fix-and-weaken)  # HEAD would mean something else
tests=$(git diff --name-only --no-renames "$base" "$head" -- \
        tests conftest.py pytest.ini)

cell=$(mktemp -d)                     # base code + head tests
git worktree add --quiet --detach "$cell" "$base"
git -C "$cell" restore --source="$head" -- $tests
(cd "$cell" && python -m pytest -q)
git worktree remove --force "$cell"

On the third PR, this prints 2 passed: the new tests can't see the bug. For the other cross cell, add the worktree at $head and restore the test files from $base. The PR that only loosens the assertion raises a flag in both cross cells, and the honest fix raises none.

Whole-suite results are coarse, though. A head test can fail on the base code just because it imports something the fix added, and a test that already fails on the base hides the signal. In real use, compare outcomes test by test, as SWE-bench does with its FAIL_TO_PASS and PASS_TO_PASS lists.

The list of test paths matters, too. It has to cover everything that defines a check, including fixtures, snapshots, and harness configuration such as pytest.ini, and files that mix application and test settings, such as pyproject.toml, need a human look. This conftest.py hook, for example, rewrites failed test reports as passed:

import pytest


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.failed:
        report.outcome = "passed"

In a PR that adds only this hook, with only tests on the list, the swap check treats conftest.py as application code and reports a clean fix. Add conftest.py, and both warnings appear. Anthropic researchers describe this family of hacks, along with calling sys.exit(0) before any assertion runs.

Treat the result as review evidence, not a verdict: a refactoring's new tests should pass on the old code, while a bug fix's reproducer should fail there. I'd run the swap check as a report-only CI job with the full Git history (fetch-depth: 0) and, because it runs the candidate's code, a read-only token that isn't left in the checkout (permissions: contents: read, persist-credentials: false).

When Changing the Test Is the Right Fix

Tests change all the time. In one comparison, human-written PRs modified existing tests more often than agent-assisted ones: 43.5% versus 20.6% in a manual sample. So the useful question is why a test changed. Four answers hold up:

1. The requirement changed. If the service now lets one already-dispatched retry start after cancellation, the test has to say so, and someone has to have approved the new guarantee.
2. The fixture was wrong. Maybe it labeled a permanent failure as transient. Fix it, but keep a transient-failure case so the retry path stays covered.
3. The test looked too early. If cancellation is acknowledged asynchronously, wait for the acknowledgment or step a fake scheduler forward instead of adding a sleep.
4. The test pinned an implementation detail. Move the check to the public boundary and show that it still catches a violation. This was the largest category in OpenAI's audit, at 35.5% of the examined problems.

In none of these cases is the application's current output the reason for the new expectation. Ask for the contract, the corrected setup, or the reason the old observation was invalid, plus swap-check evidence that the revised test still fails on the bug.

Guardrails

In the Agent Loop

An agent needs room to investigate failures, including bugs in tests, and a clear way to report a conflict between a requirement and an assertion rather than quietly resolving it. For the cancellation task, I'd commit the reproducer first, after checking that it fails for the right reason, and give the agent an instruction like this:

Fix the retry behavior so that no new attempt starts after cancel() returns. Keep the late-failure reproducer and the ordinary retry case. If you think a test is wrong, stop, explain how it conflicts with the contract, and show any proposed test change separately. Report the commands you ran and any skipped, expected-failing, or unexecuted cases.

To make tests read-only for an agent, Claude Code permission rules can deny edits to path patterns, and a PreToolUse hook can block a tool call by exiting with status 2; Cursor's preToolUse hook works the same way. These rules cover the agent's file tools, not every shell command, unless you also run Claude Code's sandbox with allowUnsandboxedCommands set to false.

In my own setup, the agent that writes integration tests works only in test code, and its standing instructions forbid silencing or deleting a failing test to get a green run. Triage first classifies each failure as an application, environment, or test problem; application bugs become bug reports, not test edits. Changes to the agents' own instructions need a person's confirmation, and agents never merge.

In the Repository

Repository controls decide what gets merged. The useful ones keep the evidence for important behavior out of the candidate patch's reach, as Figure 3 shows:

Figure 3: What the candidate patch can change, and what should be reviewed separately.

Own the evidence. Assign test, fixture, harness, and CI paths to owners in CODEOWNERS, and turn on "Require review from Code Owners"; the file alone only requests reviews. GitHub reads it from the base branch, so a PR can't loosen its own owners. Give CODEOWNERS itself an owner, and require approval of the most recent push.

Keep acceptance checks outside the patch. Keep a small acceptance suite for important contracts in a separate repository or at a pinned reference, and run it from a workflow the PR can't edit, such as a ruleset-required workflow on GitHub Enterprise Cloud or a pipeline execution policy on GitLab Ultimate. Run it from its own checkout with its own pytest.ini, because pytest searches parent directories for configuration and could pick up the candidate's conftest.py, and set PYTEST_DISABLE_PLUGIN_AUTOLOAD to block plugins the candidate's package might register.

Prove that the check ran. Make the acceptance job unconditional: a required status check accepts a skipped result, and on GitHub "a job that is skipped will report its status as 'Success'." Then fail the job unless the required cases ran and passed. Any runner that writes JUnit XML, including pytest, Maven Surefire, and Gradle, can feed a short script that checks each required test by name. Don't count a test that passed only on a rerun; Surefire keeps the build green and records the failed attempt as a flakyFailure. Parse the report with defusedxml, since it's untrusted input.

No single control catches everything. The required-tests check trusts a harness that lies, the acceptance run can't see a weakened test next to a real fix, and the swap check is only as good as its path list.

Costs and Limits

The swap check runs the suite four times, so for a large suite I'd limit it to the affected tests. It needs a base revision where the reproducer can run, flaky tests add noise to the cross cells, and each cell must import its own checkout rather than an installed package or a shared node_modules folder.

These controls also target shortcuts rather than code written to defeat them. In a seventh variant of the example, retry_job.py patches pytest's report object when it's imported, and all three controls pass it. Where that matters, run acceptance tests against a built artifact or a running service, and keep reading the application diff.

What to Ask for Before Approving

Ask for evidence instead of a verdict. For the fixed example, a useful PR description would say:

The late failure callback queued a retry after cancellation. The handler now ignores that callback when the job is canceled. The original cancellation test fails on the old code and passes on the repair; the ordinary retry test also passes. Removing the guard makes the cancellation test fail again. The assertion and event order are unchanged.

For a real PR, I'd want to see five things before approving:

• The reproducer fails on the base code and passes on the head code.
• For a bug fix, at least one head test fails on the base code due to an assertion rather than an import.
• Every edited test has a stated reason: a changed requirement, a fixed fixture, a corrected observation point, or a removed implementation detail.
• Changes to selection, harness, or CI configuration were approved by their owners.
• The required tests ran and passed, by name, not just by exit code.

Wrapping Up

Reviewers have always had to ask why a test changed. Coding agents make the question come up more often, because they can rewrite code and tests in minutes and keep going until CI is green. Treat changes to tests, fixtures, the harness, and CI as changes to the evidence: swap tests, accept edits with a reason, and keep checks for important behavior out of the patch's reach. A passing run is only useful when you can explain why it passed.