Building a Scope-Constrained AI Agent for Automated CI Failure Repair

CI failures often expose a narrow defect through a broad stream of evidence such as compiler output, test reports, stack traces, changed files, dependency metadata, and build logs. Large language models can turn that evidence into candidate repairs, and recent industrial research has shown that LLM-assisted CI repair can resolve a meaningful share of compilation failures when the model receives carefully selected context. 

The difficult part is not generating a patch. It is ensuring that automated repair changes only code justified by the failure, leaves protected files untouched, and proves that the original failure is actually fixed. Automated program repair research has repeatedly identified patch overfitting as a core risk: a patch can satisfy available tests without being semantically correct. 

Capture Failure Evidence Before Asking for a Fix

A repair agent should start from structured CI evidence rather than a raw prompt containing an entire build log. GitHub Actions artifacts can persist test output, logs, and other files after a job completes, making them a practical handoff format between the failing job and a later repair job.  

For a Maven project, the failing job can retain both console output and Surefire reports:

- name: Run tests
  run: ./mvnw test 2>&1 | tee ci.log

- name: Upload failure evidence
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: failure-evidence
    path: |
      ci.log
      target/surefire-reports/

The repair process should normalize this data into a compact incident object containing the failing test, exception type, stack trace, failing command, commit SHA, and original pull-request diff. Current surveys of LLM-based software repair describe retrieval and analysis augmentation as important layers for repository-level repair, reinforcing the value of selecting failure-relevant context before generation. 

Before patch generation, the failure should be reproduced in an isolated checkout. Git worktrees support linked working directories and can create a detached worktree for experimental changes without disturbing the primary checkout.  A repair job can create a disposable workspace and rerun only the failing test:

git worktree add --detach ../repair "$HEAD_SHA"
cd ../repair
./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test

Maven Surefire supports selecting a specific test class and, for supported providers, individual test methods. If the failure cannot be reproduced, automatic patching should stop. A passing rerun provides no reliable oracle for judging a generated fix.

Turn Failure Localization Into an Edit Boundary

The central control is an explicit distinction between readable and editable files. A failing test, configuration file, or build script may be necessary for diagnosis without being safe for autonomous modification. A scope document can represent that boundary:

failure: PaymentServiceTest.shouldRetryTimeout
readable: [src/test/java/com/acme/payment/**, src/main/java/com/acme/payment/**, pom.xml]
editable: [src/main/java/com/acme/payment/**]
protected: [src/test/**, pom.xml, .github/**, db/migration/**]

The editable set can be derived from the original PR diff, stack-trace locations, and a small neighborhood of related production symbols. Git provides machine-readable changed-path output through git diff --name-only, while pathspecs can limit comparisons to selected repository regions.  

The model can inspect broader context, but the orchestrator should reject any patch that escapes the approved surface.

A validator needs a repair baseline captured immediately before the model edits the worktree:

REPAIR_BASE=$(git rev-parse HEAD)

git diff --name-only "$REPAIR_BASE" > repair-files.txt

while read -r path; do
  case "$path" in
    src/main/java/com/acme/payment/*) ;;
    *) echo "Out-of-scope change: $path"; exit 1 ;;
  esac
done < repair-files.txt

This check matters because prompt instructions such as "modify only relevant code" are advisory, while a diff gate is enforceable. Git also provides exit-status behavior through git diff --exit-code, which is useful for automated policy checks. 

Generate a Patch, Not an Open-Ended Coding Session

The repair model should receive a bounded request like normalized failure data, relevant source fragments, the original PR diff, editable paths, protected paths, and the exact verification command. The response should be structured around a patch rather than unrestricted shell execution.

RepairRequest request = new RepairRequest(
    failure,
    relevantSource,
    originalDiff,
    editablePaths,
    protectedPaths,
    verificationCommand
);

Patch patch = repairModel.generate(request);
patchValidator.validate(patch);
patchApplier.apply(patch);

The model proposes the smallest plausible correction where the orchestrator remains responsible for permissions and validation. This separation reflects a broader trend in LLM-based program repair toward procedural and agentic workflows that combine generation with retrieval, analysis, execution feedback, and external control. 

For an initial implementation, tests and CI configuration should remain read-only. A failure that genuinely requires changing a test, dependency declaration, migration, or workflow can be escalated for manual review rather than allowing the agent to broaden its own permissions.

Validate More Than a Green Build

A candidate patch should first rerun the exact failing test. Only after that test passes should validation expand to the affected module and then the normal CI suite.

./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test
./mvnw -pl payment test
./mvnw verify

Passing tests alone are not proof of correctness. Automated program repair research documents the overfitting problem, where generated patches satisfy the available test suite without correctly repairing the underlying behavior.  

Scope validation therefore acts as an additional safeguard. A successful repair should preserve protected files, avoid dependency changes, keep the patch reviewable, and retain the original failing test unchanged.

A final guard can verify that tests were not modified during repair:

git diff --quiet "$REPAIR_BASE" -- src/test || {
  echo "Repair modified protected tests"
  exit 1
}

The resulting output should be an evidence package rather than an opaque "fixed" status, but contain the root-cause summary, exact patch, changed-file list, focused-test result, module-test result, and full-CI result. That evidence makes the automated repair reviewable.

To ensure a patch does not escape its intended scope, the workflow can run the agent in an isolated Git worktree and enforce explicit path-based allowlists. For example, a CI script might use git worktree add -d /tmp/agent main so the agent’s changes occur in a separate directory. After the agent generates a commit, the pipeline can run git diff --name-only origin/main...HEAD and check that all file paths match the allowlist (for instance, grep -qvE '^(src/payment/|config/)' rejects unauthorized modifications). Protected paths (e.g. .git/) would instantly be blocked by the allowlist. The pipeline then reruns the failing test class in isolation (mvn -Dtest=PaymentServiceTest test) to confirm the issue is fixed, and executes the full test suite. A quick diff or mutation-based check can detect if any test assertions were weakened or disabled. 

Finally, a canary merge or smoke-test job automatically undoes the change on failure. For instance, a GitHub Actions step might include the following so that any failing health check immediately triggers an automated rollback:

- name: Rollback if smoke tests fail
  if: steps.smoke.outcome == 'failure'
  run: ./rollback.sh

 These combined safeguards (worktree isolation, allowlist enforcement, retests, and rollback triggers) help guarantee the AI-generated patch is minimal, correct, and reversible.

Keep the Repair Job Less Privileged Than the Code It Examines

An autonomous repair workflow processes repository content and may execute code originating from a pull request, so CI permissions need careful separation. GitHub warns that privileged workflow_run and pull_request_target workflows must not blindly check out untrusted pull-request code, and recommends granting GITHUB_TOKEN only the minimum required permissions. 

A safer pattern is to perform diagnosis and patch generation with read-oriented credentials, store the candidate patch as an artifact, and use a separate trusted step to create a branch or pull request. Workflow artifacts should also be treated cautiously when produced by untrusted jobs. This prevents the repair agent from turning a build failure into a path toward repository write access.

Conclusion

A useful CI repair agent is not an LLM attached directly to a failing build. It is a constrained repair pipeline that captures structured failure evidence, reproduces the defect, narrows the editable surface, generates a minimal candidate patch, rejects out-of-scope changes, and validates the result from the failing test through the full CI suite. Industrial evidence shows that LLM-assisted repair can be effective in CI settings, but automated program repair research also shows why a green build cannot be treated as sufficient proof.  

Enforced edit boundaries, isolated execution, layered testing, and least-privilege automation convert model-generated code from an uncontrolled suggestion into a reviewable, evidence-backed repair mechanism.