Closing the Drift Loop: An AI Agent That Drafts Terraform Fixes

Every Terraform team I’ve worked with has drift detection. A scheduled terraform plan -detailed-exitcode, a Terraform Cloud health assessment, a Spacelift or env0 monitor. Exit code 2 fires, a Slack message lands, and someone opens a ticket.

Then the ticket sits. Drift found in minutes gets fixed in days, because the fix needs a human to read the diff, work out why the resource changed, decide whether to codify or revert, write the HCL, and push a PR. Detection is automated. Remediation is still artisanal.

The Gap 

The tempting fix is to auto-apply on drift. Almost nobody does that in production, and they’re right not to. A blind apply on a security group that someone widened during an incident re-breaks the thing they just fixed. Drift means reality and code disagree, and “code wins” is not always the right answer.

The middle path is an agent that reads the drift, drafts a remediation as a pull request with its reasoning and its prediction of the post-merge plan, and then stops. The PR is the gate. The real plan runs as a required check on that PR, the same one every human PR gets, and the agent is never in the apply path. I’ve been running this against production stacks in two regions. It turns a days-long ticket into a short review.

I’ve published the workflows as a reference implementation in this repository. The linked repo keeps the same shape (detect → remediate → PR gate → apply) with the minimum needed to run. 

Firefly and env0 sell versions of this idea, and tfdrift’s remediate command and autoai-driftguard are open-source ones. The idea is the easy part. What I want to write down is the set of design decisions that make it safe to leave running, because that’s where the failures live.

Three properties make the shape defensible. The agent’s blast radius is a branch: it can’t touch state, cloud resources, or the merge button, so the worst case is a PR a reviewer closes (for an honest agent, at least; more on that later). The existing change process is reused, not bypassed: CODEOWNERS, required reviews, and policy checks run on the agent’s PR exactly as they do on a human’s. And the judgment call stays with a person, but it arrives pre-researched.

Two Kinds of Drift

When plan says a resource differs from state, there are two directions to go:

A detector can’t tell them apart. A t3.medium that is now a t3.large is either someone’s forgotten experiment (revert) or last week’s capacity fix that never made it into code (codify). The evidence lives outside Terraform: CloudTrail, tickets, tags, Slack threads. Gathering that evidence and writing down a hypothesis a human can confirm or reject in a minute is exactly what an LLM is good at and a cron job is not.

The Workflow

Four stages. Only the second one is new.

Stage 1: Detection

A scheduled workflow runs a plan and captures a machine-readable output.

# .github/workflows/drift-detect.yml (abridged)
on:
  schedule: [{ cron: "0 */6 * * *" }]
jobs:
  plan:
    permissions: { id-token: write, contents: read, actions: write, issues: write }
    strategy:
      fail-fast: false
      matrix:
        include:   # CloudTrail LookupEvents is regional; the agent runs where the resource lives
          - { stack: network/prod-use1, region: us-east-1 }
          - { stack: network/prod-use2, region: us-east-2 }
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }}   # reads + lock-table write; plan holds the lock
          aws-region: ${{ matrix.region }}
      - name: Plan (refresh-only)
        id: plan
        run: |
          set +e
          terraform plan -refresh-only -detailed-exitcode -lock-timeout=5m \
            -out=drift.tfplan -input=false
          code=$?; set -e
          echo "code=$code" >> "$GITHUB_OUTPUT"
          [ "$code" -eq 1 ] && exit 1        # plan error: fail loudly
          terraform show -json drift.tfplan > drift.json
      - name: Count drifted resources        # refresh-only plans report under resource_drift
        id: count
        run: |
          n=$(python3 -c "import json; d=json.load(open('drift.json'));
          print(sum(1 for r in d.get('resource_drift', []) if r['change']['actions'] != ['no-op']))")
          echo "n=$n" >> "$GITHUB_OUTPUT"
      - name: Drift-storm guard              # >20 drifted resources is a provider/state problem, not drift
        if: steps.plan.outputs.code == '2' && steps.count.outputs.n > 20
        run: gh issue create --title "[drift] storm in ${{ matrix.stack }}" --body "Paging a human."
      - name: Dispatch remediation
        if: steps.plan.outputs.code == '2' && steps.count.outputs.n <= 20
        run: gh workflow run drift-remediate.yml -f stack=${{ matrix.stack }} -f region=${{ matrix.region }} -f run_id=${{ github.run_id }}

Three details do the work. -refresh-only separates real drift from HCL that merged but hasn’t been applied yet. The role is read-only in shape but needs lock-table writes because plan holds a state lock, which is also why -lock-timeout=5m is there: a scheduled plan that collides with an apply should wait, not fail. And the storm guard exists because a provider upgrade looks like drift on every resource; past 20 in one run, the workflow pages a human instead of dispatching the agent.

Stage 2: The Remediation Agent

The agent has three capabilities: read the plan JSON, read evidence, open a PR. No cloud writes, no terraform apply.

# .github/workflows/drift-remediate.yml (abridged)
on:
  workflow_dispatch:
    inputs: { stack: { required: true, type: string }, region: { required: true, type: string }, run_id: { required: true, type: string } }
jobs:
  remediate:
    permissions: { contents: write, pull-requests: write, id-token: write, actions: read }
    steps:
      - uses: actions/download-artifact@v4
        # real workflows sanitize '/' -> '-' in the stack name so this matches the upload step
        with: { name: drift-${{ inputs.stack }}, run-id: ${{ inputs.run_id }}, github-token: ${{ secrets.GITHUB_TOKEN }} }
      - uses: aws-actions/configure-aws-credentials@v4
        with: { role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }}, aws-region: ${{ inputs.region }} }
      - name: Analyze drift                # model + AWS read access; no GitHub write token here
        run: python agent/remediate.py analyze --stack "${{ inputs.stack }}" --region "${{ inputs.region }}" --plan drift.json --out proposal/
        env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}, ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL }} }
      - name: Open PR                      # bot token only; see below for why not GITHUB_TOKEN
        run: python agent/remediate.py propose --stack "${{ inputs.stack }}" --proposal proposal/
        env: { GH_TOKEN: ${{ secrets.DRIFT_BOT_TOKEN }} }

The token in that last step matters more than it looks. GitHub limits workflow runs for events created with GITHUB_TOKEN, so a PR opened with it gets a pull_request run parked in an approval-required state: no jobs, no checks on the PR, no error. If the plan and policy checks are required, the PR can’t merge until someone with write access clicks “Approve workflows to run.” 

The tool surface is small, and every tool is read-only except the last:

TOOLS = [
    read_plan_json,      # resource_drift entries from drift.json
    read_hcl,            # repo files, capped per file
    read_tags,           # tags on the drifted ARN
    cloudtrail_lookup,   # LookupEvents in the stack's region, last 14 days
    describe_resource,   # aws ec2/rds/iam describe-*
    search_issues,       # GitHub issues/PRs mentioning the resource ID
    propose_fix,         # the ONLY write: stages a branch + PR body under proposal/
]

Two limits on the CloudTrail tool. LookupEvents returns management events only, for one region, for the last 90 days, 50 per page, at two calls a second. That covers “who changed this security group” and not “who read this object”; for older questions, point the tool at CloudTrail Lake or the S3 bucket storing logs.

Notice what’s missing: no apply, no cloud modify, no merge. A “revert” decision changes no HCL, because the HCL is already right. Instead, the agent commits a decision record, .drift-decisions/<address>-<fingerprint>.md, with the direction it chose, the evidence, and the plan it expects after merge. That gives the PR a real diff (GitHub refuses PRs with no commits) and puts the decision in git history next to the resource. A revert PR’s plan check exits 2 on purpose, and the gate compares the plan to the record’s claim instead of failing on the exit code.

The system prompt sets the contract:

You are a Terraform drift analyst. For each drifted resource you will (1) determine whether the out-of-band change should be codified or reverted, (2) cite the evidence that led you there, (3) produce the minimal HCL diff for the codify case, and (4) rate the strength of the evidence as strong, partial, or none. You cannot apply changes. If evidence is insufficient to choose a direction, say so and draft the codify variant anyway so a reviewer can see what accepting the change would look like.

That last sentence matters. An agent that won’t act under uncertainty is the same as no agent. Drafting the codify variant costs cents against an engineer’s hour, and it’s reversible.

Stage 3: The PR as the Gate

Here is the agent’s output on a real drift, a security group that grew an ingress rule outside Terraform:

[drift] network/prod-use1: aws_security_group.app_alb drifted out-of-band

Recommendation: Codify (evidence: strong)

Drift: ingress tcp/8443 from 10.40.0.0/16 not present in state.

Evidence: CloudTrail AuthorizeSecurityGroupIngress at 2026-09-08T03:12Z by role IncidentResponder-oncall. Matches issue #431 (“ALB health checks failing from new monitoring subnet”), closed the same day. The CIDR is aws_subnet.monitoring, added in PR #412, which never updated the SG.

Change: adds the ingress block to modules/alb/sg.tf, parameterized on var.monitoring_cidr.

Expected post-merge plan: no changes. The plan check on this PR confirms or refutes that.

If you disagree: close this PR; the next scheduled apply removes the rule. #431 suggests that re-breaks monitoring.

The reviewer’s job goes from “investigate” to “verify.” Every line is checkable in under a minute, and the plan check proves the fix converges.

The gate is the same set of required checks human PRs pass, plus one step that judges the agent’s claim:

- name: Plan on PR
  run: |
    set +e; terraform plan -detailed-exitcode -out=pr.tfplan -input=false
    echo "code=$?" >> "$GITHUB_OUTPUT"; set -e
    terraform show -json pr.tfplan > pr.json
- name: Judge only the claimed resource
  # codify: the claimed resource is no-op/absent in pr.json
  # revert: the claimed fingerprint still appears under resource_drift
  run: python agent/gate.py --plan pr.json
- name: Policy check
  run: conftest test pr.json -p policy/
# Branch protection: 1 approval + Code Owners review. The bot can't approve
# its own PR, and "Allow GitHub Actions reviews to count" stays off.

Branch protection does the real safety work. The bot can’t approve its own PR, and because its PRs touch platform-owned paths, the platform team’s CODEOWNERS entry has to.

Stage 4: Apply (Unchanged)

On merge, your existing apply pipeline runs. If the agent was wrong and the reviewer missed it, you have git history, a plan artifact, and a normal revert PR. Same failure mode as any bad merge, which is the point.

Guardrails 

One PR per resource. A stack with six drifted resources gets six PRs, each with its own codify/revert decision. One PR for all six is how a reviewer approves five good changes and one bad one.

That rule was the first thing I learned from one of the projects. I staged three drifts at once: a tag added to an S3 bucket (codify), a security-group rule opened to the world (revert), and a subnet deleted out from under state (needs-human). My first gate deadlocked on the first two. The codify check wanted a fully clean plan, the revert check wanted exactly one drifted resource, and each PR’s plan showed the other PR’s drift. Neither could go green. The fix is the “judge only the claimed resource” step above. With it, both PRs passed while the deleted subnet stayed open as an issue. Per-resource independence has to live in the gate, not just in the agent.

Idempotent PRs. Before opening a PR, the agent looks for an open one with the same resource address and drift fingerprint, a hash of the before/after attributes with keys sorted so ordering noise doesn’t create a new fingerprint. If the resource drifts again differently, the old PR is stale: the agent closes it with a pointer to the new one.

Never auto-remediate delete drift. If a resource exists in state and not in reality, the codify path is deleting HCL, which is the easiest change to get catastrophically wrong. The agent files its evidence as a drift/needs-human issue and stops.

Deny-list sensitive types. IAM policies, KMS key policies, and anything under module.security. are evidence-only; the agent files an issue and a human opens the PR. It’s a deny-list on purpose. The human gate already makes the default safe; the list is about blast radius on the day the reviewer is tired, and IAM is where a plausible one-line change grants access to everything. An allow-list would be safer on paper and would spend its first month growing one type at a time while drift queued up.

Cap the context. The agent gets the drifted resource’s resource_drift entry, the HCL file that declares it, and evidence for that ARN. Not the repo. Less context means fewer chances to “improve” adjacent code.

Log the reasoning. Every PR body carries the evidence trail. When a codify decision turns out wrong three weeks later, you want to know whether the agent misread CloudTrail or CloudTrail was silent.

What This Doesn’t Solve

It doesn’t prevent drift. If people can write to prod from the console, they will; this makes the consequence cheap, not the cause. It doesn’t handle state corruption or provider upgrades that look like drift on every resource. And it doesn’t remove the reviewer. If your team approves agent PRs without reading them, you’ve built auto-apply with extra steps.

Conclusion

The industry spent the last few years making drift visible. The next step isn’t making it disappear on its own. It’s making the fix arrive already written, already justified, and already inside the review process you trust. An agent that can only open pull requests is an agent whose worst day is a closed PR. 

Nothing here is specific to GitHub Actions or AWS. Read-only agent, PR as the only write, existing gates unchanged: that ports to GitLab, Azure DevOps, or any Terraform platform with a VCS-driven apply.