Three Measured Outcomes of Manually Scaling Fleet in Prod

Every GitOps setup meets the same moment: something is on fire, someone runs kubectl scale on prod, and the fix works. What happens next depends on two settings that rarely get checked before an incident: self-healing on the GitRepo, and pause on the cluster.

I ran all three combinations in a lab on Rancher 2.15.1 with Fleet (biased, I work at SUSE). Three EKS clusters: dev and staging on Kubernetes 1.36, prod on 1.35. Cases 1 and 2 used the guestbook app from the public rancher/fleet-examples repo, with the frontend Deployment at 3 replicas. Case 3 used the podinfo Helm chart from my own repo. Every manual change went through the scale button in the Rancher UI, which sends a PATCH to the Deployment's scale subresource, the same request kubectl scale makes.

The Two Settings

Self-healing is spec.correctDrift.enabled on the GitRepo. It is off by default. In the Rancher UI it sits in step 4 of the GitRepo wizard, Advanced, Resource Handling, "Enable self-healing". To change it on an existing GitRepo: Continuous Delivery, App Bundles, the GitRepo, the three-dot button top right, Edit Config, then Next until step 4.

Pause lives on the cluster: Continuous Delivery, Clusters, three-dot menu, Pause. Fleet stops deploying to that cluster until you unpause it.

1. Self-Healing On

Turning self-healing on for a GitRepo that already had drift fixed the drift in about 11 seconds. Then I scaled the frontend from 3 to 1 again. Fleet had it back at 3, with 3 pods ready, in under 7 seconds. The GitRepo flickered NotReady for about 3 seconds and went back to Active.

Anyone watching the UI would likely never see the drift. That is what you want from GitOps on a normal day. During an incident it works against you: your fix is gone before you can tell whether it helped.

2. Self-Healing On, Cluster Paused

I paused the dev cluster and scaled the frontend to 5. It held, 5 of 5 ready, for the twelve minutes the cluster stayed paused.

While the cluster was paused, the GitRepo read Ready, 2/2 clusters, with no Modified state anywhere. The app looked healthy in Continuous Delivery while one cluster ran something Git did not say. If you review drift by looking at the GitRepo, a paused cluster drops out of the review.

When I unpaused, the frontend went from 5 back to 3 in about 9 seconds. The GitRepo showed Modified for a moment and then Active. The fix lasted exactly as long as the pause.

3. Self-Healing Off

With self-healing off, the manual scale stays, and Fleet reports it precisely. I scaled prod from 2 to 4 and the GitRepo went to Modified, 2/3, and stayed there. The Ready condition read:

Modified(1) [Cluster fleet-default/c-kqgtk]; deployment.apps shop/shop-podinfo modified {"spec":{"replicas":2}}

That is the best drift report in the product. Two details to know when you read one: it names the cluster by its ID rather than its display name, and the number it shows is the one Git wants. The value currently on the cluster is missing.

The surprise came with the next commit. The repo deploys one app in two lanes: two cluster groups, non-prod and prod, and one fleet.yaml with prod pinned a chart version behind. Promotion is a one-line commit that bumps the prod target. I promoted podinfo 6.14.0 to 6.15.0 while the manual scale was still in place.

The GitRepo sat in WaitApplied, 2/3, and prod stayed on the old version with 4 replicas. The Helm release on the prod cluster recorded the reason:

Upgrade "shop" failed: conflict occurred while applying object shop/shop-podinfo apps/v1, Kind=Deployment: Apply failed with 1 conflict: conflict with "Mozilla" with subresource "scale" using apps/v1: .spec.replicas

This is server-side apply doing its job. The manual scale made the browser's client the field manager of spec.replicas. "Mozilla" is the user agent the browser sent, so managedFields records the client that made the call. The person who clicked appears nowhere in the object. From a terminal the manager would be kubectl. When Fleet's upgrade tried to set spec.replicas back to 2, the API server refused to take the field from another manager without being forced.

None of that showed in Continuous Delivery. The GitRepo showed WaitApplied and the drift message. The conflict text existed only in the failed Helm release on the downstream cluster.

The fix needed no force. I scaled prod back to 2, the value Git asks for, and the promotion landed about 40 seconds later: new chart version, 2 of 2 ready. The Kubernetes docs explain why that works. When two managers set a field to the same value, they share ownership and there is no conflict.

What to Check on Your Own Clusters

• Read the setting before the incident. Check spec.correctDrift.enabled on each GitRepo that targets prod, so you know which of the three cases you are in.
• If you pause a cluster, keep a list of what you paused. Continuous Delivery will not show drift on it until you unpause.
• If a promotion sits in WaitApplied after an incident, look on the downstream cluster. Run helm history on the release and read the description of the failed revision.
• Inspect ownership with kubectl get deployment <name> -n <namespace> -o yaml --show-managed-fields. A manager like "Mozilla" or kubectl next to a field that Git also sets is a manual change waiting to block a deploy.

Limits of This Test

One run, on Rancher 2.15.1 with Fleet and EKS 1.35 and 1.36. Every timing is a single measurement, and the GitRepos polled Git every 60 seconds. I scaled from the UI, not from kubectl in a terminal. I did not test correctDrift.force, which the Fleet docs say can recreate resources, or Helm's force option for this case. After the fix, the browser client may still co-own spec.replicas. Per the server-side apply docs, a later commit that changes replicas could conflict again. I did not test that.

What I'd Take From It

• Pause is a working break-glass. Use it, and keep in mind that while a cluster is paused, Continuous Delivery tells you nothing about it.
• Put the fix in Git before you unpause, or you lose it.
• With self-healing off, an emergency change that never makes it back to Git blocks your next deploy. If a promotion sits in WaitApplied after an incident, look for a field someone changed by hand.

This is a lab finding, not official guidance.