How to Safely Deploy Control-Plane and Data-Plane Changes With Argo CD and Argo Rollouts

Deployments become harder when one release changes both the machinery governing a Kubernetes platform and the workloads that depend on it. A new CustomResourceDefinition, admission webhook, controller, or routing API can change what the cluster accepts and how it behaves; a new application image can simultaneously depend on those changes. Argo CD and Argo Rollouts solve different parts of this problem. 

Argo CD is suited to establishing declarative prerequisites in a deterministic order, while Argo Rollouts limits production exposure as a workload version moves toward stable. Safe delivery comes from composing those responsibilities rather than treating either controller as a universal deployment engine. Argo Rollouts documentation explicitly discourages using Rollouts for infrastructure components such as cert-manager, CoreDNS, and NGINX. 

Compatibility Must Exist Before Progression Starts

The core rule is an expand-and-contract sequence. Control-plane changes first add capabilities without removing behavior required by the running data plane. Only after old and new workloads can both operate against the expanded control plane should production traffic move to the new workload. Destructive cleanup follows after promotion. This separation preserves the ability to run multiple workload versions during progressive delivery while avoiding an immediate dependency on an irreversible platform change. 

This matters most for Kubernetes APIs. A CRD can serve multiple versions, with exactly one storage version, and conversion webhooks can translate between representations. Kubernetes documentation recommends removing an old API version only after stored objects have migrated away from it, and conversion support is no longer required. That lifecycle maps naturally to staged GitOps delivery: add the new version and compatible controller behavior, roll out consumers, migrate stored state if necessary, then remove the legacy version later. 

The same principle applies to admission, routing, and controller behavior. During the transition, the new policy must not reject objects still produced by the stable workload, and the new routing configuration must preserve the stable path. This compatibility window keeps rollback viable while both data-plane versions may coexist, which is also a prerequisite called out by Argo Rollouts for applications using progressive delivery. 

Let Argo CD Establish the Prerequisites

Argo CD sync phases and waves provide deterministic ordering inside an Application. Resources are ordered by phase, numeric wave, kind, and name. Argo CD applies the first wave containing an out-of-sync or unhealthy resource and continues only as earlier work becomes synchronized and healthy. Negative waves place infrastructure prerequisites ahead of workloads. 

A CRD and controller can therefore precede the Rollout resource without external pipeline orchestration:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-20"   # CRD

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-10"   # controller

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"     # Rollout

The exact values are less important than the health boundary between them. A controller Deployment that never becomes healthy prevents the dependent wave from advancing. Waves are sequencing gates rather than transactions, so they cannot make an incompatible schema change atomic; backward compatibility must still survive the interval between applied resources. For custom resources with nonstandard status, Argo CD supports custom Lua health checks, allowing readiness to reflect controller reconciliation rather than mere object creation. 

Hooks add active validation. A failing PreSync hook stops synchronization before normal resources are applied, while PostSync runs after synchronized resources are healthy and can perform smoke tests. Safety-critical flows should avoid selective sync because hooks do not run during selective synchronization and that operation is not recorded in history. 

Pruning also warrants conservative treatment during control-plane evolution. Argo CD supports Prune=false and Prune=confirm, allowing critical objects to be protected from automatic deletion. When a CRD is introduced in the same sync as its custom resources, Argo CD automatically skips dry run for those new custom-resource types, avoiding failure before the API exists. 

Let Argo Rollouts Control Production Exposure

After the control plane is compatible and healthy, the data-plane update can progress independently. A canary rollout exposes changes in steps rather than replacing all replicas at once. With a traffic-routing provider, Argo Rollouts manages stable and canary Services and adjusts routing according to Rollout state; the stable ReplicaSet remains available while traffic shifts. For Istio host-level traffic splitting, the Rollout controller also updates the referenced VirtualService weights to match the current canary step. 

A concise policy can combine small initial exposure, automated analysis, and progressively larger weights:

strategy:
  canary:
    stableService: checkout-stable
    canaryService: checkout-canary
    trafficRouting:
      istio:
        virtualService:
          name: checkout
          routes: [primary]
    steps:
      - setWeight: 5
      - pause: {duration: 2m}
      - analysis:
          templates:
            - templateName: checkout-slo
      - setWeight: 25
      - pause: {duration: 5m}
      - setWeight: 50
      - analysis:
          templates:
            - templateName: checkout-slo

The stable and canary Service references give Rollouts explicit targets for traffic control, while the named Istio route identifies the route whose weights may change during progression. This pattern follows the host-level traffic-splitting model documented for the Istio integration. 

An AnalysisTemplate defines metrics, measurement intervals, and success or failure conditions, so failed analysis can stop progression automatically. Blue-green deployments provide related gating through pre-promotion analysis before the active Service switch and post-promotion analysis that can abort and restore traffic to the previous stable ReplicaSet. 

The analysis signal should reflect behavior that distinguishes stable from canary operation. Error rate, latency, saturation, or a domain success ratio provides evidence beyond pod readiness. Timed pauses create an observation window, but time alone is not a success criterion; automated analysis turns that window into an explicit promotion gate. Readiness establishes that a process can receive traffic, while progressive analysis determines whether receiving production traffic remains acceptable. 

Prevent Git Reconciliation From Fighting Traffic Management

Progressive delivery creates an ownership boundary. Argo CD owns desired configuration in Git, while Argo Rollouts intentionally mutates live routing fields as progression advances. Without explicit diff rules, Git reconciliation can reapply static weights while Rollouts is setting dynamic weights, creating brief traffic-weight flapping. Argo Rollouts documents this conflict for Istio VirtualServices and recommends ignoring differences together with applying only out-of-sync resources. 

The Argo CD Application can exclude only controller-owned weight fields while keeping the rest of the route declarative:

spec:
  ignoreDifferences:
    - group: networking.istio.io
      kind: VirtualService
      jqPathExpressions:
        - .spec.http[].route[].weight
  syncPolicy:
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - RespectIgnoreDifferences=true

Argo Rollouts specifically documents ignoring VirtualService HTTP route weights as a way to prevent dynamic Rollouts changes from making the Argo CD Application appear out of sync. ApplyOutOfSyncOnly=true then prevents already synchronized resources from being reapplied unnecessarily. RespectIgnoreDifferences=true makes the ignore policy apply during synchronization rather than only during diff calculation. 

The boundary should remain narrow. Ignoring only route weights preserves Git ownership of hosts, matches, destinations, and other static policy while granting Rollouts authority over the values that legitimately vary during progressive delivery. Broad exclusions weaken drift detection and obscure changes unrelated to the rollout itself. Argo CD supports path-level and field-manager-based difference customization specifically to constrain such exceptions. 

Failure Must Converge Back to Git

An aborted rollout is not a complete GitOps rollback. Argo Rollouts can restore the stable ReplicaSet, but Git may still request the failed image. The Rollout then remains degraded because the live stable state differs from the desired state. Argo Rollouts documentation states that reapplying the previous stable manifest restores health and is recognized as a rollback, allowing the controller to fast-track the stable ReplicaSet instead of replaying normal analysis steps. 

That distinction is critical for combined changes. A data-plane failure should normally revert the workload commit while leaving a backward-compatible control-plane expansion in place. Removing newly added API or routing capability during the same emergency action can make recovery less predictable. Control-plane contraction belongs in a later, independently verified change. Sync windows can further restrict high-risk synchronization to approved periods because Argo CD supports allow and deny windows scoped by application, namespace, or cluster. 

Safe delivery with Argo CD and Argo Rollouts is therefore a matter of ownership, compatibility, and sequencing. Argo CD should establish and verify backward-compatible platform prerequisites, while Argo Rollouts should control how much production traffic reaches the new data plane and stop promotion when evidence turns negative. Controller-owned routing fields must be excluded narrowly from Git reconciliation, and an aborted rollout must be followed by a Git change that restores the stable desired state. When expansion precedes exposure and contraction follows verified promotion, control-plane evolution and application delivery remain independently reversible, turning progressive deployment into a reliable safety mechanism rather than merely a rollout technique.