How Long-Running ECS Tasks Survive Rolling Deployments Without Duplicating Output

The first deploy of the new pipeline shipped cleanly: metrics looked right, the deployment circuit breaker came back green, and the on-call channel stayed quiet. Two days later, a downstream consumer team pinged us. Their dashboard showed the same five-minute snapshot reported twice, by two upstream task runs that did not agree on the values. Both runs had completed normally. The problem was an assumption ECS makes about the tasks it replaces.

The pipeline ran as an ECS service on Fargate with a desired count of one. The container looped on a five-minute cadence: at each boundary it pulled inputs, computed a network state snapshot for that window, wrote the result to S3, and slept until the next boundary. One iteration took about ninety seconds. The service used a textbook rolling update: minimum healthy percent of 100, maximum percent of 200, deployment circuit breaker enabled. The configuration was correct for the workload it was designed for.

A rolling deploy of a stateless web server and a rolling deploy of a long-running task are different problems. For a web server, each request is a discrete unit and the load balancer routes around tasks being drained. For a task that runs ninety seconds and produces an output, a deploy that lands mid-iteration leaves two instances alive: the old one finishing under the old image and the new one starting under the new image. Both compute the same window, read the same inputs, and target the same output destination. Both exit cleanly, so the circuit breaker sees nothing. Downstream, there are now two versions of the same logical work, and the storage layer cannot tell which one is authoritative.

The fix lives in the output layer rather than the deployment configuration, because ECS has no visibility into what a task's output means. Idempotent keying at the write boundary lets the system tolerate duplicate execution by collapsing duplicate outputs into a single canonical result. It adds no coordination between tasks, does not slow deployments, and holds for unplanned restarts as well as planned rollouts.

Why ECS Rolling Deployments Break Long-Running Tasks

Two parameters govern an ECS service deployment: minimumHealthyPercent and maximumPercent. At 100 and 200, the scheduler starts tasks under the new task definition before stopping any existing ones, and it considers the deployment complete once the new task count reaches the desired count and the old tasks have stopped. For a stateless service behind a load balancer, this is the right model. For a task that produces output, it assumes the running task can be replaced without semantic consequence, and that assumption does not hold.

Here is how it played out for us. The service task, running revision N, started an iteration at the top of a five-minute window. About thirty seconds in, a deploy rolled out revision N+1. ECS started the new task and left the old one running until its stop timeout. The new task began an iteration for the current window on startup, because that is what its loop does. The old task wrote its result. The new task wrote its result. Both writes went to S3 under keys that included the task's launch timestamp, so they landed as two objects. The downstream consumer listed the window prefix, found two candidates, and picked one by listing order. Different consumers picked differently, and their metrics disagreed.

This failure does not show up in load tests. Health checks pass and the deploy dashboard stays green. The bug surfaces later, somewhere downstream, as an inconsistency in metrics, a flickering chart, or a model that disagrees with itself. By the time someone correlates the symptom to a deployment window, several deploys have passed, and consumers have already mixed the duplicate writes into their reads. We found it only because the consumer team's anomaly detector tripped on the divergence.

Code block 1 shows the Terraform that produces this behavior by default. The task definition and the deployment configuration are both correct. The problem is the interaction between the rolling update and the application's write semantics, which Terraform cannot see.

Code block 1: ECS service and task definition with a by-the-book rolling update

# ECS service running a single long-lived task. The container loops on a
# 5-minute cadence and writes one network state snapshot per window.
# The deployment configuration is by-the-book and still leaves two tasks
# alive, computing the same window, during every rolling deployment.
resource "aws_ecs_task_definition" "network_state_job" {
  family                   = "network-state-snapshot"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "2048"
  memory                   = "4096"
  execution_role_arn       = aws_iam_role.task_execution.arn
  task_role_arn            = aws_iam_role.task.arn
  container_definitions = jsonencode([{
    name      = "snapshotter"
    image     = "${aws_ecr_repository.snapshotter.repository_url}:${var.image_tag}"
    essential = true
    environment = [
      { name = "OUTPUT_BUCKET",  value = aws_s3_bucket.snapshots.bucket },
      { name = "JOB_VERSION",    value = var.job_version },
      { name = "WINDOW_SECONDS", value = "300" }
    ]
  }])
}

# Rolling update: start the new task before stopping the old one. With a
# 90-second iteration, the old task is usually mid-write when the new task
# starts its own iteration for the same window.
resource "aws_ecs_service" "network_state" {
  name            = "network-state"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.network_state_job.arn
  desired_count   = 1
  launch_type     = "FARGATE"
  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200
  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
  network_configuration {
    subnets         = var.private_subnet_ids
    security_groups = [aws_security_group.task.id]
  }
}

Detecting Duplicate Output Without Already Knowing You Have It

Detection is often harder than the fix, because by the time the symptom is loud enough to investigate, the data has already been blended downstream. Two approaches cover it.

Forensic scanning looks backward. Scan output destinations for multiple writes against the same logical work identifier. For S3, correlate PutObject records from CloudTrail or S3 access logs with the task ARN that wrote them: two distinct task ARNs writing to the same window key is the signature of this failure. The first write is not the problem; the second write from a different task is.

Preventive instrumentation gives a live signal. Stamp every write with the writing task's ECS task ARN and task definition revision, then scan for windows with more than one distinct task ARN. Run this hourly during and after any deployment to the service.

Code block 2 shows the audit script. It lists objects under a window prefix, groups by logical window, and reports windows with writes from more than one task.

Code block 2: Audit script that flags duplicate task writes per window

"""
Audit logical-key duplicates across an S3 prefix where outputs are tagged
with the writing ECS task ARN. Surfaces cases where two tasks wrote to the
same logical job-key window, typically during rolling deployments.
Run this hourly during and after any deployment to the service.
Exit code 1 if duplicates are found, 0 otherwise, so it composes with CI.
"""
import boto3
from collections import defaultdict
from datetime import datetime, timedelta, timezone

S3 = boto3.client("s3")
BUCKET = "network-state-snapshots"

def window_of(key: str) -> str:
    # Expected key shape: snapshots/<yyyy-mm-dd>/<HHMM>/...
    parts = key.split("/")
    return "/".join(parts[1:3])

def audit(hours: int = 24) -> dict[str, set[str]]:
    cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
    paginator = S3.get_paginator("list_objects_v2")
    dupes: dict[str, set[str]] = defaultdict(set)
    for page in paginator.paginate(Bucket=BUCKET, Prefix="snapshots/"):
        for obj in page.get("Contents", []):
            if obj["LastModified"] < cutoff:
                continue
            head = S3.head_object(Bucket=BUCKET, Key=obj["Key"])
            task_arn = head["Metadata"].get("x-task-arn", "unknown")
            dupes[window_of(obj["Key"])].add(task_arn)
    return {w: arns for w, arns in dupes.items() if len(arns) > 1}

def main() -> int:
    dupes = audit()
    for window, arns in sorted(dupes.items()):
        print(f"DUPLICATE window={window} writers={len(arns)}")
        for arn in sorted(arns):
            print(f"  writer={arn}")
    print(f"scanned windows with duplicates: {len(dupes)}")
    return 1 if dupes else 0

if __name__ == "__main__":
    raise SystemExit(main())

Idempotent Keying as the Structural Fix

Idempotency at the output layer means a duplicate execution cannot produce a duplicate output. Two tasks doing the same logical work resolve to the same key, so the storage layer keeps exactly one result. The mechanism has two halves: a deterministic key derived from the work identity, and a conditional write that makes the second write a no-op.

A workable scheme follows three rules. First, the key derives deterministically from the logical work identity, never from the task's own attributes. Task ARNs, launch timestamps, and container IDs all change between the old and new task, so any key that includes them defeats the purpose. The window boundary, the job version, and a fingerprint over the pinned input versions are stable across both tasks.

Second, the key must be content-aware enough that genuinely different inputs never collide. Pin the input versions at the window boundary and hash them into the key. Two tasks that read the same pinned inputs produce the same key. A correction that arrives later pins different input versions and lands on a different key, which is correct: it is new work, not a retry.

Third, publish through a conditional write. S3's If-None-Match conditional write fails the PUT when the key already exists, which is as close to a compare-and-swap as object storage offers. The first writer wins; the second sees the precondition failure and treats it as success, because the object it would have written is byte-identical to what is already there.

Code block 3 shows the Rust implementation we landed on. The key is derived from the trigger window, a job version, and a fingerprint over the pinned input table versions.

Code block 3: Deterministic output key derivation

//! Two task executions for the same trigger window and same pinned inputs
//! always produce the same key, and therefore collide at the storage layer
//! where the conditional write enforces single-writer semantics.
use sha2::{Digest, Sha256};

pub struct WorkIdentity {
    pub window_start_epoch: u64,
    pub job_version: u32,
    pub source_table_versions: Vec<(String, u64)>,  // resolved as of the window boundary, never read live
}

impl WorkIdentity {
    pub fn output_key(&self) -> String {
        let mut h = Sha256::new();
        h.update(self.window_start_epoch.to_be_bytes());
        h.update(self.job_version.to_be_bytes());
        let mut pinned = self.source_table_versions.clone();
        pinned.sort();
        for (table, version) in &pinned {
            h.update(table.as_bytes());
            h.update(version.to_be_bytes());
        }
        let digest = h.finalize();
        format!("snapshots/{}/{:x}/state.parquet", self.window_start_epoch, digest)
    }
}

The other half is the conditional write. The deterministic key collapses duplicate logical work into a single destination, and the conditional write turns the race between the two tasks into a single winner with no coordination between them.

Code block 4: Conditional write with conflict retry

//! Of two tasks racing toward the same key, one PutObject succeeds and the
//! other sees PreconditionFailed (treated as a no-op) or, if both are in
//! flight together, ConditionalRequestConflict (retried, then no-op).
use std::time::Duration;
use aws_sdk_s3::{Client, primitives::ByteStream};

const MAX_ATTEMPTS: u32 = 5;

pub async fn publish_once(
    client: &Client,
    bucket: &str,
    key: &str,
    payload: Vec<u8>,
    task_arn: &str,
) -> anyhow::Result<()> {
    for attempt in 1..=MAX_ATTEMPTS {
        // If-None-Match: "*" tells S3 to fail the PUT if the key already exists.
        // The check is server-enforced, so it holds even when client-side
        // coordination is impossible, as during a rolling deployment.
        let result = client
            .put_object()
            .bucket(bucket)
            .key(key)
            .body(ByteStream::from(payload.clone()))
            .if_none_match("*")
            .metadata("x-task-arn", task_arn)
            .send()
            .await;
        match result {
            Ok(_) => return Ok(()),
            Err(e) if is_precondition_failed(&e) => return Ok(()), // already published: no-op
            Err(e) if is_conditional_conflict(&e) && attempt < MAX_ATTEMPTS => {
                tokio::time::sleep(Duration::from_millis(200 * attempt as u64)).await;
                continue;
            }
            Err(e) => return Err(anyhow::Error::from(e)),
        }
    }
    Err(anyhow::anyhow!(
        "conditional write to {key} did not settle after {MAX_ATTEMPTS} attempts"
    ))
}

Trade-Offs and Adoption Guidance

Idempotent keying has costs. The deterministic hash adds single-digit milliseconds per job when it runs over input metadata, and more if it hashes payloads; keep the fingerprint over pinned input versions, not over data. The conditional write adds one round trip with retry backoff, which is negligible next to a ninety-second iteration.

A less obvious cost is operational flexibility. Once outputs are content-keyed, you cannot rerun a job and overwrite its result by accident, which is the point, but it also means deliberate replays need a new job version. Build the job version into the key from the start so replays are a first-class operation rather than a hack around the scheme.

Storage growth is the second consideration. Duplicate writes are dropped rather than overwritten, but successive job versions and new logical windows each add objects. A lifecycle policy on the output prefix keeps this bounded; the keying scheme makes objects immutable, so expiring old windows is safe.

Roll this out incrementally. Start with the audit script running hourly against production to measure how often duplicate writes actually happen. Then ship the keyed write path behind the old path in shadow mode: compute the key, log it, but keep writing to the old location. When the shadow keys match the expected single-winner behavior across a few deployments, cut over. The audit script becomes your ongoing verification.

The general lesson is that long-running compute under rolling deployments is a regime where the orchestrator cannot give you correctness, because the orchestrator does not understand your output. Pushing correctness down to the write boundary, where the semantics live, turns a deployment-time race into a property of the data.

If you take one habit away from this, take this one. Write the property-based test for your keying scheme before you build the rest of the pipeline. Two executions with the same work identity must produce the same key; two executions with different pinned inputs must not. That test is the contract everything else hangs on.

References

AWS, "Amazon S3 now supports conditional writes," AWS What's New, August 2024. https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/

AWS, "Deploy Amazon ECS services by replacing tasks (rolling update)," AWS Documentation. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html

AWS, "How to prevent object overwrites with conditional writes," Amazon S3 User Guide. https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html

AWS SDK for Rust, aws-sdk-s3 crate documentation. https://docs.rs/aws-sdk-s3/

proptest crate, property-based testing for Rust. https://docs.rs/proptest/