DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Beyond State Consistency: Preventing Stale Decisions in AI Agent Systems
  • Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud
  • Applying Microservices Lessons to Multi-Agent AI Systems
  • Giving AI Agents Safe Access to Legacy Systems: Introducing legacy2mcp

Trending

  • Debugging senro Pipelines: Replay a Run, Shell Into a Live Step, Audit a Cache
  • Carry the Bag of Reality Into the New Year? What Would Gödel Say?
  • Where Should a Transaction End? Designing Consistency Boundaries in Distributed Java Systems
  • Data Integration Landscape 2026: Event Streaming, API, and Batch in the Era of Agentic AI

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. AI/ML
  4. Crash-Proof Agents: Durable Execution for Long-Running AI Systems

Crash-Proof Agents: Durable Execution for Long-Running AI Systems

Learn how write-ahead logs, checkpoints, idempotency keys, and sagas make long-running AI agents resilient to crashes and failures.

By 
Khushan Adatiya
·
Nov. 20, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
125 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Your Agent Is a Distributed System Wearing a Trench Coat

Sometime this week, TrendForce reported that server CPU lead times stretched from 16-20 weeks to roughly 25-30 weeks, and the driver was not a new chatbot launch. It was agentic AI: workloads that stay alive for hours, holding state, invoking tools, orchestrating background work while a GPU does inference next door. The short interactive prompt-response session is giving way to persistent execution, and infrastructure planning is scrambling to catch up.

Here is the uncomfortable part. Most agent code being written today assumes the machine it runs on will still be there in an hour. It assumes no deploy lands mid-run, no spot instance gets reclaimed, no pod gets evicted, no network partition eats an acknowledgment. Every one of those assumptions is false at scale, and we have known this for decades. The patterns that survive machine failure in distributed systems have names: write-ahead logs, checkpoints, idempotency keys, sagas. They work exactly as well for agents as they do for payment pipelines. They are just wearing different clothes.

I spend my days at Google on Apigee Analytics and Monetization, running near-real-time API monitoring at roughly 440k transactions per second against 99.99% availability targets. When a box dies in my world, the run does not die with it. This article is the durable-execution playbook I would hand to a team building agents that run longer than a coffee break: why chat history is a fragile recovery medium, the three primitives that make execution durable, and the observability you need to operate it.

Why Chat History Is a Fragile Save File

The naive recovery strategy for an agent is to persist the conversation and replay it on restart. This fails in three specific ways.

First, replaying thousands of tokens of conversation to remind the model where it left off is expensive and slow. You are paying inference cost to reconstruct position, and position is the cheapest thing in the system to store durably. A single integer — how many steps completed — replaces a context rebuild.

Second, replayed context drifts. A model reading thirty steps of its own history can misinterpret what happened, especially if a tool output changed the plan mid-run. The log says what happened; the model infers what it thinks happened. Those are different things, and the gap is where bugs hide.

Third, replay re-executes side effects. If step 14 called an API that charged a customer or provisioned infrastructure, a naive restart calls it again. This is the exact problem distributed systems solved with idempotency decades ago, and agents get it for free only if they are designed for it.

The durable alternative: record progress as an explicit, structured log of events and checkpoints, and make recovery mean reading the log, not re-running the world.

Primitive 1: The Write-Ahead Log and the Checkpoint

The pattern is old and boring, which is why it works. Before a side effect happens, you write your intent to durable storage. After it completes, you write a checkpoint. Recovery means finding the last checkpoint and replaying only the unacknowledged tail. Temporal is the clearest articulation of this model: workflow code stays deterministic, activities are where side effects live, and a crashed worker resumes by replaying decisions from event history instead of re-executing them. LangGraph brings the same idea to the agent world with a checkpointer: pass one to compile(), and every superstep persists, keyed by thread_id. After a crash, invoking the graph with the same thread_id picks up at the last completed node rather than at the start.

You do not need a framework to apply the pattern. Here is a compact, production-flavored implementation in Python using SQLite as the durable log. It is deliberately simple — one file, no services — because the logic is what matters, and you should be able to test every line of it.

Python
 
import json
import sqlite3
import time
import uuid
from typing import Any, Callable, Optional


class DurableStore:
    """Write-ahead log + checkpoint store backed by SQLite.

    The WAL journal mode means a crashed process never corrupts the log:
    a committed row is a committed row.
    """

    def __init__(self, path: str):
        self._db = sqlite3.connect(path, timeout=30)
        self._db.execute("PRAGMA journal_mode=WAL")
        self._db.executescript(
            """
            CREATE TABLE IF NOT EXISTS agent_log (
                seq INTEGER PRIMARY KEY AUTOINCREMENT,
                run_id TEXT NOT NULL,
                kind TEXT NOT NULL,      -- 'intent' | 'checkpoint'
                payload TEXT NOT NULL,
                created_at REAL NOT NULL
            );
            CREATE INDEX IF NOT EXISTS idx_agent_log_run
                ON agent_log(run_id, seq);
            """
        )

    def append(self, run_id: str, kind: str, payload: dict) -> int:
        cur = self._db.execute(
            "INSERT INTO agent_log (run_id, kind, payload, created_at)"
            " VALUES (?, ?, ?, ?)",
            (run_id, kind, json.dumps(payload), time.time()),
        )
        self._db.commit()
        return cur.lastrowid

    def latest_checkpoint(self, run_id: str) -> Optional[dict]:
        row = self._db.execute(
            "SELECT payload FROM agent_log"
            " WHERE run_id = ? AND kind = 'checkpoint'"
            " ORDER BY seq DESC LIMIT 1",
            (run_id,),
        ).fetchone()
        return json.loads(row[0]) if row else None

    def unacked_intents(self, run_id: str, after_seq: int) -> list[dict]:
        """Intents logged after the last checkpoint with no completion."""
        rows = self._db.execute(
            "SELECT payload FROM agent_log"
            " WHERE run_id = ? AND seq > ? AND kind = 'intent'"
            " ORDER BY seq",
            (run_id, after_seq),
        ).fetchall()
        return [json.loads(r[0]) for r in rows]
Python
 
class DurableRunner:
    def __init__(self, store: DurableStore, run_id: str):
        self._store = store
        self._run_id = run_id

    def run(self, steps: list[tuple[str, Callable[[], Any]]]) -> list[Any]:
        """Execute steps, resuming from the last checkpoint on restart.

        Each step is (idempotency_key, callable). A crash between the intent
        log and the checkpoint is safe: on resume, the step's idempotency
        key tells us whether it actually executed (see Primitive 2).
        """
        checkpoint = self._store.latest_checkpoint(self._run_id)
        done = checkpoint["completed"] if checkpoint else 0
        results: list[Any] = checkpoint["results"] if checkpoint else []

        for i in range(done, len(steps)):
            key, action = steps[i]
            # 1. Write-ahead: log INTENT before the side effect.
            self._store.append(self._run_id, "intent", {"key": key, "step": i})
            # 2. Execute. If this raises, the intent row stays unacked and
            #    resume logic can reconcile it safely.
            result = action()
            results.append(result)
            # 3. Checkpoint: durable proof the step finished.
            self._store.append(
                self._run_id,
                "checkpoint",
                {"completed": i + 1, "results": results},
            )
        return results


def new_run(store: DurableStore) -> str:
    run_id = f"run-{uuid.uuid4().hex[:12]}"
    store.append(run_id, "intent", {"key": "run-start", "step": -1})
    return run_id


Three things to notice. The intent row goes to disk before the side effect, so a crash can never leave you wondering whether the step was attempted — the log tells you, deterministically. The checkpoint stores results, not model context, so resume is a cheap database read rather than a context rebuild. And the results list is append-only history, which doubles as your audit trail for debugging. In my monitoring work, the equivalent insight is that you never reconstruct system state from metrics dashboards; you read the source-of-truth log.

Write-ahead log and the checkpoint

Primitive 2: Idempotency Keys on Every Tool Call

The write-ahead log tells you a step was attempted. It does not tell you whether it succeeded. The killer case: the tool call executed, the process crashed before the checkpoint, and on resume you have an unacked intent for a side effect of unknown outcome. Retrying blindly can double-charge, double-provision, or double-notify.

The fix is an idempotency key on every tool call, plus an executor that enforces at-most-once execution per key even across restarts. The key should be derived from the run and the step — for example, run_id plus step_index — never from a timestamp or a random value, because the resumed run must compute the identical key for the identical step.

Go
 
package durable

import (
    "context"
    "errors"
    "fmt"
    "math"
    "sync"
    "time"
)

// Ledger is the durable record of completed tool calls, keyed by
// idempotency key. Back it with Postgres, SQLite, anything that
// survives the process. The interface keeps this testable.
type Ledger interface {
    // Completed returns the stored result and true if key already ran.
    Completed(ctx context.Context, key string) (any, bool, error)
    // Record persists the result of a completed call. Must be atomic.
    Record(ctx context.Context, key string, result any) error
}

type Executor struct {
    ledger Ledger
    // inflight serializes concurrent Execute calls sharing one key.
    mu       sync.Mutex
    inflight map[string]*call
}

type call struct {
    done   chan struct{}
    result any
    err    error
}

func NewExecutor(ledger Ledger) *Executor {
    return &Executor{ledger: ledger, inflight: map[string]*call{}}
}

// Execute runs fn at most once per key. A duplicate key returns the
// recorded result without re-running fn — this is what makes crash
// recovery safe when the checkpoint write was lost.
func (e *Executor) Execute(ctx context.Context, key string, fn func(context.Context) (any, error)) (any, error) {
    // Fast path: already completed in a previous attempt or process.
    if result, ok, err := e.ledger.Completed(ctx, key); err != nil {
        return nil, fmt.Errorf("ledger read for key %s: %w", key, err)
    } else if ok {
        return result, nil
    }

    e.mu.Lock()
    if c, exists := e.inflight[key]; exists {
        e.mu.Unlock()
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-c.done:
            return c.result, c.err
        }
    }
    c := &call{done: make(chan struct{})}
    e.inflight[key] = c
    e.mu.Unlock()

    c.result, c.err = e.withRetry(ctx, key, fn)

    e.mu.Lock()
    delete(e.inflight, key)
    e.mu.Unlock()
    close(c.done)
    return c.result, c.err
}

func (e *Executor) withRetry(ctx context.Context, key string, fn func(context.Context) (any, error)) (any, error) {
    const maxAttempts = 5
    var lastErr error
    for attempt := 1; attempt <= maxAttempts; attempt++ {
        result, err := fn(ctx)
        if err == nil {
            // Record BEFORE returning: a crash after success but before
            // the record is the exact window the ledger exists to close.
            // If Record fails, the next attempt's Completed check may
            // return false and fn could run again — so Record must be
            // atomic and fn must itself be idempotent where possible.
            if rerr := e.ledger.Record(ctx, key, result); rerr != nil {
                return nil, fmt.Errorf("ledger record for key %s: %w", key, rerr)
            }
            return result, nil
        }
        lastErr = err
        if ctx.Err() != nil {
            return nil, ctx.Err()
        }
        backoff := time.Duration(math.Pow(2, float64(attempt-1))) * 200 * time.Millisecond
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(backoff):
        }
    }
    return nil, fmt.Errorf("key %s failed after %d attempts: %w", key, maxAttempts, lastErr)
}

var ErrDuplicateKey = errors.New("durable: duplicate idempotency key with different payload")


The honest part of this code: there is always a tiny window — success happened, the record did not land — and no client-side trick closes it completely. The durable fix is to push idempotency to the server side (the tool API accepts the key and dedupes), which is why payment APIs all take idempotency keys. Client-side ledgers handle the common case; server-side keys handle the adversarial one. If your tools do not accept idempotency keys, add a reconciliation step on resume: before retrying an unacked intent, query the tool's current state to check whether the action took effect. That "check first, then act" step is the difference between a retry and a duplicate.

Primitive 3: Sagas for Multi-Step Side Effects

Checkpoints and idempotency make individual steps safe. But agents chain steps, and a failure at step 9 of 12 can leave steps 1-8 half-applied: a VM provisioned, a DNS record created, a ticket filed, and nobody to clean up. Retrying forward does not help when the plan itself is dead. You need compensations — each step paired with an undo action, executed in reverse order on failure. That is the saga pattern, and it is the standard answer in distributed transaction design.

Python
 
import logging
from dataclasses import dataclass, field

log = logging.getLogger("saga")


@dataclass
class SagaStep:
    name: str
    action: callable          # forward action; raises on failure
    compensation: callable    # undo; must be best-effort and idempotent


@dataclass
class Saga:
    steps: list[SagaStep] = field(default_factory=list)

    def add(self, name, action, compensation):
        self.steps.append(SagaStep(name, action, compensation))
        return self  # fluent chaining

    def run(self):
        completed: list[SagaStep] = []
        for step in self.steps:
            try:
                log.info("saga: running step %s", step.name)
                step.action()
            except Exception as exc:
                log.error("saga: step %s failed (%s); compensating %d steps",
                          step.name, exc, len(completed))
                self._compensate(completed)
                raise
            completed.append(step)

    def _compensate(self, completed: list[SagaStep]) -> None:
        failures = []
        for step in reversed(completed):
            try:
                log.info("saga: compensating step %s", step.name)
                step.compensation()
            except Exception as exc:
                # Never let a failed compensation mask the original error
                # or abort the remaining rollbacks. Record and continue.
                log.exception("saga: compensation for %s failed", step.name)
                failures.append((step.name, exc))
        if failures:
            # Surface for human review: partial rollback is an incident,
            # not a silent success.
            raise SagaCompensationError(failures)


class SagaCompensationError(Exception):
    pass


Two rules make sagas work in practice. Compensations must be idempotent, because the compensation itself can crash midway and be retried — deleting a VM that is already deleted must be a no-op, not an error. And a failed compensation is an incident, not a bug to swallow: the saga raises so a human or an alerting path sees the partial rollback. In fifteen years of distributed systems work, I have never seen a rollback path that was tested as thoroughly as the forward path, and I have never seen a production incident that cared.

Sagas for multi-step side effects

Operating It: What to Measure

Durable execution is not a library you drop in; it is a runtime property you operate. The metrics that matter:

  • Checkpoint age and checkpoint lag. How far behind the durable log is the last checkpoint? A growing lag means your checkpoint writes are failing silently or your steps are too coarse. Alert on it the way you would alert on consumer lag in a queue.
  • Resume count and replay cost. Every resume has a price: how many intents were reconciled, how many tool calls were deduped by the ledger, how many tokens were spent rebuilding context. If resume cost climbs, your steps are too long or your checkpoints too sparse.
  • Unacked intent age. An intent row with no checkpoint after N minutes is either a stuck tool call or a crashed worker nobody noticed. This is your dead-man's switch.
  • Compensation rate. How often sagas roll back. A rising rate is not a reliability problem — it is a planning problem. The agent's plans are failing more often, and that is a model or prompt issue, not an infrastructure one.

And one operational practice that matters more than any metric: drain-aware deploys. Before replacing a worker, stop admitting new runs, let in-flight runs reach their next checkpoint, and only then shut down. The resume path should be ordinary work — a run continuing from its last checkpoint on a new machine — not an incident. Teams that run preemptible capacity already think this way; agent infrastructure needs the same discipline now that runs last hours instead of seconds.

Build, Buy, or Borrow

You do not have to hand-roll this. The landscape, honestly assessed:

  • Temporal is the most complete answer for complex multi-step workflows: durable timers, signals for human-in-the-loop gates, replay-based recovery. The cost is operational — you run the Temporal cluster — and workflow code must be deterministic, which constrains how you write agent loops.
  • LangGraph's checkpointer is the shortest path if you are already in the LangChain ecosystem: one argument to compile(), persistence keyed by thread. It checkpoints graph nodes, not arbitrary side effects, so pair it with idempotent tools.
  • Restate and DBOS journal handler invocations with the database as the log. Good fit if your agents are structured as services rather than scripts.
  • Airflow 3.3 recently moved its durability boundary from between tasks to inside a task, with an opt-in durable mode for AI agent steps that caches model responses and tool results so retries replay rather than re-pay. If your agents already live in Airflow, this is nearly free.

My bias, from running high-throughput systems: start with the primitives, not the platform. A write-ahead log, idempotency keys, and checkpoint-on-completion cover 80% of crash scenarios with code you fully understand. Adopt a framework when your workflows need durable timers, human approval gates, or cross-service coordination — the point where hand-rolled state machines become the bug factory.

The Boring Truth

Nothing in this article is new. Write-ahead logging predates most agent frameworks by decades. Idempotency keys are how every payment API on earth survives retries. Sagas are textbook distributed transactions. The novelty is only in where the ideas are being applied: to systems that plan, act, and hold state for hours, built by teams that have never had to think about what happens when the machine disappears mid-thought.

The agents that survive production will not be the smartest ones. They will be the ones that can crash at 3 AM, lose a machine, get deployed over, and resume from the last checkpoint as if nothing happened — because someone wrote the boring code that makes forgetting impossible. Be that someone.

The boring truth

AI systems

Opinions expressed by DZone contributors are their own.

Related

  • Beyond State Consistency: Preventing Stale Decisions in AI Agent Systems
  • Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud
  • Applying Microservices Lessons to Multi-Agent AI Systems
  • Giving AI Agents Safe Access to Legacy Systems: Introducing legacy2mcp

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook