Designing the Runtime Layer for Stateful AI Agents: Where Agent Substrate Fits

Introduction: The Layer Nobody Designs

AI agents are evolving from simple prompt-response systems into stateful, long-running applications that can reason, execute tasks, and interact with external systems. Yet, most agent architectures are driven by models, tools, memory, and orchestration — not runtime infrastructure that can run agents reliably.

Unlike traditional applications, stateful AI agents need execution context to run (and recover from failures), concurrent tasks, and the ability to resume workflows without losing progress if they fail, etc. These capabilities cannot be provided by prompts or orchestration frameworks alone — there needs to be a runtime layer.

This is where Agent Substrate becomes relevant. It provides a runtime foundation for stateful agent execution so that developers can separate the intelligence of the agent from the infrastructure that runs it (aka runtime).

A closer look at stateful AI agents reveals why a dedicated runtime layer is essential, the architectural challenges it must address, and how Agent Substrate fits into the broader agentic AI technology stack.

What Makes Agent Workloads Different

AI agents might look like traditional workloads, but their execution patterns challenge the fundamental assumptions of application infrastructure. Request-driven services or batch jobs cannot run forever; agents can make random decisions, interact with other systems, and change their own state continuously. Five characteristics that make their runtime requirements fundamentally different.

• Long-lived, mostly waiting: An agent task can take minutes, hours, or days, and most of that time is spent waiting on a model, a tool, or a human. Holding a process open for the whole duration is wasteful and fragile. 
  • Requirement: Execution must be able to pause, release resources, and resume.
• Non-deterministic: The same input can take a different path each run. You can’t recover by re-running the code and expecting the same result, so what the agent did and decided has to be recorded as it happens. 
  • Requirement: Durable records of steps and outcomes, not just final state.
• Side effects on the real world: Agents send emails, move money, and modify systems. A naive retry after a crash can repeat an action that already happened. 
  • Requirement: Know exactly which steps completed, and make retries safe.
• State is the product: Memory, plans, and open tasks accumulate and are read and rewritten constantly. This isn’t a cache you can throw away; losing it means losing the agent. 
  • Requirement: State that is durable, versioned, and structured enough to tell current from stale.
• Many actors, shared state: Agents act for users, hand work to other agents, and collaborate with humans. Without boundaries, they overwrite each other, and nobody can say who decided what. 
  • Requirement: Isolation, access control, and an audit trail.

Stateful AI agents need the durability of a database, the recoverability of a workflow engine, and the isolation of a multi-tenant platform, all integrated into a coherent execution environment. While traditional infrastructure offers these capabilities independently, agent workloads demand them together. This is the architectural gap that an agent runtime layer must address.

The Lifecycle of a Stateful AI Agent

Traditional applications run in predictable request-response cycles, but stateful AI agents run through dynamic long-running workflows. Their execution involves multiple reasoning steps, tool calls from the outside world, human approvals, and periods of inactivity.

A typical stateful agent lifecycle involves six stages:

1. Initialize: Receive a task, establish execution context, and create an initial state.
2. Execute: Perform reasoning, make decisions, invoke tools, and update execution state.
3. Persist: Save progress, intermediate results, and relevant context at durable checkpoints.
4. Suspend: Pause execution while waiting for an external event, tool response, or human approval.
5. Resume or Recover: Continue from the last valid checkpoint when the required event arrives or recoverable failures occur, avoiding duplicate side effects.
6. Complete: Finalize the task, record the outcome, and retain relevant execution history for auditability.

Importantly, this lifecycle is not strictly linear. An agent may repeatedly execute, persist, suspend, and resume before reaching completion. Failures may also trigger retries, recovery, or escalation.

The runtime layer is responsible for managing these transitions reliably, ensuring that agent execution remains durable, observable, and recoverable without requiring the application developer to implement the lifecycle infrastructure from scratch.

Why Traditional Infrastructure Is Not Enough

Traditional infrastructure provides many of the building blocks needed to run AI agents, but no single piece of it covers the entire lifecycle of a stateful agent. The table below shows where traditional infrastructure falls short.

The real challenge is not the absence of infrastructure primitives. It is the need to combine them into a coherent runtime layer that can preserve state, resume execution, recover from failure, and enforce policy across long-running agent workflows.

Designing the Runtime Layer for Stateful AI Agents

A stateful AI agent needs more than compute resources and an execution environment. It also needs a runtime that preserves execution state, handles long-running processes, recovers from interruptions, and efficiently allocates resources.

Unlike traditional application runtimes, which handle processes and requests only, an agent runtime has workloads that are frequently switching between active execution and waiting for external events.

A well-designed agent runtime should address five core capabilities. These capabilities lay the foundation for running stateful agents reliably without requiring every application to build its own execution infrastructure.

Agent Substrate is an open-source runtime for running stateful workloads (e.g., AI agents) on Kubernetes. Instead of treating each agent as a container permanently running, it uses an actor-based execution model where instances of agents can be suspended when idle and resumed when needed.

Its architecture combines:

• Actor lifecycle management: Tracks actor instances and manages their suspension and resumption.
• Checkpointing and restoration: Preserves process memory and writable filesystem state through snapshots.
• Worker pools: Maintains available compute capacity for scheduling and restoring actors.
• Execution isolation: Supports sandbox technologies such as gVisor and microVMs.
• Kubernetes integration: Uses Kubernetes for underlying worker provisioning and infrastructure management.

Separating Reasoning, Orchestration, and Runtime Infrastructure

A reliable agent architecture separates three responsibilities: what the agent decides; how its work is coordinated; and where its execution is maintained. They are tightly coupled, but they address different problems (though they interact).

The reasoning layer decides what to do next; the orchestration layer coordinates when and how that action is executed. The runtime infrastructure ensures that execution can continue reliably, even when agents become idle, encounter failures, or require additional resources.

Where Agent Substrate Fits

Traditional infrastructure provides the foundation for running applications, but stateful AI agents need a runtime layer to manage execution continuity, isolation, and resource utilization.

Agent Substrate is an open-source Kubernetes-based runtime that addresses these requirements. Instead of treating each agent as a running container, it uses actor-based execution so stateful workloads can be suspended when idle and restored when execution needs to continue.

Agent Substrate introduces several capabilities that align with the runtime architecture discussed earlier:

• Actor-based execution environments: Each agent workload is represented as an actor with its own execution context, enabling independent management of stateful instances.
• Actor lifecycle management: Coordinates actor creation, placement, execution, suspension, resumption, and termination.
• Suspension and resumption: Allows inactive actors to be suspended and restored when new requests arrive, reducing the need for continuously allocated compute.
• Memory and filesystem preservation: Uses snapshots to capture process memory and writable filesystem state, enabling execution to continue across suspension and restoration.
• Worker pools and resource multiplexing: Maps a large number of stateful actors onto a smaller pool of available workers, improving infrastructure utilization.
• Isolation and request routing: Uses sandbox technologies such as gVisor and microVMs to isolate execution, with ingress routing and controlled network access to manage communication securely.

Mapping Agent Substrate to the Runtime Architecture

Why This Matters

Consider an AI agent that initiates a task, invokes external tools, and then waits for human approval. In a conventional deployment, its execution environment may remain allocated throughout the waiting period.

With Agent Substrate, the actor can be suspended, its execution state preserved, and its compute resources released. When execution needs to continue, the actor can be restored onto an available worker.

This separation allows many stateful workloads to be managed without maintaining a dedicated running environment for every idle agent. 

Agent Substrate does not replace agent frameworks or workflow orchestration engines. It provides the execution infrastructure; frameworks do reasoning and application logic, while runtime does isolation, state preservation, scheduling, and continuity of execution.

Real-World Use Case: Stateful AI Incident Investigation

Various use cases of stateful AI agents where Substrate can be useful are –

Implementation Details: AI Incident Investigation

Consider an AI incident investigation agent that diagnoses infrastructure failures. When an alert fires, the agent collects telemetry, looks at logs, correlates recent deployments, and builds an investigation context.

Unlike a request-response application, the investigation may take several minutes or hours. The agent might have to wait for monitoring data, an engineering response, or human approval before continuing.

Keeping a dedicated execution environment running during this wait time is unnecessary. Restarting the agent, however, requires reconstructing its investigation context and intermediate results.

An agent runtime (e.g., Agent Substrate) solves this problem by providing a stateful execution environment that can be suspended and restored. For example:

1. Initialize: An infrastructure alert starts an investigation actor.
2. Execute: The actor collects logs, metrics, and deployment information.
3. Persist: Investigation context and intermediate findings are retained in the actor's execution state.
4. Suspend: The actor is suspended while awaiting additional information or human approval.
5. Resume: When new information becomes available, the actor is restored and continues using its preserved context.
6. Complete: The investigation produces a diagnostic summary and recommended remediation actions.

Step 1: Build a Minimal Incident Investigation Service

Create a Python or Go application that exposes:

Step 2: Maintain Investigation State

The actor could preserve a state object similar to:

{
  "incident_id": "INC-1042",
  "service": "payment-api",
  "status": "waiting_for_approval",
  "investigation": {
    "error_rate": "12%",
    "suspected_cause": "recent deployment",
    "deployment": "v2.4.1"
  },
  "approval_required": true,
  "recommended_action": "review rollback"
}

This represents application-level investigation context. The actor must actually load and update this data for the demo to demonstrate preservation.

Step 3: Deploy the Service as an Agent Substrate Actor

Package the application as an OCI image and create an Agent Substrate actor template configured with an appropriate sandbox, worker pool, and full-state snapshot policy.

The existing counter example provides a useful reference for configuring an actor template.

Step 4: Demonstrate Suspension and Resumption

Once the actor has stored its investigation context, manually suspend it:

kubectl ate suspend actor incident-1042 -a incident-demo

When a new request arrives through the configured actor router, the suspended actor can be activated and its state restored. You can also use the explicit resume lifecycle operation.

These commands assume you have created an actor named incident-1042 in an atespace named incident-demo; they are not commands for the existing counter demo.

Step 5: Verify the Result

After resumption, retrieve the investigation status and verify that the agent still has its original findings. 

Hands-on Tutorial: Running and Resuming a Stateful Workload with Agent Substrate

The objective of this tutorial is to execute a stateful workload, suspend it, and resume it without losing its execution state. This experiment uses a local Kubernetes environment with Podman and Kind.

Setting Up Agent Substrate

First, clone the Agent Substrate repository

git clone https://github.com/agent-substrate/substrate.git
cd substrate

With Podman configured as the container provider for Kind, create the local Kubernetes cluster and install the Agent Substrate runtime:

export KIND_EXPERIMENTAL_PROVIDER=podman

./hack/create-kind-cluster.sh

./hack/install-ate-kind.sh \
  --deploy-ate-system \
  --credential-provider='{"name":"k8s.io"}'

Note: The upstream installation scripts assume Docker. Our Podman-based setup requires local compatibility adjustments, so this should be treated as an experimental installation path rather than an officially supported deployment configuration.

Once deployed, verify that the runtime components are running: kubectl get pods -n ate-system

Deploying the Stateful Counter

Agent Substrate provides a counter demo that maintains state across three locations:

• Process memory: An in-memory counter.
• Filesystem: A counter stored in a file.
• Root filesystem: A counter stored in the actor's writable root filesystem.

Deploy the demo and create an actor instance:

./hack/install-ate-kind.sh --deploy-demo-counter

kubectl ate create actor my-counter-1 \
  -a ate-demo-counter \
  --template counter

Forward the router service to the local machine:

kubectl port-forward -n ate-system \
  svc/atenet-router 8000:80

From another terminal, invoke the actor:

curl -X POST \
  -H "ate-target-actor: ate-demo-counter/my-counter-1" \
  http://localhost:8000/

The initial execution returned:

Each subsequent request increments all three counters, demonstrating stateful execution within the actor.

Check the status of the actor:

Suspending the Actor

After invoking the actor multiple times, suspend it using Agent Substrate's lifecycle controls.  

During suspension, Agent Substrate checkpoints the actor's execution state and releases its active execution slot. Unlike terminating a conventional container, suspension is designed to preserve the state required to continue execution later.

Resuming the Actor

Resume the suspended actor and invoke the actor again.  

Instead of restarting the counters from their initial values, the actor continues incrementing from its previously preserved state. For example, if all counters reached 5 before suspension, the next successful invocation produces:

This illustrates the key capability of the runtime: execution can be suspended and restored while preserving both in-memory and filesystem state.

Observing the Results

The experiment demonstrates the following behavior:

Illustrative values for five requests before suspension; actual counts depend on the number of requests executed.

The important observation is not the counter value itself, but its continuity across the suspension and resumption boundary. For the complete Podman, Kind, and Agent Substrate setup instructions, along with all tutorial execution commands, refer to the GitHub repository.

What Happens Behind the Scenes?

Agent Substrate coordinates several runtime mechanisms to support this behavior:

• Actor lifecycle management: Tracks running, suspended, and resumed actors.
• Checkpointing: Captures execution state, including process memory and filesystem changes, according to the configured snapshot scope.
• Snapshot storage: Retains checkpoints so the actor can be restored.
• Worker management: Releases execution resources during suspension and allocates resources when the actor resumes.
• Request routing: Directs subsequent requests to the appropriate actor execution environment.

The experiment shows that there is a fundamental difference between normal process execution and stateful actor runtime: an actor can preserve its logical execution state without continuously occupying an active worker execution slot.

This is particularly useful for AI agents that spend large portions of their lifetime waiting for external tools, model responses, or human decisions.

However, preserving execution state does not guarantee exactly-once execution of external side effects or transactional consistency. Those concerns require additional application-level controls.

Architectural Trade-Offs and Production Considerations

Agent Substrate provides capabilities for running stateful AI agents efficiently, but having a runtime layer of its own has architectural trade-offs too. Production environments need to balance execution continuity with resource use, performance, security, and operational complexity.

The following considerations are important when designing a production-ready agent runtime.

Beyond Infrastructure-Level Recovery

One important point to note is that preserving an agent’s execution state is not the same as guaranteeing correctness of its actions.

For example, an agent calls a payment API and fails immediately after the payment succeeds but before recording the response. Restoring the agent from its last checkpoint could cause the payment request to execute again.

A runtime can restore execution context, but it cannot tell whether an external operation was done successfully or not.

Production systems therefore need additional mechanisms (idempotency keys, durable operation records, transactional outbox patterns and reconciliation workflows) to reduce duplicate side effects and ensure consistency across system boundaries.

Similarly, runtime-level checkpointing does not guarantee transactional consistency between agent state, databases, and external services.

Security and Operational Readiness

Beyond execution management, production deployments require integration with broader platform capabilities, including identity-based access controls, policy enforcement, secrets management, observability, and resource governance.

These capabilities help ensure that agents operate within defined security boundaries and that their execution can be monitored, audited, and controlled.

The key architectural principle is that an agent runtime provides execution continuity, while application-level orchestration and governance remain responsible for ensuring operational correctness.

Agent Substrate addresses important infrastructure challenges, but building production-ready agentic systems still requires coordinated design across the reasoning, orchestration, runtime, and platform layers.

Conclusion

As AI agents evolve from simple request-response systems to autonomous long-running workloads, the infrastructure supporting them has to evolve too. Reasoning capabilities aren’t enough; agents need reliable execution, persistent state, isolation, and recovery to work in production.

Traditional infrastructure provides the building blocks, but integrating them into a coherent runtime remains a challenge. Agent Substrate demonstrates how an actor-based execution model, combined with checkpointing, suspension, resumption, and resource multiplexing, can help bridge this gap.

However, a runtime is only one part of the broader agentic architecture. Workflow orchestration, security, policy enforcement, and application-level consistency remain equally important for building trustworthy systems.

Ultimately, the next step in agentic AI is not just making agents more intelligent, but making their execution more reliable, resilient, and operationally sustainable. A well-designed runtime foundation is what enables intelligent agents to move from experimentation to production-scale systems.