DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Architecting Production AI Across Clouds: Patterns That Decide System Survival
  • Principles for Operating Large-Scale Global Production Systems with AI Innovation Across the Stack
  • Observability With AI SREs: A 4-Step Implementation Guide
  • Treating Web Scrapers Like Production Services

Trending

  • Engineering Production Agentic Systems: Part 3: The Topology
  • SEO Writing 101: A Simple Guide to Getting Found
  • Searching Kafka Without an Index: Lessons From Building Message Search for Kafdrop
  • Building Multi-Rail Payment Failover to Stop Latency Spikes and Double Debits

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. AI/ML
  4. Treat AI Agents Like Long-Running Background Jobs: A Production Reference Architecture

Treat AI Agents Like Long-Running Background Jobs: A Production Reference Architecture

Production AI agents behave like long-running background jobs: the model is the cheap part, and retries, idempotency, cost caps, and observability are the real work.

By 
Ido Fishman
·
Nov. 20, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
126 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

I spent six months replacing my own daily admin work with a small fleet of AI agents instead of writing about them. Three run in steady state; a fourth spins up on demand. Between them, they touch email, calendar, Slack, GitHub, a dozen SaaS APIs, a stack of PDFs, and a SQLite database I keep refusing to migrate. This is a field report on the engineering, written for the people who will actually have to build and operate this in production.

The headline finding, for an audience of architects and engineers, is simple: the model is the cheap part. The work is everything downstream of it, and it looks far more like site reliability engineering than like AI research.

The Model Stopped Being the Bottleneck

I run roughly 600 euros a month across frontier models plus the agent layer. Adding another agent is a rounding error. I route most traffic to one provider and fall back to another on a bad day. By month two, the model was picking up context, calling tools, summarizing documents, and writing code well enough that it was no longer the limiting factor for anything I shipped. Model choice became an implementation detail behind an interface, the way you would treat any swappable dependency.

That reframes the whole problem. If the model is a commodity capability behind a clean boundary, then the value and the risk both live in the system around it.

The Seven Concerns That Are Actually Hard

When I add a new agent, here is what I spend my time on. None of the frameworks I tried handle these to a production standard, so treat this as the reference architecture you build first.

1. Identity and least-privilege secrets. Every agent gets credentials scoped to exactly what it touches. If the email agent is compromised, the blast radius must stop there. This is standard service-account hygiene, applied per agent.

2. State management. Agents need durable, structured memory across days and weeks, not vector search over chat history. The real shape is operational state: the 18 active negotiations and their status, the document approved last Thursday, the record that changed on Monday. Building that was about 70 percent of the total work and broke most often. Model it like a domain database, not a transcript.

3. Write paths with safety. Read-only agents are toys. The moment an agent can mutate the outside world, you need idempotency keys, audit trails, rollback, and confirmation loops. These are the same primitives you would demand of any service that writes to production. The model does not give them to you.

4. Rate limiting and backoff against external tools. Not the model's rate limit. The downstream API's limit, including the 429 that gets returned as a body the model then happily parses as a success. Wrap every tool with explicit backoff and treat partial failures as first-class.

5. Retries with resumable state. If an agent fails on step 14 of a 20-step plan, it should resume, not restart. This is the durable-execution problem. A multi-step agent run is a workflow with checkpoints, not a single call. The pattern you want is closer to Temporal than to a request handler.

6. Observability with agent semantics. For every run, I capture the input, the model output, which tools were called, what they returned, and the final action. Stored, queryable, grep-able. Standard trace spans are not enough. You want to answer "which tool did the agent pick, and what did it expect to happen next?"

7. Escalation and kill switches. Every agent has an explicit "when to ask the human" rule and a channel to ask on. If I do not answer within two minutes, it stops and waits. And every agent has a literal off switch, not "redeploy after a fix."

Cost and Prompt Drift Are Operational Risks, Not Setup Tasks

Two failure modes deserve their own monitoring.

Cost does not creep; it spikes. An agent running at 2 euros a day becomes 40 euros a day because it started looping on a tool error and nobody noticed. I run a daily digest of tokens per agent per task. Without it, I would be staring at a five-figure bill. Budget caps per agent, per task, per day, with auto-throttle on runaway loops, belong in your platform layer.

Prompts rot. A prompt that was perfect at launch degrades over weeks as the model updates, the data shape shifts, or the task drifts into edges it never handled. I version every prompt and diff outputs on a sample set weekly. Treat prompts like code under test, or pay for treating them like stale documentation.

Input is hostile. A document arrived last month with a prompt injection buried in the line items, instructing my agent to forward a contract to an external address. A paranoid parser caught it. Treat every external input as untrusted user input, because it is.

The Build Order That Works

Do not start with a framework. Start with the smallest, most annoying daily task that has a clean input and a clean output. Write a 60-line script. Plug a model into it. Watch it fail. Fix the failures. Only after you have shipped three of those and run them for a month without intervention should you reach for abstractions. The teams I see succeeding work this way. The teams that picked a framework first and went looking for a task are mostly still searching.

The payoff is a compounding curve. Once the glue exists, each new agent is cheaper than the last. My first took three weeks. My fourth took two hours.

If you remember one thing: "agentic" is much closer to a well-instrumented, long-running background job than to anything futuristic. Build the SRE layer first. The prompt is the last 20 percent.

This piece is adapted from an essay first published at https://idofishman.ai/writing/agent-stack-six-months.

AI Site reliability engineering Production (computer science)

Published at DZone with permission of Ido Fishman. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Architecting Production AI Across Clouds: Patterns That Decide System Survival
  • Principles for Operating Large-Scale Global Production Systems with AI Innovation Across the Stack
  • Observability With AI SREs: A 4-Step Implementation Guide
  • Treating Web Scrapers Like Production Services

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook