Treat AI Agents Like Long-Running Background Jobs: A Production Reference Architecture
Production AI agents behave like long-running background jobs: the model is the cheap part, and retries, idempotency, cost caps, and observability are the real work.
I spent six months replacing my own daily admin work with a small fleet of AI agents instead of writing about them. Three run in steady state; a fourth spins up on demand. Between them, they touch email, calendar, Slack, GitHub, a dozen SaaS APIs, a stack of PDFs, and a SQLite database I keep refusing to migrate. This is a field report on the engineering, written for the people who will actually have to build and operate this in production.
The headline finding, for an audience of architects and engineers, is simple: the model is the cheap part. The work is everything downstream of it, and it looks far more like site reliability engineering than like AI research.
The Model Stopped Being the Bottleneck
I run roughly 600 euros a month across frontier models plus the agent layer. Adding another agent is a rounding error. I route most traffic to one provider and fall back to another on a bad day. By month two, the model was picking up context, calling tools, summarizing documents, and writing code well enough that it was no longer the limiting factor for anything I shipped. Model choice became an implementation detail behind an interface, the way you would treat any swappable dependency.
That reframes the whole problem. If the model is a commodity capability behind a clean boundary, then the value and the risk both live in the system around it.
The Seven Concerns That Are Actually Hard
When I add a new agent, here is what I spend my time on. None of the frameworks I tried handle these to a production standard, so treat this as the reference architecture you build first.
1. Identity and least-privilege secrets. Every agent gets credentials scoped to exactly what it touches. If the email agent is compromised, the blast radius must stop there. This is standard service-account hygiene, applied per agent.
2. State management. Agents need durable, structured memory across days and weeks, not vector search over chat history. The real shape is operational state: the 18 active negotiations and their status, the document approved last Thursday, the record that changed on Monday. Building that was about 70 percent of the total work and broke most often. Model it like a domain database, not a transcript.
3. Write paths with safety. Read-only agents are toys. The moment an agent can mutate the outside world, you need idempotency keys, audit trails, rollback, and confirmation loops. These are the same primitives you would demand of any service that writes to production. The model does not give them to you.
4. Rate limiting and backoff against external tools. Not the model's rate limit. The downstream API's limit, including the 429 that gets returned as a body the model then happily parses as a success. Wrap every tool with explicit backoff and treat partial failures as first-class.
5. Retries with resumable state. If an agent fails on step 14 of a 20-step plan, it should resume, not restart. This is the durable-execution problem. A multi-step agent run is a workflow with checkpoints, not a single call. The pattern you want is closer to Temporal than to a request handler.
6. Observability with agent semantics. For every run, I capture the input, the model output, which tools were called, what they returned, and the final action. Stored, queryable, grep-able. Standard trace spans are not enough. You want to answer "which tool did the agent pick, and what did it expect to happen next?"
7. Escalation and kill switches. Every agent has an explicit "when to ask the human" rule and a channel to ask on. If I do not answer within two minutes, it stops and waits. And every agent has a literal off switch, not "redeploy after a fix."
Cost and Prompt Drift Are Operational Risks, Not Setup Tasks
Two failure modes deserve their own monitoring.
Cost does not creep; it spikes. An agent running at 2 euros a day becomes 40 euros a day because it started looping on a tool error and nobody noticed. I run a daily digest of tokens per agent per task. Without it, I would be staring at a five-figure bill. Budget caps per agent, per task, per day, with auto-throttle on runaway loops, belong in your platform layer.
Prompts rot. A prompt that was perfect at launch degrades over weeks as the model updates, the data shape shifts, or the task drifts into edges it never handled. I version every prompt and diff outputs on a sample set weekly. Treat prompts like code under test, or pay for treating them like stale documentation.
Input is hostile. A document arrived last month with a prompt injection buried in the line items, instructing my agent to forward a contract to an external address. A paranoid parser caught it. Treat every external input as untrusted user input, because it is.
The Build Order That Works
Do not start with a framework. Start with the smallest, most annoying daily task that has a clean input and a clean output. Write a 60-line script. Plug a model into it. Watch it fail. Fix the failures. Only after you have shipped three of those and run them for a month without intervention should you reach for abstractions. The teams I see succeeding work this way. The teams that picked a framework first and went looking for a task are mostly still searching.
The payoff is a compounding curve. Once the glue exists, each new agent is cheaper than the last. My first took three weeks. My fourth took two hours.
If you remember one thing: "agentic" is much closer to a well-instrumented, long-running background job than to anything futuristic. Build the SRE layer first. The prompt is the last 20 percent.
This piece is adapted from an essay first published at https://idofishman.ai/writing/agent-stack-six-months.
Published at DZone with permission of Ido Fishman. See the original article here.
Opinions expressed by DZone contributors are their own.