Applying Microservices Lessons to Multi-Agent AI Systems

Imagine asking an AI assistant why a production API became slow after a deployment.

One agent examines metrics. Another searches logs. A third inspects deployment history. A coordinator combines their findings and proposes the next step.

The design looks straightforward until the logs service times out, two agents repeat the same expensive query, and the coordinator interprets a missing result as evidence that nothing went wrong.

Components exchanging messages. Dependencies with variable latency. Shared resources. Failures that affect only part of a request. We have seen all of this before.

Multi-agent AI systems inherit distributed systems problems wherever their execution crosses a network boundary. They also add variability in how tasks are interpreted, tools are selected, and conclusions are formed, so correctness testing must account for that variability alongside familiar failures. Microservices give us a vocabulary for reasoning about both, and for the costs of the decomposition that creates them.

A Role Is Not a Boundary

An agent describes behavior: it receives a task, uses a model and available tools, and may choose additional steps based on intermediate results. A microservice describes an architectural boundary around a capability, including its interface and independent deployment. Microsoft's microservices architecture guidance makes independent deployment a central property.

Several agents can run inside one process and use the same underlying model. An agent's role does not automatically give it a separate failure domain, scaling boundary, or security boundary. Those properties come from the implementation, not from the job description in the prompt.

The Distributed Monolith Test

Service architecture already has a name for what happens when that distinction is ignored. An agent system becomes a distributed monolith when its components communicate over a network but remain tightly coupled in their contracts, state, and release process.

The test is coupling, not shared infrastructure — independently deployable services routinely share a database cluster or a rate limit without ceasing to be independent. Three questions settle it:

• Does changing one agent force a coordinated release of the others?
• Do they read and write the same internal state?
• Can any of them be tested in isolation?

Splitting a prompt into five roles can create coordination overhead long before it creates any of that independence.

The Case for Splitting Anyway

There is a real argument for decomposition that has nothing to do with deployment: context isolation.

An agent investigating an incident can accumulate everything it reads. Two hundred kilobytes of log output ends up in the same context as the metric summary and the deployment diff, competing for attention and inviting the model to connect things that are adjacent only because they are adjacent in the prompt. Give the log search its own context, and it returns a bounded summary, and the coordinator reasons over three clean summaries rather than one crowded transcript.

The choice is not between one bloated context and several clean ones. A single agent can use retrieval, filtering, compaction, and external state to manage the same problem; Anthropic's context-engineering guidance discusses those approaches alongside subagents.

Isolation has its own cost. A summary can omit exactly the detail a cross-domain check needs — the log line whose timestamp would have connected a pattern to a deployment change is the kind of thing a summary drops. The coordinator needs a path back to the underlying evidence, which is what the evidence references below are for.

Separate contexts and focused evaluation can justify splitting when their benefits outweigh coordination costs. A different role description, on its own, cannot.

Mapping the Concerns

When agents run as separate services, the architectural mapping becomes concrete.

For the incident assistant, I would begin with three bounded capabilities: analyze metric changes, identify relevant log patterns, and reconstruct the deployment timeline. Each has a defined data source and an output the coordinator can inspect. If they need separate release cycles, access policies, or resource limits, service boundaries help — with versioned contracts and workflow evaluations to catch incompatible changes.

Routine operations such as fetching deployment records or calculating percentiles stay in ordinary code. An agent earns its place where the workflow needs interpretation: choosing which metric to investigate next, deciding whether a log pattern deserves attention, identifying a gap in the available evidence.

Contracts: What a Task Is, and What a Result Means

A task such as "find the problem" leaves too much implicit. A useful request carries the incident identifier, service, region, time window, available evidence, budget, and permitted tools.

{
  "task_id": "inv-4471-logs",
  "incident_id": "INC-4471",
  "service": "checkout-api",
  "region": "us-east-1",
  "window": { "start": "2026-09-14T18:40:00Z",
              "end":   "2026-09-14T19:10:00Z" },
  "budget_remaining_ms": 8000,
  "allowed_tools": ["logs.search", "logs.aggregate"],
  "prior_evidence": ["ev-4471-metrics-01"]
}

Two fields need their semantics stated rather than assumed.

Budget Propagation

budget_remaining_ms is the time this task may still consume. The engineering requirement is that the remaining budget survives queueing, execution, and downstream calls — not that it takes any particular representation. gRPC's deadline guidance propagates a remaining timeout by subtracting elapsed time at each hop, which handles queueing without depending on synchronized clocks; an absolute timestamp propagates unchanged but introduces clock skew between services. Either works if elapsed time is actually deducted.

Expiry is not simply "return what you have," because the caller may already have stopped waiting. Give workers a cutoff earlier than the caller's, so partial findings arrive with time left for synthesis, and have the runtime request cancellation of outstanding work when the budget runs out.

Declared Tools Are Not Enforced Tools

allowed_tools declares which tools this task may use. It does not enforce anything. The model can request any tool its prompt suggests, so authorization has to be enforced in the runtime or the receiving service. A context boundary is not an authorization boundary — the same distinction as the one between a role and a service boundary, appearing in a second place.

The Result Contract

{
  "task_id": "inv-4471-logs",
  "status": "incomplete",
  "coverage": {
    "searched": [{ "start": "2026-09-14T18:40:00Z",
                   "end":   "2026-09-14T18:52:00Z" }],
    "missing":  [{ "start": "2026-09-14T18:52:00Z",
                   "end":   "2026-09-14T19:10:00Z" }]
  },
  "findings": [
    {
      "claim": "upstream timeout rate rose from 0.2% to 4.1%",
      "evidence": ["ev-4471-logs-07"],
      "supports": "correlation_only"
    }
  ],
  "open_questions": ["did the elevated timeout rate continue past 18:52Z?"]
}

The coverage block is the part that matters. A logs agent that could not reach the second half of the window returns incomplete with the gap named, and that stays distinguishable from complete with an empty findings array. Collapse those two states and the coordinator converts absence of evidence into evidence of absence, which can hide the missing evidence from the final answer.

Schema validation is one layer of the contract. The coordinator also has to check whether the evidence supports the finding. A syntactically valid response can reference the wrong region, compare incompatible time windows, or claim that a deployment caused a regression merely because both happened close together.

Traditional services also produce incorrect results: a payment service can return successfully while calculating the wrong amount. The difference here is variability. The same input can yield a valid response that is wrong in a different way on each run, because interpretation and tool selection vary between executions. Testing has to cover interface compatibility and task outcomes, and the outcome tests need enough repetitions to characterize that variance rather than sample it once.

Latency: Measure the Whole Path

Consider one illustrative execution. Planning takes two seconds. The metrics, logs, and deployment investigations take four, six, and ten seconds respectively. Synthesis takes another three.

Those numbers assume unchanged branch durations and no additional overhead or contention.

For the fixed execution assumed here, parallel scheduling changes elapsed time while leaving the underlying calls unchanged — same prompts, same tools, same outputs, same token and quota consumption. Adaptive workflows change both the amount and the timing of work: a sequential run can use the metrics result to skip the deployment query entirely, and parallel branches can duplicate a query neither knew the other was issuing. Which arrangement costs less is an empirical question about a particular workflow, not a property of the schedule.

Synthesis grows with branch count either way. More branches means more material to reconcile before an answer exists.

Tails and Shared Bottlenecks

Dean and Barroso's The Tail at Scale describes the classic problem: when a request fans out, its latency is governed by the slowest branch, so responses that are rarely slow become requests that are commonly slow. With three independent branches each having a 1% chance of exceeding a threshold, the request exceeds it about 3% of the time. If their exceedances always occur together, it is 1%. These examples show why the dependence structure matters; neither alone predicts the production distribution.

What matters operationally is the shared bottleneck underneath. Agent branches commonly share a model endpoint, a rate limit, and an observability backend, and when that endpoint queues, each branch's individual probability of being slow rises. Hedging against the same saturated dependency can become ineffective or counterproductive, since its value depends on routing, spare capacity, and load controls that saturation removes.

So the quantity worth instrumenting is the distribution of complete task times. Per-agent latency can reveal the slowdown; dependency metrics and traces help identify its shared cause.

Waiting Policy and Capacity

The coordinator needs an explicit waiting policy. Which findings are required? Which are optional? When should it return a partial diagnosis? How much time should remain for synthesis and validation?

For our assistant, missing logs might still permit a useful report about a metric change, provided the report names the gap. A definitive causal diagnosis requires more. The completion rule should encode that difference in the runtime rather than leaving it to whatever the coordinator's prompt decides on a given run.

Capacity needs the same treatment, for a related reason: three independently deployed agents can still contend for one model quota and one observability backend, and adding replicas does little for throughput when the shared dependency is the constraint. Enforce concurrency limits and queue bounds in the runtime, allocate a budget for model and tool calls, and reserve time for the final response. Prompts can guide how an agent spends its allowance; the runtime has to enforce the allowance.

Otherwise, an assistant investigating an overloaded service becomes another source of load on the systems needed to diagnose it.

Retries and Idempotency

A timed-out request leaves uncertainty about what happened downstream. Suppose the assistant eventually gains permission to create an incident ticket. The ticket service creates the ticket, but its response is lost. A retry creates a duplicate unless the operation has idempotency handling.

AWS describes using caller-provided request identifiers to recognize repeated intent in Making retries safe with idempotent APIs. For this design, I would persist an operation identifier and the intended action before dispatch, and repeating that action preserves the identifier. The receiving service has to enforce the duplicate-handling contract; putting an identifier in a prompt guarantees nothing.

A retry of an action also differs from asking the agent to reconsider the task. Reconsideration may produce a different action with different parameters. Keeping the two distinct makes recovery tractable.

Durable State

Long investigations need durable state for similar reasons. The coordinator should record completed tasks, pending operations, evidence references, and remaining budgets, so that after a restart it can resume recorded progress and reconcile uncertain side effects. A checkpoint alone does not guarantee that an external action happened exactly once.

In our example, the coordinator owns investigation status and stores each agent's findings as separate evidence records. Every record identifies the service, region, observation window, and retrieval time. Where the underlying systems support snapshots or versions, those references are retained too.

This makes conflicting findings inspectable. Two agents may appear to disagree simply because one queried data before a recovery and the other queried afterward.

Prompt context then serves as a working view of durable task state. Large logs and artifacts stay in storage, with bounded excerpts and references supplied to the agents that need them.

There is production experience behind these concerns. In its account of building a multi-agent research system, Anthropic describes expensive restarts, the need for tracing, and bottlenecks caused by waiting for subagents. Those observations put recovery and coordination alongside model behavior rather than beneath it.

Validating the Outcome

A completed workflow still needs an outcome check.

Suppose all three agents report that a deployment caused the slowdown. That agreement deserves examination rather than confidence. They may have used the same timeline, inherited the same assumption from the coordinator, or treated correlation as causation. Consensus among agents drawing on shared context is not independent confirmation.

Validation should connect each claim to evidence and identify alternative explanations. A reported database slowdown points to the relevant measurements. A claim about a deployment explains what changed and what observation would strengthen or weaken the proposed connection.

Where the evidence supports only a hypothesis, the output preserves that uncertainty and proposes the next discriminating check. Giving a confident answer should never be the workflow's only definition of success.

Observability and What to Measure

The tracing has to support that evaluation. Trace from the incoming request through delegation, model calls, tool execution, and synthesis, capturing task relationships, timing, retries, token usage, model and prompt versions, and references to the evidence used. Sensitive content needs access controls and retention limits.

That lets us interrogate a poor result with concrete questions. Was the evidence unavailable? Did a tool query the wrong interval? Did the coordinator ignore an incomplete status? Did one slow dependency consume the budget reserved for verification?

That last denominator matters. Cheap failures should not make an architecture look efficient.

Compare Against the Boring Baseline

Evaluate the multi-agent design against a single agent with tools and, where practical, a conventional workflow, giving all three the same task set and evidence access. Comparisons either hold resource budgets constant or report the quality, latency, and cost tradeoffs explicitly; giving one architecture substantially more inference compute changes what the experiment demonstrates. Microsoft's agent orchestration guidance recommends starting with the lowest complexity that reliably meets the requirement, which is a good default here.

Conclusion

The distributed systems problems in a multi-agent architecture are not new, and the microservices vocabulary describes most of them accurately: boundaries, contracts, state ownership, failure isolation, capacity, and observability. What agent systems add is variability in interpretation, which means correctness has to be tested as an outcome rather than assumed from a passing schema check.

Start with one path through the problem. Split out a capability when separate context, permissions, ownership, or parallel execution offers a measurable advantage. Then test slow dependencies, incomplete evidence, duplicate deliveries, and recovery from interruption before increasing the number of agents.

For the next agent proposed in the design review, ask what it improves, what it costs, and how the system behaves when it fails.

Every agent should earn its place in the request path.

Key Takeaways

• An agent's role does not give it a failure domain, scaling boundary, or security boundary. Those properties come from the implementation, not from the job description in the prompt.
• An agent system becomes a distributed monolith when components communicate over a network but stay coupled in contracts, state, and release process. The test is coupling, not shared infrastructure.
• Task contracts need an explicit coverage field. "Searched and found nothing" and "could not search" must be distinguishable, or the coordinator will convert absence of evidence into evidence of absence.
• Parallel execution changes elapsed time, not the work performed, for a fixed execution. Adaptive workflows change both.
• Tool authorization has to be enforced at runtime or by the receiving service. Declaring permitted tools in a request documents intent; it does not enforce anything.