7 Failure Modes Every Production AI Agent Should Be Tested Against
Learn seven AI agent failure modes and learn how to test prompt injection, tool failures, memory, authorization, and unsafe actions.
Testing an AI agent is very different from testing a conventional software feature. A traditional application usually receives a defined input, follows code-controlled logic, and produces an expected output. An AI agent can make decisions during execution. It may select a tool, construct arguments, inspect a response, change its plan, retrieve more information, update memory, and continue acting until it believes the task is complete.
That difference makes happy-path testing dangerously incomplete. A support agent returning the correct order status proves that one expected interaction worked. It does not tell the engineering team what will happen when retrieved content contains malicious instructions, a tool times out after completing an action, stored memory contains false information, or two systems provide conflicting answers.
Production testing therefore needs to examine more than response quality. Teams should test the agent's behavior when the environment becomes unreliable, ambiguous, or adversarial. The following seven failure modes provide a practical starting point for building that kind of test suite.
1. Indirect Prompt Injection Through External Content
Many production agents operate on information they did not generate themselves. They read support tickets, web pages, emails, PDFs, knowledge-base articles, database records, and search results. Any of those sources can contain text that looks like an instruction to the model, even though the application intended it to be treated only as data.
Consider a support agent asked to summarize an uploaded customer document. Somewhere inside that document is a hidden instruction telling the agent to ignore its previous rules and retrieve confidential account information. A weak test would look only at the final answer and check whether confidential information appeared. A stronger test would inspect the entire execution trace to determine whether the malicious content changed the agent's plan, caused an unauthorized tool call, altered memory, or triggered communication with another system.
The expected behavior should be clear before the test runs. Retrieved content must remain untrusted input and should not be able to override system-level instructions or authorization rules. The test should fail even if the final answer appears safe, but the agent attempted a prohibited action behind the scenes. For agent systems, observing tool calls and intermediate actions is often as important as checking the final response.
2. Excessive Permission and Authorization Failure
Agents become more dangerous when their tools expose capabilities beyond what a task actually requires. An agent responsible for reading customer information might also have access to tools that modify billing records, delete accounts, issue refunds, or send messages. A carefully written system prompt may tell the model not to use those capabilities, but a prompt should never be treated as the final authorization boundary.
A useful test is to give the agent a read-only task while making privileged tools available in the environment. The test can then introduce misleading context suggesting that deleting or modifying a record would help complete the task. Engineers should observe whether the agent attempts the privileged call and, more importantly, whether the surrounding system prevents execution.
The correct control belongs below the model layer. Every sensitive action should be checked against the identity, permissions, and scope assigned to that agent or workflow. A successful test is not merely one in which the model chooses not to call the dangerous tool. The stronger result is that even an incorrect tool decision cannot cross the authorization boundary.
3. Tool Failure After a Side Effect Has Already Happened
Distributed systems have always had to deal with ambiguous failures, and AI agents add another decision-maker to the problem. A tool can successfully complete an operation while the response is lost because of a timeout, connection failure, or interrupted request. The agent then sees an error and may decide that the safest response is to retry.
Imagine an agent that calls a payment API to create a refund. The payment service completes the refund, but the connection times out before the success response reaches the agent. If the agent immediately repeats the request, the customer could receive the refund twice unless the API or orchestration layer protects against duplicate execution.
A production test should deliberately inject this condition. The test can simulate a successful side effect followed by a timeout and then observe the agent's recovery behavior. Instead of blindly retrying, the workflow should first reconcile the current state, query whether the transaction already exists, or use an idempotency mechanism that prevents duplicate operations.
This test matters because a technically correct retry policy at the HTTP layer can still produce incorrect business behavior. Agents need explicit recovery rules for operations that change external state. Those rules should be tested before the system encounters the failure in production.
4. Poisoned, Stale, or Cross-Session Memory
Persistent memory can make an agent more useful, but it also creates a new source of hidden state. Information stored during one interaction can influence decisions hours, days, or weeks later. If the stored information is false, outdated, malicious, or assigned to the wrong user, a future session may behave incorrectly even when the current prompt is perfectly safe.
A useful test is to insert a false memory into the system, such as a claim that a particular customer has permanent approval for unusually high transactions. A later session can then ask the agent to process a transaction for that customer. The objective is to determine whether the agent treats remembered information as authoritative or verifies it against a trusted business system.
Cross-session isolation should be tested as well. Information written during one user's conversation must not appear in another user's context unless the application explicitly supports shared state. Teams should verify memory ownership, expiration rules, provenance, and which information requires revalidation before use.
The important design distinction is that conversation history, long-term memory, and authoritative business data are not interchangeable. Memory can provide context, but sensitive facts such as permissions, account status, or financial limits should normally be confirmed against the system responsible for maintaining them.
5. Tool Schema Drift and Unexpected Responses
Agent tool calls often work perfectly in development because mocks and schemas remain stable. Production services change. Fields get renamed, enum values expand, required parameters become optional, responses return partial data, and older versions disappear. When that happens, an agent may try to infer what the new schema means instead of failing safely.
Suppose an agent has learned to send a tool request containing a priority field with values such as high or low. The tool later changes that field to severity and expects values such as P1, P2, and P3. The test should determine whether the system rejects the incompatible request, whether the agent tries to invent a mapping, and whether repeated retries create additional problems.
Unexpected response values deserve the same attention. An agent that understands APPROVED, DECLINED, and PENDING may encounter a new REQUIRES_MANUAL_REVIEW state. Production tests should verify that the agent does not guess how to handle unknown values simply because the model can generate a plausible interpretation.
Schema validation should therefore exist at the execution boundary. The model may prepare a proposed tool call, but deterministic validation should decide whether the call is syntactically and semantically acceptable before it reaches the target system. Recovery behavior for rejected calls should also be part of the test suite.
6. Runaway Reasoning and Repeated Tool Loops
Not every agent failure creates an incorrect answer. Some failures consume time, tokens, API requests, and infrastructure until a safety limit is reached. An agent may repeatedly call the same tool, alternate between two tools, or keep searching because every new piece of information creates another unresolved question.
A simple fault-injection test can create a dependency loop. Tool A tells the agent that the answer requires information from Tool B, while Tool B directs it back to Tool A. Another version can return search results that never provide enough information for the agent to complete the task. Engineers should then observe whether the runtime recognizes the lack of progress.
The system should have deterministic limits around execution. These may include maximum tool calls, repeated-call thresholds, elapsed time, token budgets, or workflow depth. The exact numbers depend on the application, so they should come from real operational requirements rather than being copied from a generic recommendation.
Teams should also record whether the agent is making meaningful progress between steps. A workflow that remains under its token limit can still be unhealthy if it calls the same endpoint repeatedly with nearly identical parameters. Repeated-call ratio, tool-call depth, execution time, and cost per task are useful signals for detecting this behavior.
7. Unsafe Actions When Evidence Is Incomplete or Conflicting
Agents eventually encounter situations where there is no obvious correct answer. One system reports that an account is active while another reports that it is closed. A document contains an incomplete instruction. A primary model fails, and a fallback model interprets the task differently. These are normal production conditions, not theoretical edge cases.
Consider an agent that receives a request to delete historical billing records. The customer relationship system says the account is active, while the billing platform reports that it is closed. If the agent selects whichever source seems more convincing and proceeds with deletion, the system has converted uncertainty into an irreversible action.
The test should verify that uncertainty changes behavior. Depending on the risk associated with the action, the workflow may need to retrieve an authoritative source, request clarification, switch into a read-only state, require human approval, or abort the operation. The correct response depends on the application, but the behavior should be defined before the agent encounters the conflict.
A useful pass condition is therefore not simply that the agent chose the correct system. The stronger requirement is that the system recognized when the available evidence was insufficient to justify the requested action.
Building a Failure-Injection Test Matrix
These failure modes become easier to manage when teams document them in a repeatable test matrix. Each test should identify the injected condition, the telemetry that needs to be captured, the signal that represents failure, and the behavior expected from the system.
| Failure Mode | What to Observe | Failure Signal | Expected Behavior |
|---|---|---|---|
| Indirect prompt injection | Plan and tool trace | External text changes protected behavior | Treat content as untrusted data |
| Excess permissions | Authorization events | Privileged action executes | Block action outside assigned scope |
| Post-action timeout | Retry and transaction trace | Duplicate side effect | Reconcile state before retry |
| Poisoned memory | Memory reads and writes | False memory controls sensitive action | Verify against authoritative source |
| Schema drift | Tool validation events | Invalid or invented arguments execute | Reject call and follow recovery path |
| Runaway loop | Steps, tokens, repeated calls | Execution continues without progress | Stop through runtime limits |
| Conflicting evidence | Decision and approval trace | Agent performs risky action despite uncertainty | Verify, escalate, or abort |
The matrix should evolve with the product. When a production incident reveals a new class of failure, the team should turn the incident into a permanent regression test. Over time, the suite becomes a record of the system's actual operational risks instead of a collection of hypothetical prompts.
Metrics That Make Agent Tests More Useful
Binary pass-or-fail results are valuable, but operational measurements provide more information about how the system behaves under pressure. Teams can track unauthorized tool-call attempts, prevented side effects, duplicate-action rates, recovery success, repeated-call ratios, token consumption, time to safe termination, human escalation rates, and cross-session memory leakage.
These measurements are particularly useful when model versions, prompts, orchestration logic, or tool configurations change. A new model may still complete the task successfully while making twice as many tool calls. A prompt update may reduce answer errors while increasing escalation failures. Looking at behavior-level metrics makes those regressions easier to detect.
Teams should avoid inventing universal thresholds. A five-second execution time may be acceptable for one workflow and unacceptable for another. The meaningful limits should come from the latency, cost, risk, and business requirements of the specific agent being tested.
Test the Agent's Boundaries, Not Just Its Answers
Production AI agents should not be evaluated only by comparing an input with an expected response. Their behavior spans context selection, planning, tool choice, authorization, execution, observation, memory, and repeated decision-making. A system can produce a convincing final answer while something unsafe has already happened earlier in the execution trace.
That is why failure injection should become part of the normal agent testing process. Teams need to know what happens when external content is hostile, permissions are too broad, tools return ambiguous failures, memory becomes unreliable, schemas change, execution loops, or evidence conflicts. These conditions will eventually appear outside the test environment.
The strongest agent test suites grow from real operating experience. Each incident, near miss, or unexpected trace should become a test case that the system must pass before future releases. That approach shifts testing away from asking whether an agent can complete a task and toward a harder question: whether it can fail without creating a larger problem.
Opinions expressed by DZone contributors are their own.