Context Rot: Why Your AI Agent Gets Worse the Longer It Works

AI-powered features often behave perfectly during testing and quietly degrade in production. The model has not changed. The prompts have not changed. Latency looks normal. Error rates are clean. Yet the responses gradually feel off, slightly disconnected, missing nuance, referencing things that are no longer relevant to the task at hand.

This pattern has a name: context rot. It does not throw exceptions. It does not appear in dashboards. It is one of the more subtle failure modes in production AI systems, and understanding it early makes a meaningful difference in the quality of what gets built.

How Attention Works in LLMs

To understand context rot, just enough of the underlying mechanic is needed. Before an LLM generates each new token, it looks at every token in the context and decides how much weight to give each one. This is called attention.

The key insight: attention scores are normalized, and they sum to 1.0 across all tokens. That means attention is a fixed budget. When the context has 500 tokens, each important piece of information might receive 0.15 or 0.20 of the total attention. When the context has 50,000 tokens, that same important piece might receive only 0.002, even if it is equally critical to the task.

// Simplified illustration — not actual LLM code

public float[] generateNextToken(String[] contextTokens) {
    float[] scores = new float[contextTokens.length];

    for (int i = 0; i < contextTokens.length; i++) {
        // How relevant is each past token to what we are generating?
        scores[i] = computeRelevance(currentState, contextTokens[i]);
    }

    // Scores must sum to 1.0 — a fixed attention budget
    float[] weights = softmax(scores);

    return weightedCombination(contextTokens, weights);
}

Every token added, relevant or not, slightly dilutes the attention available for everything else. That is the seed of context rot.

Context Position and Attention

A well-known multi-document question-answering experiment revealed something that should give every engineer building AI systems reason to pause. The correct answer was hidden at different positions across a long context, and retrieval accuracy was measured purely by position:

• Answer at the beginning: ~75% accuracy
• Answer at the end: ~72% accuracy
• Answer in the middle: ~55% accuracy

A 20 percentage point drop caused entirely by where the information sat, not by its quality or relevance. The information was present. The model could technically see it. It simply was not attending to it properly.

This is known as the Lost-in-the-Middle effect. It is an emergent architectural property of the transformer training process itself. Models learn to attend strongly to the beginning and end of their inputs, where the most signal-dense content tends to appear in human writing. The middle of a long context becomes an attention dead zone as a natural consequence of how these models are trained, not as an oversight.

Does this still apply to modern models? The honest answer is: yes, with important nuance. Newer models have largely resolved the effect for simple factoid retrieval — finding a specific fact at a specific position in a long context is something recent architectures handle well. The problem persists, and arguably intensifies, on multi-step reasoning tasks where the model must synthesize information across several documents simultaneously. That is precisely the category most production AI systems fall into, so the practical risk remains significant even as benchmark numbers improve.

What Context Rot Looks Like in Practice

Scenario 1: The wandering coding agent. An agent is asked to fix a bug. It reads 15 files, explores 3 wrong leads, and backtracks. Each file, each search result, each dead end accumulates in context. By the time the agent finds the right file, buried in the middle of 20,000 tokens, attention is spread thin. The analysis of the one file that actually matters is noticeably weaker than it would have been with a clean context.

Scenario 2: The RAG pipeline that drifts. A retrieval pipeline fetches 10 document chunks per query, roughly 5,000 tokens. For most queries, this works fine. But longer queries trigger larger system prompts and conversation history. Total context grows to 40,000 tokens, and the documents retrieved third and fourth, sitting in the middle, fall into the attention dead zone. The model answers confidently, drawing on what it can see well. A crucial nuance from chunk 4 gets missed.

The pattern is always the same: no error, no warning, just answers that are subtly less accurate than they should be.

How to Detect It

Step 1: Log context length alongside every LLM call. What cannot be measured cannot be managed.

Step 2: Run a positional accuracy test. Place a key fact at different positions in a realistic context and check whether the model retrieves it correctly.

public void positionalAccuracyTest(LlmClient client, String keyFact, String fillerText) {
    double[] positions = {0.1, 0.5, 0.9}; // beginning, middle, end

    for (double pos : positions) {
        int split = (int) (fillerText.length() * pos);
        String context = fillerText.substring(0, split)
            + "\nKEY: " + keyFact + "\n"
            + fillerText.substring(split);

        String response = client.complete(context, "Summarise the most important information from the context.");
        boolean found = response.toLowerCase().contains(keyFact.toLowerCase());

        System.out.printf("Position %d%%: %s%n",
            (int)(pos * 100), found ? "RECALLED" : "MISSED");
    }
}

If the model passes at 10% and 90% but fails at 50%, context rot is measurable in that system at that context length.

Step 3: Alert on context length thresholds. Set a warning at around 50,000 tokens and a hard alert at 100,000. These are starting points — the positional accuracy test above will help calibrate the right numbers for a specific model and task type.

Context Rot Is Also a Cost Problem

Most conversations about context rot focus on quality, and rightly so. But at any meaningful scale, it is equally a financial problem, and that dimension tends to get overlooked until the infrastructure bill arrives.

LLM providers charge by the token. Every token in the context window is billed on every single call. A context that has grown to 80,000 tokens costs roughly 8x more per call than one held at 10,000 tokens, for the same task, often with worse output quality. That is not a trade-off; it is strictly worse in both dimensions simultaneously. The exact cost per token varies by provider and model tier, but the ratio holds universally — longer context means a proportionally larger bill.

The compute reality makes this more pronounced. Transformer attention scales quadratically with context length. Doubling the number of tokens does not double the compute required; it roughly quadruples it. At low volumes, this is invisible. With millions of calls per day, it becomes one of the largest line items in an AI system's operating cost.

The numbers are illustrative, but the ratio is the point. A context that has grown to 80,000 tokens costs roughly 8x more per call than one held at 10,000 tokens, for the same task, often with worse output quality. That is not a trade-off; it is strictly worse in both dimensions simultaneously. Context rot at scale is not a quality inconvenience. It is a budget problem. Compaction, precise retrieval, and subagent isolation are not just engineering best practices; they are cost controls.

4 Practical Mitigations

1. Compact early — do not wait until quality degrades. Summarize older conversation turns before the context gets large, not after the damage is done.

public List<Message> compactIfNeeded(List<Message> messages, LlmClient client) {
    int limit = 30_000;
    if (estimateTokens(messages) < limit) return messages;

    // Need at least a system prompt + messages to summarise + recent turns
    if (messages.size() < 7) return messages;

    // Everything except system prompt and last 5 turns
    List<Message> older = messages.subList(1, messages.size() - 5);
    String summary = client.complete("Summarise concisely: " + format(older));

    List<Message> compacted = new ArrayList<>();
    compacted.add(messages.get(0)); // system prompt
    compacted.add(new Message("system", summary));
    compacted.addAll(messages.subList(messages.size() - 5, messages.size()));
    return compacted;
}

2. Use subagents for exploration. When an agent needs to search or explore, do it in a dedicated subagent with its own context window. Only the compact result, not the exploration trace, returns to the parent agent. Noise stays isolated.

3. In RAG, retrieve less and rerank. Three precisely relevant chunks consistently outperform ten loosely relevant ones. Retrieval quantity does not equal retrieval quality. Fetch a wider candidate set, rerank by relevance, and pass only the top results to the model.

4. Position critical content deliberately. Given what is known about the attention curve, the most important context belongs at the beginning or end, not sandwiched in the middle. The system prompt and the current user query naturally occupy those positions. Keep them there, and be intentional about what fills the space between.

What This Means at Each Level

For early-career engineers: when an AI feature works in local testing but feels off in production, check context length first. Adding llm.context_tokens to an observability stack, alongside latency and error rate, is a small change with a meaningful signal.

For tech leads and architects: context is not a free resource. Every design session for an LLM-powered feature should include a clear answer to "what is in this context window and why?" If that question cannot be answered clearly, the design is incomplete.

For engineering managers and leaders: context rot does not appear in standard dashboards. Error rate and latency can look perfectly healthy while response quality silently degrades. Correlating context length with downstream quality metrics, task success rates, and user satisfaction is the monitoring work that production AI systems now require.

Conclusion

Context rot is one of those concepts that feels advanced until it is encountered in production, and then it feels like something that should have been understood from day one.

The core reality is simple: transformer attention is a finite, dilutable resource. Every token added to a context window reduces the focus available for everything else. When contexts grow long, and important information ends up in the middle, quality degrades in ways that are real, measurable, and unfortunately silent.

The good news is that it is manageable. Compact early. Isolate exploration into subagents. Be precise with retrieval. Position critical content deliberately. None of these requires advanced machine learning knowledge; they are engineering disciplines applied to a new kind of resource.

The mental model that tends to help most is treating context the way experienced engineers treat memory: allocate it deliberately, release what is no longer needed, and keep the working set small and focused. The models are already capable of doing remarkable work, if given a clean signal and kept free of noise.