Prompt Caching for LLMs, Part 3: Model-Side Caching and Production Rollout

Recap: Where This Fits

Part 1 of this series covered exact-match caching, and Part 2 covered semantic caching. Both sit in front of the model and can skip the call entirely on a hit. This article covers what happens to the requests that still reach the model: reusing computed model state, either inside your own inference engine or inside the API provider's system, plus the production details that turn a caching plan into something a team can actually run and trust.

Layers 3 and 4 are two versions of the same basic idea, reusing computed model state, split by who runs the model server. If you run your own GPUs, layer 3 applies. If you call a hosted API, layer 4 applies. You rarely need both at once for the same model call.

3. Inference-Engine KV Cache Reuse (Self-Hosted Serving)

This layer gets left out of most write-ups because it lives below the application, inside the model-serving system itself, but it is where prefix caching really started, and it is the standard answer for any team running open-weight models on their own GPUs.

What Is Actually Being Cached

When a model processes a prompt, it computes a key and value tensor for every token, at every layer, during the first pass over the input. That stored state is what the model reads from while it writes its answer. If two requests share the first 3,000 tokens (a shared system prompt, tool list, or conversation history up to a point), redoing the work for those 3,000 tokens on every request is pure waste. Inference engines keep that state and reuse it right on the GPU. This still runs work for the new tokens; it just skips redoing the tokens it has already seen.

The Two Main Approaches

• vLLM, using PagedAttention. vLLM manages this stored state the way an operating system manages memory, in fixed-size pages handed out as needed. Its prefix caching reuses state at the page level whenever a new request starts with pages that match an earlier request exactly. Prefix caching is on by default on recent vLLM V1 engines. Turn it off with --no-enable-prefix-caching for testing. Older versions worked the other way, needing --enable-prefix-caching to turn it on, so check which one your deployed version uses.
• SGLang, using RadixAttention. SGLang keeps all its stored state in a tree structure built from token sequences, and finds the longest shared starting point for any new request, at any depth, even across branching conversations where several replies split off from a common point. It works by default at the token level. Turn it off with --disable-radix-cache if you want a no-cache baseline. Because the match works at the token level, not a fixed block size, SGLang tends to find more reuse on deep, branching, or unevenly-growing conversations, which is exactly the shape of most agent workflows.

The diagram above shows the core difference: vLLM's blocks reuse cleanly when requests share a prefix that lines up with block edges, while SGLang's tree finds the longest shared path at any point, which is why it tends to win on deeply branching agent sessions, where different requests split off at different points, not just at the very end.

A Minimal Setup

# vLLM: prefix caching is on by default in V1, shown here for clarity
vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --enable-prefix-caching \
    --port 8000

# SGLang: RadixAttention runs by default, no extra flag needed
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.3-70B-Instruct \
    --port 8000

Both give you an OpenAI-compatible /v1/chat/completions endpoint. Your application code, including the exact-match and semantic cache layers from Parts 1 and 2, does not change at all.

Where It Really Pays Off

• Reported hit rates of 60 to 85% on agent loops, multi-tenant apps, repo Q&A, and long-document RAG, wherever a large shared prefix repeats across many requests.
• Per-call cost drops of roughly 5 to 12x on a hit, since the first-pass work for the shared prefix is skipped entirely rather than just discounted.
• On prefix-heavy workloads, SGLang's RadixAttention has been shown to reach several times vLLM's throughput, though the gap narrows on unique, non-overlapping prompts.

Trade-Offs and Things to Watch

• This is a GPU memory problem as much as a caching problem. Stored state competes with model weights and the active batch for GPU memory, and past a few hundred thousand tokens of context, it can use more memory than the model weights themselves. State that does not fit gets removed, and the removal policy (vLLM roughly follows LRU at the block level, SGLang removes along the tree) decides what survives under pressure.
• Hit rate depends on the prefix staying the same, the same way provider prefix caching does. Reordering tool definitions, or adding a timestamp early in the prompt, breaks it just as easily here as on a hosted API.
• LoRA adapters, quantization, and speculative decoding all quietly become part of the cache key. Two requests with the exact same text but a different adapter ID or a different quantization level do not share cached state, because the underlying computed values are genuinely different. Fewer, more shared setups cache better than many narrow ones.
• Keeping tenants apart. Two tenants sharing one inference engine could, in theory, end up reusing each other's cached state. vLLM handles this with a cache_salt setting, a hidden value added to the block hash so only requests with the same salt can share cached blocks. This also closes a timing leak where an attacker could guess cached content just from how fast a response comes back. Set a per-tenant salt on any multi-tenant self-hosted setup.

4. Provider-Side Prompt/Prefix Caching

When a request has to reach the model because there was no exact or semantic hit, provider-side prefix caching reuses the expensive first-pass work on long, unchanging parts of the prompt: system prompts, tool lists, RAG documents, few-shot examples.

On the Claude API, a cache read costs roughly one-tenth of the normal input price, while cache writes cost more than a normal input token. Anthropic offers two TTL options, a 5-minute default and a longer 1-hour option, and moved the default down from 1 hour in early March 2026. Any setup built before that change, that does not explicitly set ttl on the cache_control block, will see its hit rate drop on workloads with gaps longer than 5 minutes. Check cache_read_input_tokens in your usage data to confirm you are still getting hits. The shortest prefix length that qualifies for caching depends on the model, generally a few thousand tokens, and cache breakpoints depend on order: put unchanging content first and changing content last, so the unchanging part forms a stable prefix. Reordering tool definitions is a common way to silently break the cache without any error showing up.

Provider Comparison at a Glance

The numbers below shift often and are reported inconsistently across vendor pages. Treat this as a starting point, not something to plan a budget on directly, and confirm every figure on the provider's live docs first.

Every provider now offers something in the 50 to 90% read-discount range, but the default behavior is quite different: an explicit marker with a short default TTL, a fully automatic system, or a split between an automatic and a developer-managed mode. Assuming one provider's defaults carry over to another is a common way to lose savings after a provider switch.

Keeping the Cache Warm

A session that pauses for 6 minutes while a person reviews a tool result pays full price on the next call, under Anthropic's 5-minute default. Two fixes: send a cheap keep-alive request just before the TTL would run out, if your traffic has gaps you can predict, or pay for the longer TTL option where gaps regularly run past the short default. The trade-off is simple: the extra write cost of the longer tier against the input-token cost of the misses it would otherwise cause. If a system prompt gets reused more than about 5 to 7 times within the window, the longer TTL usually wins out.

Best-of-Breed Production Scenarios

Cost and Latency Expectations

• Exact hit: basically free, usually under 10 ms.
• Semantic hit: an embedding call plus a vector search, tens of milliseconds, no model cost.
• Inference-engine KV cache hit: roughly 5 to 12x lower cost per call on the matched part of the prompt, with hit rates of 60 to 85% on agent and RAG workloads.
• Provider prefix hit: roughly a 90% cut on the cached part of the input tokens, though write cost and TTL choice affect how much of that saving you actually keep.
• Combined setups commonly report 50 to 90% lower overall cost, with templated traffic at the high end and open-ended chat at the low end.

Observability: What to Actually Watch

"Log hits, misses, and similarity scores" is correct, but too vague to act on:

• Hit rate for each layer. Exact, semantic, and inference-engine or provider cache hit rate (cache_read_input_tokens on Anthropic, the matching field elsewhere, or the engine's own stats on vLLM/SGLang), tracked apart from each other. A drop in one layer can easily hide behind stability in another.
• False-positive rate on semantic hits, checked on an ongoing basis. This is the metric most often skipped, and the one most likely to catch a wrong answer before a user complains about it.
• Stampede rate and wait time, from the request coalescing pattern in Part 1. A rising wait time under steady load is an early sign the model call itself is getting slower.
• GPU memory use for stored state, on self-hosted setups. A hit rate that drops under load is often explained by state being removed under memory pressure, not by a change in traffic.
• Cost per request, broken down by layer. This tells you which layer to spend more time on next.

If you can only add one thing first, make it hit rate per layer. Everything else is used to work out why that number moved.

A Worked Cost Example

Take a support chatbot with an 8,000-token system prompt plus tool definitions, a Claude Sonnet-class model at roughly $3 per million input tokens, and 1 million requests a month. Without caching, that prefix alone costs roughly $24,000 a month, before a single user token is even counted.

Exact and semantic caching catch 60% of requests as either identical or reworded repeats, which is common on a support FAQ. Those never reach the model. Of the remaining 400,000 requests, 80% of the prefix tokens hit the provider's prefix cache at roughly $0.30 per million instead of $3 per million. That works out to (400,000 x 8,000 x 0.8 x $0.30/M) + (400,000 x 8,000 x 0.2 x $3/M), which is about $2,688, against $9,600 without caching for that same slice of traffic.

Total prefix cost across all traffic lands around $2,688, against a $24,000 uncached baseline: about an 89% cut, matching the higher end of the range mentioned earlier. That number depends entirely on the 60% and 80% hit-rate guesses used here. A very repetitive FAQ can genuinely land here. An open-ended assistant will land closer to the 50% floor. Run this same math against your own measured hit rates from the metrics above, instead of assuming a headline number from a vendor blog matches your own traffic.

Implementation Checklist

• Clean up prompts before hashing. Include every setting that changes the output in the exact-match key.
• Pick the embedding model and distance measure on purpose, and test it against real traffic before trusting a threshold in production.
• Set TTLs and size limits on every cache layer, including semantic. Keep semantic TTLs shorter than exact-match TTLs by default.
• Log hits, misses, and similarity scores broken down by layer, not as one combined number.
• Guard against personal data and cross-tenant leaks at the vector search and KV cache steps directly, not just in app code.
• Handle repeated concurrent requests with request coalescing, instead of letting a stampede hit the model.
• Prefer built-in provider prefix caching, or your inference engine's default KV cache reuse if self-hosted, and set TTLs by hand.
• Keep application-level caching on top of model-side reuse. They solve different problems, and neither one replaces the other.
• If you self-host, pick vLLM or SGLang based on your workload (throughput on batches vs. deep multi-turn agent state), not just habit.
• Start with exact match. Add semantic caching only after you measure the rate of reworded queries and false positives on your own traffic.

Closing

The right way to think about this, across all three parts of this series, is to get the exact-match case right first, then add semantic matching on top while being honest about the trade-offs: some error, extra latency, and a harder invalidation problem. Under both of those, whatever reaches the model gains from reusing computed state, whether that happens inside your own inference engine or inside the provider's system. The pattern that works is layered: exact first, semantic second, then reuse on the model side for whatever is left, with careful thresholds, TTLs, and monitoring at every step.

None of this is a one-time job. Provider defaults change, traffic patterns shift, and a threshold tuned to last quarter's queries will not necessarily hold up this quarter. Build the exact-match layer today, watch the traffic you actually have, and let the measured rate of reworded queries and false positives tell you whether semantic caching's extra hit rate is worth the added work for your own workload.

Further Reading

• Anthropic, "Prompt caching," Claude Platform Docs
• OpenAI, "Prompt caching," OpenAI API Docs
• Google, Gemini API context caching documentation (implicit and explicit modes)
• Amazon Bedrock prompt caching documentation
• vLLM documentation, automatic prefix caching and PagedAttention
• SGLang documentation, RadixAttention