Prompt Caching for LLMs, Part 2: Semantic Caching in Production

Recap: Where This Fits

Part 1 of this series covered exact-match caching: hashing the full request and returning a stored response when a new request matches an old one exactly. That layer is fast and safe, but it only helps when the input matches exactly, which is common for templated traffic and rare in open conversation. 

This article covers the layer that closes that gap: semantic caching. Part 3 covers what happens once a request actually reaches the model: self-hosted KV cache reuse and provider prefix caching, plus the production details: watching your metrics, working out your costs, and a full checklist.

2. Semantic Caching

When a user asks the same thing in different words ("I am going from Bangalore to Mysore" vs. "I'm traveling Bangalore to Mysore and need hotels"), exact match fails. Semantic caching embeds the query and finds the nearest earlier response, and returns it if the two are similar enough.

How It Works

1. Embed the new query with a fast embedding model.
2. Search a vector store, such as Redis Stack with RedisVL, pgvector, FAISS, or Milvus, for the closest matches.
3. Use cosine similarity, the most common measure. Typical thresholds in production sit between 0.90 and 0.95.
4. On a hit, return the stored response. On a miss, call the model and store both the embedding and the response.

Trade-Offs

• Loss of detail. Embeddings compress meaning into numbers. "Laptop under $1000" and "laptop under $3000" can land close together in that number space, even though the right answer for each is different.
• Latency. A vector search, even a fast approximate one (HNSW), is slower than a simple hash lookup.
• Cost. Embedding a query is cheap and quick next to a full model call, but it is not free, and at high volume the embedding step becomes its own cost to plan for.
• Wrong answers. False positives are the main risk here. Reduce this risk with a stricter threshold, an embedding model tuned to your domain, a check after retrieval, or a mix of exact and semantic keys.

Picking an Embedding Model

The model you embed with has a direct effect on how many false positives you get, and just picking whatever is quickest to install is a common early mistake:

• General-purpose open models (sentence-transformers/all-MiniLM-L6-v2, all-mpnet-base-v2) are free, run on your own machines, and are fine for a first prototype. They struggle with domain-specific terms, product names, and short queries where one small word changes the meaning.
• Domain-tuned or retrieval-specific models (Redis's own redis/langcache-embed-v2, or a model tuned on your own past queries) usually tell near-miss pairs apart better, at the cost of an extra dependency, and, for hosted embedding APIs, a small per-call cost.
• Hosted embedding APIs (OpenAI, Cohere, Voyage) trade a small per-call cost for strong general performance, with no local GPU or model to manage. Most teams move to this once they are past the prototype stage.

Whichever you pick, test it against your own real traffic instead of trusting a leaderboard. General embedding quality and "tells apart this exact near-miss pair in my domain" are not the same test.

from redisvl.extensions.cache.llm import SemanticCache
from redisvl.utils.vectorize import HFTextVectorizer  # or OpenAIEmbeddings, etc.

vectorizer = HFTextVectorizer(model="sentence-transformers/all-MiniLM-L6-v2")  # or a stronger, domain-tuned model
cache = SemanticCache(
    name="llm_semantic",
    redis_url="redis://localhost:6379",
    distance_threshold=0.1,          # lower = stricter, this is cosine distance, not similarity
    vectorizer=vectorizer,
    ttl=3600
)

def cached_llm_call(prompt: str, llm_callable):
    hits = cache.check(prompt=prompt)
    if hits:
        return hits[0]["response"]   # consider a cheap check here before trusting this blindly

    response = llm_callable(prompt)
    cache.store(prompt=prompt, response=response)
    return response

Check the RedisVL docs for the current parameter names before you ship. Libraries like this change fast, and a name can shift between minor versions. One easy trap in the code above: distance_threshold is cosine distance, not cosine similarity, and the two run in opposite directions. As a rule of thumb, cosine distance is roughly 1 minus cosine similarity, so a similarity threshold of 0.92 is close to a distance threshold of 0.08. Reading distance_threshold=0.1 as "90% similar" instead of doing that conversion is a common way a threshold ends up looser or stricter than the team meant it to be. GPTCache follows the same embed, search, check, store pattern, and can sit in front of any OpenAI-compatible client if you would rather not build this loop yourself.

Shipping Semantic Caching Safely: Threshold Plus Verification

A raw similarity threshold, on its own, means "we tried semantic caching," not "we shipped it safely." The gap between those two is usually one cheap check placed between the vector hit and the return.

Three patterns cover most production systems:

1. High threshold, plus a cheap second check. Set the similarity bar high (0.94 to 0.97, or the matching distance value), then run the pair (new query, cached query) through a small, fast model or a rule-based check that answers one question: "do these two queries really want the same answer?" This costs a small fraction of a full model call and catches near-miss cases a threshold alone would let through.
2. A verified semantic cache. Store a short note next to the cached answer when you write it, not just the answer itself; for example, a one-line summary of when the cached answer applies ("valid for laptop budgets between $900 and $1,100"). When there is a candidate hit, check the new query against that stored note before returning it, instead of trusting similarity alone.
3. Fall back to a full call when confidence is low. Treat similarity as a sliding scale, not a strict yes or no, as the diagram above shows. Above a high threshold, return right away. In a middle band, run the check. Below that, treat it as a miss and call the model. This avoids the common mistake of picking one threshold and treating it as a perfect line, when it never really is.

def cached_llm_call_verified(prompt: str, llm_callable, verifier_callable):
    hits = cache.check(prompt=prompt)
    if hits:
        candidate = hits[0]
        # Cheap check before trusting a vector-space match.
        if verifier_callable(prompt, candidate["prompt"], candidate["response"]):
            return candidate["response"]
        # Similarity said "close enough," verification disagreed. Treat as a miss.

    response = llm_callable(prompt)
    cache.store(prompt=prompt, response=response)
    return response

verifier_callable can be as simple as a small classification model, a check against a second, differently-tuned embedding model, or a rule that checks specific details pulled out of both queries (price ranges, dates, entity names). The point is not to be fancy; it is to have a second, independent signal before a similarity score alone decides what a user sees.

Semantic Cache Invalidation Is Harder Than It Looks

TTL and LRU work well for exact matches, because a stale exact match is still at least a match for the same question. A stale semantic match is riskier: the cached answer can still look "close enough" to a new, slightly different question forever, because nothing about vector distance fades with time on its own. Practical fixes:

• Keep semantic TTLs shorter than exact-match TTLs by default, especially for anything tied to price, stock, or policy.
• Version the cache namespace whenever the underlying source of truth changes (a new product catalog, an updated policy page), instead of relying on TTL alone to catch it.
• Store a content hash or a last-updated time next to the cached response, and check it when reading, not just when writing.

Multi-Turn Conversations

Everything above treats a prompt as one standalone unit. Most production chat systems do not work that way. A conversation builds up history, and both the hashing and the embedding steps need to account for that:

• Exact match on multi-turn: Hash the full message history plus the new turn, not just the latest message. This means the exact-match hit rate on turn N of a conversation is close to zero unless the whole earlier conversation is replayed exactly, which rarely happens. Exact match is far more useful on the first turn of a session, or on isolated tool calls inside an agent loop, than on a deep conversation.
• Semantic match on multi-turn: Embedding only the latest user message loses the context that gives it meaning ("what about the second one" means nothing without the earlier turn). Common fixes are to embed a short summary of the running conversation state, or to embed the latest turn together with a summary of the relevant earlier turns. Neither one fully removes the risk described above, and that risk grows as the conversation gets longer.
• In practice, most teams apply exact and semantic caching to single tool calls and retrieval steps inside an agent loop, rather than to the full multi-turn exchange, and rely on model-side prefix reuse, covered in Part 3, for the conversation history itself.

Streaming Responses

If the model call streams tokens back to the client, the caching step cannot just wrap a normal, blocking call. Two common patterns:

1. Buffer, then cache. Stream to the client as normal, build up the full response on the server, and write to the cache only once the stream is done. Simple, but a request that gets canceled mid-stream never gets cached, and a very long response delays the cache write.
2. Cache and replay. On a cache hit, play back the stored response as a fake stream (in reasonable chunks) so the client's streaming UI still behaves as expected, instead of getting one big block of text. A UI built around token-by-token display looks broken if a cache hit suddenly returns the whole answer at once with no streaming pace.

Some setups go a step further and cache partial answers: for a very long response, write the first N tokens to the cache as they come in, instead of waiting for the full stream, then return those N tokens right away on a hit while the rest streams live. This extra work is worth it mainly for long content, like reports or long code files, where the wait for the first token matters more to users than the total time to finish.

Security Considerations

• Semantic cache poisoning. Because semantic caching returns a response based on similarity, not an exact match, an attacker who understands the embedding space can craft an input that lands close enough to a cached response that leaks sensitive information, gets around a guardrail, or returns a wrong but confident-sounding answer. High thresholds and a check after retrieval are the main defenses.
• One tenant seeing another tenant's data. In a multi-tenant app, a semantic cache without strict per-tenant filters can show one customer's cached response to a different customer's similar query. Filtering by tenant ID has to happen at the vector search step itself, not just in application code after the fact, or a bug in the app becomes a data leak instead of just a wrong answer. Part 3 covers the same risk at the model-serving layer.
• Personal data in cached responses. Any cached response that includes data tied to a specific person needs a retention and access policy at least as strict as the original response, since the cache is now a second copy of that data with its own TTL and its own access path.

Testing Semantic Thresholds Before You Ship

"Measure the false-positive rate" is easy to say and easy to skip. A simple process that works:

1. Pull a sample of real (or realistic) query pairs from your own domain, marked by a person as "should share a cached answer" or "should not," including some deliberately tricky near-misses (same topic, different detail, like the laptop price example above).
2. Run the sample through candidate embedding models at a few different thresholds, and check precision on the "should share" label. Optimize for precision here, not recall. A missed cache hit only costs latency. A wrong cache hit gives a wrong answer with full confidence.
3. Re-run this test whenever the embedding model changes, and check it now and then against live traffic, since how real users phrase things shifts over time in ways a fixed test set does not catch.

What's Next

Exact and semantic caching skip the model call entirely on a hit, which is why they sit in front of everything else. But most requests still reach the model, and what happens there — self-hosted KV cache reuse on engines like vLLM and SGLang, or provider-side prefix caching on Anthropic, OpenAI, and Bedrock — is often the single biggest cost saver on repetitive workloads. Part 3 covers both, along with a provider pricing comparison, a worked cost example, the metrics that actually catch problems, and a full production checklist.

Further Reading

• RedisVL documentation, semantic cache extension and vectorizers
• GPTCache project documentation