RAG Architecture Patterns: Proxy, Caching, and Routing

A few months ago, our RAG (Retrieval-Augmented Generation) setup did what most first versions do: the Streamlit app called OpenAI directly, pulled chunks from MongoDB, and stitched an answer together. It worked. It also had three problems nobody wants to discover in production.

First, the OpenAI bill. Users ask the same question in ten different ways: "How many leave days do I get?" and "What's the annual leave entitlement?" are the same question, but if you're calling the LLM for every single one of them, you're paying for the same answer over and over.

Second, if OpenAI had a slow minute, every single user felt it. Nothing was cached anywhere, so every request, no matter how routine, went all the way to the model.

Third, and honestly the scarier one: an LLM will happily answer a question even when the answer isn't in your documents. It doesn't say "I don't know"; it just generates something plausible. For an internal knowledge assistant, that's not a bug you can shrug off — that's a wrong HR policy answer someone acts on.

So we built a gateway. Nothing exotic — a single FastAPI service that sits between the UI and everything else (OpenAI, Gemini, MongoDB). The client never talks to those systems directly anymore; it only ever talks to the gateway, and the gateway decides what happens next.

Why a Gateway and Not Just “Add Caching Somewhere”

The tempting shortcut is to bolt caching onto the existing call site and move on. We didn't do that, mainly because caching, grounding, routing, rate limiting, and logging all need to see the same request at the same point in its lifecycle. Splitting that logic across the UI and a dozen helper functions means every future change touches five files. Put it all in one place instead, and it's one file you read to understand the entire request path.

Nothing here is framework magic. No LangChain, no LlamaIndex. It's the OpenAI SDK and MongoDB's native $vectorSearch called directly. That was a deliberate choice — when something behaves oddly, we want to be able to step through our own code, not a framework's abstraction layer.

What Actually Happens on a /Query Call

This is the part that matters most, so here's the full path a question takes:

Each of those boxes guards against one of the three problems above. Rate limiting protects against abuse. The cache lookup protects against paying twice for the same question. The grounding gate protects against confidently wrong answers. The budget check protects the invoice. And the last step means none of this is a black box — every request is logged with its cache status, provider, tokens, and cost, and shows up live in the Analytics tab.

Semantic Caching: Matching What People Mean, Not What They Typed

Here's the part people usually find counterintuitive. We don't do exact-string caching. We embed the incoming query the same way we embed document chunks, and compare it against previously answered queries stored in their own cache collection, using cosine similarity. If the match is above 90%, we return the stored answer straight away — no LLM call at all.

What that looks like in practice:

• "How many annual leave days?" -- miss (first time anyone's asked)
• Same question again -- hit (exact repeat)
• "What's the yearly vacation entitlement...?" -- hit (different words, same question)
• "Meal allowance during travel?" -- miss (genuinely different topic)

That third line is the whole point. An exact-match cache would have missed it and burned an LLM call for a question we'd already answered five minutes earlier. Cosine similarity catches the paraphrase.

We also cache at ingestion time, which is easy to overlook but saves real money. Before embedding a document chunk, we hash it. If that hash already exists, we skip it — no embedding call, no reprocessing. Re-uploading a document you already ingested costs nothing. Only the chunks that actually changed get new embeddings.

Making Sure the Model Doesn’t Just Make Something Up

This is the guardrail we care about most, and it's deliberately built as two separate layers rather than one clever one:

1. A confidence gate before generation. We retrieve the top-k chunks and check the best similarity score. If it's below our grounding threshold, we never even call the LLM — we return a "not in the documents" response instantly. Cheap, fast, and it can't hallucinate because it never ran.
2. A strict system prompt during generation. Even when the gate passes, the model is explicitly instructed to refuse if the retrieved context doesn't actually contain the answer.

Neither layer alone is bulletproof. Together, they catch most of the ways an LLM can wander off and answer a question you never asked it to answer with information it invented.

Routing on Cost, Not Just on Uptime

Our primary model is gpt-4o-mini. The gateway keeps a running total of what we've spent since the last budget reset, and once that total hits the configured limit, new requests get routed to Gemini 2.5 Flash instead — called through OpenRouter's OpenAI-compatible API, so it's the same SDK, just a different base URL and model name. No separate integration to maintain.

The part worth calling out: the budget number is a runtime setting, not a constant baked into the code. Whoever owns the budget can change it without anyone redeploying anything. And if the fallback provider is ever misconfigured, we fail back to the primary rather than failing the request outright.

What We Get to See

None of this is useful if it's invisible. Every request - cache status, grounding result, provider used, tokens, cost, latency — gets logged, and it's all visible in the Analytics screen. There's also a Routing screen that shows the currently active provider alongside spend versus budget, in real time. Rate limiting runs on a 60-second sliding window per client IP, so one runaway script can't drown out everyone else.

Where That Leaves Us

None of this required a heavyweight framework — it's FastAPI, MongoDB Atlas, and the OpenAI SDK, wired together deliberately. The bigger lesson for us wasn't any single pattern; it was that caching, grounding, and routing all want to live at the same layer, because they're all answering the same underlying question: Should this request even reach the model, and if so, which one?

If you're running a RAG app that talks to the LLM directly today, that question is worth asking before the bill or a bad answer asks it for you.