I Built a RAG Agent on Azure AI Foundry in an Afternoon. Here's What Nobody Tells You.

Six months ago, building a RAG pipeline meant a full week of plumbing: an embedding job here, a vector store there, a retriever glued on with duct tape, and an orchestration layer that broke every time you touched it. I've built enough of these the hard way — hand-rolled vector search, custom chunking scripts, the works — to know exactly how much pain that "week" usually hides.

Last week, I rebuilt the same thing on Azure AI Foundry. It took an afternoon. Not because the underlying problem got easier — grounding an LLM in your own data is still genuinely hard — but because Microsoft finally killed most of the integration tax that used to eat the first sprint of every RAG project.

Here's what actually happened, warts included.

The Old Way Was a Trap

If you've built RAG before, you know the pattern: you don't fail at RAG, you fail at the seams between the pieces. Your chunking strategy doesn't match your embedding model's context window. Your retriever returns great results in a notebook and garbage in production because nobody wired up hybrid search. Your "agent" is really just a for-loop that stuffs retrieved text into a prompt and hopes.

Foundry's whole pitch is that it owns those seams instead of leaving them to you. I was skeptical. I'm less skeptical now.

What I Actually Did

Step one: spin up a Foundry project. Not a hub-based one — those are legacy at this point, and if a tutorial has you creating one, skip it. The newer Foundry project type is the one to use.

Step two: deploy two models. A chat model and an embedding model. Click, click, done. Both show up with their own endpoints. This part genuinely takes five minutes, and it's the first sign you're not building infrastructure anymore — you're configuring it.

Step three: point Foundry at my documents. Blob storage in, Azure AI Search out. Foundry handles the chunking and embedding generation itself. I turned on hybrid search (keyword plus vector) because pure vector search on enterprise docs tends to miss exact terms people actually search for — product names, error codes, that sort of thing. If your content has a lot of that, don't skip this.

Step four — and this is the part that's different from every tutorial I read two years ago. I didn't write a retrieval pipeline. I registered the search index as a tool on the agent and let the agent decide when to call it. Here's the whole thing:

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient.from_connection_string(
    credential=DefaultAzureCredential(),
    conn_str=os.environ["AIPROJECT_CONNECTION_STRING"],
)

agent = project.agents.create_agent(
    model="gpt-4o-mini",
    name="docs-assistant",
    instructions=(
        "Answer only using retrieved context. "
        "Cite the source document for every claim. "
        "If the answer isn't in the retrieved content, say so."
    ),
    tools=[{
        "type": "azure_ai_search",
        "index_connection_id": search_connection_id,
        "index_name": "example-index",
    }],
)

thread = project.agents.create_thread()
project.agents.create_message(thread.id, role="user", content="What's our refund policy for enterprise plans?")
run = project.agents.create_and_process_run(thread.id, agent.id)

No manual embedding calls at query time. No hand-written "retrieve top-k, stuff into prompt" logic. The agent framework does that internally, and it does it well enough that I stopped fighting it after the first try.

Step five: For anything beyond simple lookups, I turned on agentic retrieval in Azure AI Search. Classic RAG fires one query per user turn, which quietly falls apart the moment someone asks a compound question — "compare our Q3 and Q4 policy and tell me what changed for renewals" is two questions wearing a trench coat. Agentic retrieval breaks that into sub-queries, runs them in parallel, and merges the results before generation. If your users ask messy, multi-part questions — and they do — turn this on from day one. Retrofitting it later is more annoying than it should be.

Step six: Tested in the playground, then deployed the same agent behind a REST endpoint. Nothing about the agent changed between prototype and production. That alone would've saved me a full day on past projects.

Now, the Part Everyone Skips

I'm not going to pretend this is magic, because it isn't, and the tutorials that pretend otherwise are setting people up to get burned in a security review.

Access control is on you. Foundry doesn't look at your documents and infer that HR files shouldn't be visible to the sales team. You configure document-level security filters in Azure AI Search yourself, and if you skip this, you've built a very articulate way to leak sensitive data.

API keys are a prototype crutch, not a production plan. Move to Microsoft Entra ID before anything customer-facing goes live. This migration is a real afternoon of work, not a checkbox — budget for it.

Retrieved documents are untrusted input. Prompt injection through a poisoned PDF is a real attack surface in every RAG system, Foundry included. Your system instructions need to assume the retrieved content might be trying to manipulate the model, because eventually it will.

The costs stack. Embedding generation, index storage, and the extra tokens from stuffing retrieved passages into every call — none of this is free, and it compounds faster than people expect once you're past a demo and into real traffic. Model it before you commit to a chunking strategy at scale, not after.

Was It Actually Worth It?

Yes — but not for the reason most "look how easy this is" posts claim. The value isn't that RAG got simple. Grounding a model in the right data, with the right access controls, still takes real thought. The value is that Foundry took the boring week — the SDK wrangling, the manual retrieval loops, the glue code nobody wants to own — and turned it into an afternoon of configuration. That frees up the time you actually need for the parts that matter: is your data any good, is it chunked sensibly, and can you trust what comes back?

If you've been putting off a RAG project because the infrastructure felt like too much, this is the moment to try again. Just don't skip the access control step to save time. That's the part that actually bites.