Building Agentic RAG, Step by Step: From Static Retrieval to Reasoning Pipelines

Retrieval-augmented generation solved a real problem: it grounded LLM outputs in facts the model was never trained on. But classic RAG has a ceiling. It retrieves once, stuffs the results into a prompt, and hopes the top-k chunks happen to contain the answer. There's no self-correction, no multi-step reasoning, and no way to recover when the first retrieval misses.

Agentic RAG removes that ceiling by putting an LLM-driven agent in the loop — deciding what to retrieve, when to retrieve again, whether the retrieved context is actually good enough, and how to combine multiple sources before answering. This article walks through building one from scratch, step by step, with working code you can adapt to your own stack.

Why "Agentic" Changes the Architecture

In naive RAG, the flow is linear:

Query → Embed → Vector Search → Stuff Context → Generate Answer

In agentic RAG, retrieval becomes a tool the agent chooses to call, possibly more than once, possibly against more than one source, with a reasoning step wrapped around every hop:

Query → Agent Plans → Calls Retrieval Tool(s) → Grades Results →
  (Insufficient? → Reformulate → Retrieve Again) →
  Sufficient? → Synthesize → Self-Check → Answer

That loop is the entire value proposition. It costs more tokens and more latency per query, but it converts a system that silently fails on hard questions into one that visibly tries harder before giving up.

Step 1: Define the Tools, Not Just the Index

The first mistake teams make when porting from classic RAG is treating the vector store as the only retrieval surface. An agent needs a toolbox, and the toolbox should reflect the actual shapes of knowledge in your domain:

• Vector search tool – semantic similarity over unstructured docs
• Keyword/BM25 tool – exact term and code/identifier matches vector search misses
• Structured query tool – SQL or API calls against systems of record
• Web search tool – for anything outside your corpus, if permitted

Each tool gets a clear name, a docstring the agent can reason over, and a narrow, single-purpose contract. This is also exactly where Model Context Protocol (MCP) earns its keep — it standardizes how these tools are described and invoked, so the same retrieval tool can be reused across agents and orchestration frameworks instead of being reimplemented per project.

from typing import List, Dict

def vector_search(query: str, top_k: int = 5) -> List[Dict]:
    """Semantic search over the document vector store. Best for
    conceptual questions, paraphrased queries, and 'how does X work'
    style requests."""
    embedding = embed(query)
    return vector_db.query(embedding, top_k=top_k)

def keyword_search(query: str, top_k: int = 5) -> List[Dict]:
    """Exact/BM25 search. Best for error codes, identifiers, config
    keys, and anything where wording must match verbatim."""
    return bm25_index.search(query, top_k=top_k)

def sql_lookup(question: str) -> Dict:
    """Structured lookup against systems of record (claims status,
    account data, ticket state). Use when the question asks for a
    current, specific fact rather than an explanation."""
    query = nl_to_sql(question)
    return db.execute(query)

Step 2: Give the Agent a Retrieval Plan, Not Just Tool Access

Handing an LLM a list of tools and hoping it calls them well is how you get expensive, undisciplined agents. Instead, prompt for an explicit plan before any tool call happens:

PLANNER_PROMPT = """
You are a retrieval planner. Given the user's question, decide:
1. What sub-questions need to be answered
2. Which tool(s) best fit each sub-question
3. Whether this requires one retrieval pass or several sequential ones

Return a JSON plan:
{
  "sub_questions": [...],
  "tool_calls": [{"tool": "...", "query": "..."}],
  "requires_iteration": true/false
}
"""

This planning step is where agentic RAG earns its name — the system is reasoning about the retrieval strategy itself, not just executing a fixed pipeline. For a multi-part question ("compare our Q3 claims volume to Q2 and explain the driver"), the plan might route one sub-question to sql_lookup and another to vector_search, then merge both before generating.

Step 3: Retrieve, Then Grade Before You Generate

This is the step classic RAG skips entirely, and it's the single highest-leverage addition you can make. After retrieval, insert a grading pass that checks relevance before the results ever reach the generation prompt:

GRADER_PROMPT = """
Question: {question}
Retrieved chunk: {chunk}

Is this chunk relevant and sufficient to help answer the question?
Answer strictly: RELEVANT, PARTIALLY_RELEVANT, or IRRELEVANT.
"""

def grade_chunks(question: str, chunks: List[Dict]) -> List[Dict]:
    graded = []
    for chunk in chunks:
        verdict = llm_call(GRADER_PROMPT.format(
            question=question, chunk=chunk["text"]
        ))
        graded.append({**chunk, "grade": verdict})
    return [c for c in graded if c["grade"] != "IRRELEVANT"]

If everything comes back IRRELEVANT, that's a signal, not a dead end — it routes back into Step 4.

Step 4: Reformulate and Retry on Weak Retrieval

When grading fails to produce enough relevant context, the agent should rewrite the query rather than silently generating from thin evidence:

def agentic_retrieve(question: str, max_attempts: int = 3) -> List[Dict]:
    query = question
    for attempt in range(max_attempts):
        raw_results = vector_search(query)
        good_results = grade_chunks(question, raw_results)

        if good_results:
            return good_results

        query = llm_call(
            f"The search for '{query}' returned nothing useful for "
            f"the question '{question}'. Rewrite the search query to "
            f"use different terms or a narrower/broader scope."
        )
    return []  # exhausted attempts — surface this honestly downstream

This is the difference between a system that degrades gracefully and one that hallucinates confidently. Three attempts is a reasonable default; tune it against your latency budget.

Step 5: Synthesize Across Sources, Not Just Chunks

Once you have graded, relevant context — possibly from more than one tool — the generation prompt should make the agent explicitly reconcile sources rather than concatenate them:

SYNTHESIS_PROMPT = """
Question: {question}

You have retrieved information from multiple sources. Synthesize an
answer that:
- Cites which source supports each claim
- Flags any contradictions between sources explicitly
- States clearly if the retrieved context is insufficient, rather
  than filling gaps with unsupported assumptions

Sources:
{sources}
"""

Explicitly asking the model to flag contradictions and insufficiency here reduces silent hallucination more than almost any other prompt-engineering change in the pipeline.

Step 6: Self-Check Before Returning an Answer

A final verification pass — cheap relative to the rest of the pipeline — catches cases where the synthesis drifted from the retrieved evidence:

VERIFY_PROMPT = """
Answer: {answer}
Source context used: {sources}

Does every factual claim in the answer trace back to the source
context? List any claim that does not. If all claims are supported,
say VERIFIED.
"""

If verification fails, route back to Step 3 with the flagged claim as a new sub-question, rather than returning an answer you can't trace.

Step 7: Orchestrate the Loop

Tie the steps together with an explicit state machine rather than a single giant prompt. A minimal LangGraph-style graph looks like:

plan → retrieve → grade → (insufficient? → reformulate → retrieve)
     → synthesize → verify → (unsupported claim? → retrieve)
     → return answer

Keeping this as an explicit graph — rather than trusting one long agent prompt to "figure it out" — is what makes the system debuggable in production. Each node logs its own decision, so when an answer is wrong, you can see exactly which stage introduced the error: a bad retrieval, a bad grade, or a synthesis that overreached its sources.

What This Costs You, Honestly

Agentic RAG is not free. Expect:

• 2–5x the token spend of single-pass RAG on the same question, since grading, reformulation, and verification all consume LLM calls
• Higher latency — a multi-hop query can take several seconds longer end to end
• More moving parts to monitor — retrieval, grading, and verification all need their own observability, not just the final answer

The payoff is a system that fails visibly and recovers automatically on hard, multi-part, or ambiguous questions — instead of quietly returning a confident answer built on the wrong three paragraphs. For high-stakes domains (claims processing, compliance, clinical or legal contexts), that trade is usually worth it. For simple FAQ-style lookups, classic RAG is often still the right call — agentic RAG is a scalpel for hard questions, not a universal upgrade.