Agentic RAG: Basic RAG Plus MCP Tool Calls

That second question isn't a retrieval problem — it's a computation problem that depends on live, structured, user-specific data. This is exactly the gap agentic RAG closes: it keeps the semantic search RAG is good at, and bolts on tool calls (via MCP) so the agent can reach into live systems, fetch real numbers, and reason over them before answering.

A simple way to think about it:

Agentic RAG = Basic RAG (semantic search over documents)
            + MCP Tool Calls (structured lookups + computation over live systems)
            + an orchestration loop that decides which one(s) to use

Let's build this out with a use case everyone can relate to: an HR policy + leave balance assistant.

The Problem With "Just RAG" Here

Say your company has an HR Policy document — PTO rules, carry-forward limits, accrual rates, blackout periods, etc. Classic RAG handles this well:

1. Chunk the HR policy PDF/doc.
2. Embed each chunk and store it in a vector DB (Pinecone, Weaviate, pgvector, whatever).
3. On a user question, embed the query, run semantic search, retrieve top-k chunks.
4. Stuff those chunks into the LLM's context and generate an answer.

This works beautifully for: "How many days of PTO carry-forward am I allowed under company policy?"

It completely falls apart for: "How many leave days do I have left this year, including my carry-forward from last year?"

Why? Because that answer isn't in any document. It lives in a row in an HR database, tied to this specific employee, and it requires arithmetic (accrued − used + carried-forward, capped by policy rules). No amount of semantic search over a policy PDF will produce that number — the LLM would just hallucinate it.

Enter the Agentic Layer

This is where Model Context Protocol (MCP) tool calls come in. Instead of treating the LLM as a pure text-in/text-out retrieval consumer, we give it tools — callable functions exposed by an MCP server — that can hit real systems: HR databases, payroll APIs, ticketing systems, whatever.

The agent now has two capabilities instead of one:

The orchestration layer — usually just the LLM itself, given tool definitions — decides which capability (or both) a given question needs.

Architecture

                        ┌─────────────────────┐
                        │   User Question      │
                        └──────────┬───────────┘
                                   │
                          ┌────────▼─────────┐
                          │   Agent / LLM      │
                          │ (decides: RAG,     │
                          │  tool call, both)  │
                          └───┬───────────┬────┘
                              │           │
                 ┌────────────▼───┐   ┌───▼─────────────────┐
                 │  Vector DB      │   │  MCP Server          │
                 │  (HR policy     │   │  → get_leave_balance │
                 │   chunks)       │   │  → get_carry_forward │
                 └─────────────────┘   │  → get_employee_info │
                                       └──────────┬───────────┘
                                                  │
                                        ┌─────────▼─────────┐
                                        │  HR Database        │
                                        │  (SQL tables)        │
                                        └──────────────────────┘

Step 1: Classic RAG for the Policy Document

Nothing new here — this is bog-standard RAG.

from vector_db_client import VectorDB
from embeddings import embed

hr_policy_chunks = chunk_document("hr_leave_policy.pdf", chunk_size=500, overlap=50)

vector_db = VectorDB(collection="hr_policy")
for chunk in hr_policy_chunks:
    vector_db.upsert(
        id=chunk.id,
        vector=embed(chunk.text),
        metadata={"text": chunk.text, "section": chunk.section}
    )

def semantic_search(query, top_k=4):
    query_vector = embed(query)
    return vector_db.query(query_vector, top_k=top_k)

Step 2: Expose HR Data as MCP Tools

This is the piece that turns plain RAG into agentic RAG. We wrap the HR database behind an MCP server, exposing a small set of well-defined tools rather than raw SQL access.

# mcp_server.py
from mcp import Server, tool

server = Server("hr-tools")

@tool()
def get_employee_leave_summary(employee_id: str, year: int) -> dict:
    """
    Returns accrued leave, used leave, and carried-forward
    balance for a given employee and year.
    """
    row = db.query(
        "SELECT accrued, used, carried_forward, max_carry_forward "
        "FROM leave_ledger WHERE employee_id = %s AND year = %s",
        (employee_id, year)
    )
    remaining = row.accrued - row.used + min(
        row.carried_forward, row.max_carry_forward
    )
    return {
        "accrued": row.accrued,
        "used": row.used,
        "carried_forward": min(row.carried_forward, row.max_carry_forward),
        "remaining": remaining
    }

@tool()
def get_carry_forward_rules(policy_year: int) -> dict:
    """Returns max carry-forward days and expiry rules for a given policy year."""
    return db.query(
        "SELECT max_days, expiry_month FROM carry_forward_rules WHERE year = %s",
        (policy_year,)
    )

Each @tool() becomes a callable function the LLM can invoke, with a name, description, and typed schema — that's the MCP contract.

Step 3: Let the Agent Orchestrate

The LLM is given both the semantic_search retriever and the MCP tools as available "functions." Given a question, it decides what to call.

tools = [semantic_search_tool, get_employee_leave_summary_tool, get_carry_forward_rules_tool]

response = agent.run(
    system_prompt="You are an HR assistant. Use policy search for general "
                  "policy questions. Use the HR tools for anything specific "
                  "to the current user's leave balance or history.",
    user_message="How many leave days do I have left this year, including "
                 "my carry-forward from last year?",
    tools=tools,
    context={"employee_id": current_user.id}
)

For this question, a well-orchestrated agent will typically:

1. Call get_employee_leave_summary for the current year → gets accrued/used/carried-forward.
2. Call get_carry_forward_rules (if it needs to double check the cap, or if the raw carry-forward exceeds policy) → cross-references against the policy limit.
3. Optionally call semantic_search if the user also asks something like "...and can I still use my carry-forward days after March?" — that expiry rule may only exist in the policy text, not the DB.
4. Compose the final answer by combining the structured numbers with any policy language retrieved, e.g.:

"You have 14 days remaining for this year (18 accrued − 9 used + 5 carried forward, capped at your policy limit of 5). Per company policy, your carried-forward days must be used by March 31st."

That final sentence is the tell-tale sign of agentic RAG in action — a computed number from a live tool call, fused with a rule pulled from semantic search over a document, in one coherent answer.

Why This Matters

• RAG alone hallucinates on anything personalized or computed — it has no way to "look up" a fact that isn't embedded as text.
• Tool calls alone (no RAG) work fine for structured lookups but can't answer open-ended policy questions phrased in natural language, or reason over long unstructured documents.
• Agentic RAG picks the right capability per sub-question, and — crucially — can chain them: retrieve a rule, then use it to validate or cap a computed value.

This pattern generalizes far beyond HR: think claims platforms pulling policy documents + live claim status via tool calls, or support bots blending product docs (RAG) with live order status (MCP tool call). The shape of the problem is the same: static knowledge → RAG, live/structured/computed truth → tool calls, and an agent smart enough to know which is which.

Takeaway

If your RAG pipeline only ever answers "what does the document say," you're leaving the more valuable, more personalized questions unanswered — or worse, hallucinated. Wiring in MCP tool calls alongside your vector search turns a static Q&A bot into something that can actually reason over a user's real, current data. That's the "agentic" in agentic RAG.