Beyond Retrieval: How Knowledge Graphs Supercharge RAG

Retrieval-augmented generation (RAG) enhances the factual accuracy and contextual relevance of large language models (LLMs) by connecting them to external data sources. RAG systems use semantic similarity to identify text relevant to a user query. However, they often fail to explain how the query and retrieved pieces of information are related, which limits their reasoning capability.

Graph RAG addresses this limitation by leveraging knowledge graphs, which represent entities (nodes) and their relationships (edges) in a structured, machine-readable format. This framework enables AI systems to link related facts and draw coherent, explainable conclusions, moving closer to human-like reasoning (Hogan et al., 2021).

In this article and the accompanying tutorial, we explore how Graph RAG compares to traditional RAG by answering questions drawn from A Study in Scarlet, the first novel in the Sherlock Holmes series, demonstrating how structured knowledge supports more accurate, nuanced, and explainable insights.

Constructing Knowledge Graphs: From Expert Ontologies to Large-Scale Extraction

The first step in implementing graph RAG is building a knowledge graph. Traditionally, domain experts manually define ontologies and map entities and relationships, producing high-quality structures. However, this approach does not scale well for large volumes of text. To handle bigger datasets, LLMs and NLP techniques automatically extract entities and relationships from unstructured content. In the tutorial accompanying this article, we demonstrate this process by breaking it down into three practical stages:

1. Entity and Relationship Extraction

The first step in building the graph is to extract entities and relationships from raw text. In our tutorial, this is done using Cohere’s command-a model, guided by a structured JSON schema and a carefully engineered prompt. This approach enables the LLM to systematically identify and extract the relevant components.

For example, the sentence: 

“I was dispatched, accordingly, in the troopship ‘Orontes,’ and landed a month later on Portsmouth jetty.” 

can be converted into the following graph triple: (Watson) – [travelled_on] → (Orontes)

Instead of processing the entire novel in one pass, we split it into chunks of about 4,000 characters. Each chunk is processed with two pieces of global context:

1. Current entity list: all entities discovered so far.
2. Current relation types: the set of relations already in use.

This approach prevents the model from creating duplicate nodes and keeps relation labels consistent across the book. An incremental merging step then assigns stable IDs to new entities and reuses them whenever known entities reappear.

For instance, once the relation type meets is established, later chunks will consistently reuse it rather than inventing variants such as encounters or introduced_to. This keeps the graph coherent as it grows across chunks.

2. Entity Resolution

Even with a global context, the same entity often appears in different forms; a character, location, or organization may be mentioned in multiple ways. For example, “Stamford” might also appear as “young Stamford”, while “Dr Watson” is later shortened to “Watson.”

To handle this, we apply a dedicated entity resolution step after extraction. This process:

1. Identify and merge duplicate nodes based on fuzzy string similarity (e.g., Watson ≈ Dr Watson).
2. Expand and unify alias lists so that each canonical entity maintains a complete record of its variants.
3. Normalize relationships so that all edges consistently point to the resolved nodes.

This ensures that queries such as “Who introduced Watson to Holmes?” always return the correct canonical entity, regardless of which alias appears in the text.

3. Graph Population and Visualization

With resolved entities and cleaned relationships, the data is stored in a graph structure. Nodes can be enriched with further attributes such as type (Person, Location, Object), aliases, and optionally traits like profession or role. Edges are explicitly typed (e.g., meets, travels_to, lives_at), enabling structured queries and traversal.

Together, these stages transform unstructured narrative text into a queryable, explainable Knowledge Graph, laying the foundation for Graph RAG to deliver richer insights than traditional retrieval methods. Once the Knowledge Graph is constructed, it is stored in a graph database designed to handle interconnected data efficiently. Databases like Neo4j allow nodes and relationships to be queried and traversed intuitively using declarative languages such as Cypher.

Graph RAG vs. Traditional RAG

The main advantage of Graph RAG over traditional RAG lies in its ability to leverage structured relationships between entities. Traditional RAG retrieves semantically similar text fragments but struggles to combine information from multiple sources, making it difficult to answer questions that require multi-step reasoning.

• In the tutorial, for example, we can ask: “Who helped Watson after the battle of Maiwand, and where did this occur?”
• Graph RAG answers this by traversing the subgraph: Dr Watson → [HELPED_BY] → Murray → [LOCATED_AT] → Peshawar.
• Traditional RAG would only retrieve sentences mentioning Watson and Murray without connecting the location or providing a reasoning chain.

This shows that graph RAG produces richer, more accurate, and explainable answers by combining entity connections, attributes, and relationships captured in the knowledge graph.

A key benefit of graph RAG is transparency. Knowledge graphs provide explicit reasoning chains rather than opaque, black-box outputs. Each node and edge is traceable to its source, and attributes such as timestamps, provenance, and confidence scores can be included. This level of explainability is particularly important in high-stakes domains like healthcare or finance, but it also enhances educational value in literary analysis, allowing students to follow narrative reasoning in a structured, visual way.

Enhancing Retrieval With Graph Embeddings

Graph data can be enriched through graph embeddings, which transform nodes and their relationships into a vector space. This representation captures both semantic meaning and structural context, making it possible to identify similarities beyond surface-level text. Embedding algorithms such as FastRP and Node2Vec enable the retrieval of relationally similar nodes, even when their textual descriptions differ

By integrating LLMs with knowledge graphs, graph RAG transforms retrieval from a simple text lookup into a structured reasoning engine. It enables AI systems to link facts, answer complex questions, and provide explainable, verifiable insights. Graph RAG represents a step toward AI systems that are not only larger and more capable but also smarter, more transparent, and more trustworthy, capable of structured reasoning over rich information sources.

References

• Hogan, A., Blomqvist, E., Cochez, M., d’Amato, C., de Melo, G., Gutierrez, C., Gayo, J.E.L., Kirrane, S., Neumaier, S., Polleres, A., Navigli, R., Ngomo, A.C.N., Rashid, S.M., Rula, A., Schmelzeisen, L., Sequeda, J., Staab, S., and Zimmermann, A. (2021) ‘Knowledge Graphs’, ACM Computing Surveys, 54(4), pp. 1–37.
• Larson, J. and Truitt, A. (2024) ‘Beyond RAG: Graph-based retrieval for multi-step reasoning’, Microsoft Research Blog. Available at: https://www.microsoft.com/en-us/research/blog/beyond-rag-graph-based-retrieval-for-multi-step-reasoning/ (Accessed: 20 July 2025).
• Neo4j (2024) ‘Graph data science documentation’, Neo4j. Available at: https://neo4j.com/docs/graph-data-science/current/ (Accessed: 20 July 2025).