Extracting Entities and Relationships From Engineering Documents With spaCy

Engineering teams generate a lot of useful knowledge, but most of it is locked inside text.

A service ownership note may tell you who owns an application programming interface (API), while a runbook may tell you which database a service relies on. An incident review may detail how one fault impacted the other systems. Each of these is individually useful. It’s when we are able to link together all of these facts that we get our greatest value.

As an example, think about a developer who wants to know:

Which team owns the API that Checkout Service depends on?

To answer this question, the developer would need to make several connections:

Checkout Service depends on Payment API.
Platform Team owns Payment API.

With a keyword search, you could locate documents that include “Checkout Service,” “Payment API.” With a retrieval-augmented generation(RAG) pipeline, you could locate relevant chunks and forward them to your LLM. If there isn’t some understanding of the relationships between services, APIs, databases, etc., then your RAG may still fail to identify a connection.

This is where entity and relationship extraction can help.

We want to move beyond viewing engineering documents as plain text; instead, we’d like to see the text as a source of structured facts, such as:

Checkout Service --DEPENDS_ON--> Payment API
Platform Team --OWNS--> Payment API
Payment API --STORES_IN--> PostgreSQL

These facts can then provide the basis for knowledge graphs, dependency analysis, impact analysis, and GraphRAG pipelines.

Here we will define a simple Python pipeline using spaCy to extract specific types of engineering entities and relationship triplets from text. Our objective is to produce a working prototype, rather than creating the ultimate extraction tool. We also hope that this provides a practical starting point for developers to test, view, and modify their own documentation.

Why This Pattern Matters in Real Systems

This pattern applies to multiple aspects of real-world engineering systems.

Service Ownership Lookup

A developer can query a service owner by looking at a service they are unaware of (as opposed to manual searches of a service catalog).

Analyzing Dependencies

Using a graph to find all other services impacted when a system (API, Queue, Database) goes down.

Response During Incidents

Relationship extraction can transform incident notes and runbooks into navigationally easier-to-use dependency maps during outages.

GraphRAG Pipelines

Entity/relationship extraction is required for GraphRAGs prior to being able to get the appropriate graph contextual information. Poor entity/relationship extraction will result in poor graph contextual information regardless of how good the LLM is.

Discovering Architecture

Many large organizations have their architectural knowledge dispersed across various documents, diagrams, and repositories. Entity/relationship extraction provides a method to make this knowledge searchable and reusable.

We'll use a simple dataset in this tutorial, but you could easily expand on it using your organization's service catalog(s), repository metadata, cloud inventory, or documentation from real engineering systems.

What We Are Building

We will create a small Python script that reads engineering notes and produces two files:

entities.json
triples.json

The first file contains detected entities:

[
  {
    "text": "Checkout Service",
    "label": "SERVICE"
  },
  {
    "text": "Payment API",
    "label": "API"
  }
]

The second file contains relationships:

[
  {
    "subject": "Checkout Service",
    "relation": "DEPENDS_ON",
    "object": "Payment API",
    "source_text": "Checkout Service depends on Payment API during payment authorization."
  }
]

This output can later be loaded into NetworkX, Neo4j, a vector database, or a GraphRAG pipeline.

Project Setup

Create a new folder:

mkdir spacy-entity-relationship-extraction
cd spacy-entity-relationship-extraction

Install a virtual environment, spaCy, and the small English model:

python3.12 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

pip install spacy  # installs spacy

python -m spacy download en_core_web_sm  # installs english model

The small English model is enough for this tutorial. For production work, you may need a larger model, domain-specific rules, or custom training data.

Step 1: Define Sample Engineering Documents

Create a file named extract_entities.py.

documents = [
    "Checkout Service depends on Payment API during payment authorization.",
    "Payment API stores transaction metadata in PostgreSQL.",
    "Platform Team owns Payment API and manages its deployment pipeline.",
    "Recommendation Service calls Catalog API to retrieve product details.",
    "Catalog API indexes product data in Elasticsearch.",
    "Search Team owns Catalog API.",
]

Examples like these are simple; however, they illustrate typical engineering documentation patterns:

• one System Depends Upon Another System,
• a service stores information in a database,
• a team is responsible for a service,
• one application calls an application programming interface (API).

Step 2: Add Custom Entity Rules

Most of the time, general-purpose named entity recognition (NER) models are trained to identify persons, organizations, locations, and other types of information such as dates. The engineering documentation our NLP tools will process contains a range of domain-specific entities, including services, APIs, databases, queues, teams, repositories, and cloud-based services.

In order for these entities to be identified in the pipeline, it would make sense to incorporate domain-specific rules into our code.

import spacy


def create_pipeline():
    nlp = spacy.load("en_core_web_sm") 			# Load English Model
    ruler = nlp.add_pipe("entity_ruler", before="ner")  # Create Ruler
    patterns = [
        {"label": "SERVICE", "pattern": "Checkout Service"},
        {"label": "SERVICE", "pattern": "Recommendation Service"},
        {"label": "API", "pattern": "Payment API"},
        {"label": "API", "pattern": "Catalog API"},
        {"label": "DATABASE", "pattern": "PostgreSQL"},
        {"label": "SEARCH_INDEX", "pattern": "Elasticsearch"},
        {"label": "TEAM", "pattern": "Platform Team"},
        {"label": "TEAM", "pattern": "Search Team"},
    ]
    ruler.add_patterns(patterns)	# Add Patterns to Ruler
    return nlp

EntityRuler enables you to include common domain-specific named entities from your company’s service catalog, team listing, repository listing, and database inventory, etc., without training a custom model.

Step 3: Extract Entities

Now add a function to extract entities from each document.

def extract_entities(nlp, documents):
    results = []

    for text in documents:
        doc = nlp(text)

        for entity in doc.ents:
            results.append(
                {
                    "text": entity.text,
                    "label": entity.label_,
                    "source_text": text,
                }
            )

    return results

Run it:

if __name__ == "__main__":
    nlp = create_pipeline()
    entities = extract_entities(nlp, documents)

    for entity in entities:
        print(entity)

Expected output:

{'text': 'Checkout Service', 'label': 'SERVICE', 'source_text': 'Checkout Service depends on Payment API during payment authorization.'} 
{'text': 'Payment API', 'label': 'API', 'source_text': 'Checkout Service depends on Payment API during payment authorization.'} 
{'text': 'Payment API', 'label': 'API', 'source_text': 'Payment API stores transaction metadata in PostgreSQL.'} 
{'text': 'PostgreSQL', 'label': 'DATABASE', 'source_text': 'Payment API stores transaction metadata in PostgreSQL.'}

The above output is much better than plain text. The output indicates which tokens within the document refer to engineering entities.

Step 4: Normalize Duplicate Entities

Because entities are almost always referred to by different names within the same document (eg., payment-api, Payment-API, payment service, etc.), you need an approach that uses aliases to link all of these references back into a single “concept” or node.

The first approach to doing this would be creating an alias map. This could look something like this:

ALIASES = {
    "payment-api": "Payment API",
    "Payments API": "Payment API",
    "payment service": "Payment API",
}

The second part of this solution involves defining an additional helper function that will take each found entity reference and “normalize” it by replacing it with its corresponding concept label. The normalized value is then stored along with the original extracted data in order to be returned as output.

Here's how we can do that:

def normalize_entity_name(name): 
  return ALIASES.get(name, name)

Then update the entity extraction function:

def extract_entities(nlp, documents):
    results = []
    for text in documents:
        doc = nlp(text)
        for entity in doc.ents:
            results.append(
                {
                    "text": normalize_entity_name(entity.text),
                    "label": entity.label_,
                    "source_text": text,
                }
            )
    return results

As previously stated, while this process appears to be relatively minor, normalizing entities is probably one of the most critical steps in developing a knowledge graph because if done poorly, what may appear to be a single system in the world may end up being represented as many separate nodes.

Step 5: Extract Relationship Triples

Now that we have identified all of our entities, we want to find the relationships.

We will begin with the simplest way to do this using verb phrases. The simplicity of this method does not allow for much flexibility or learning from the data, as a trained relation extraction model would, but it has the advantage of being completely transparent and very easy to debug.

RELATION_VERBS = {
    "depends on": "DEPENDS_ON",
    "stores": "STORES_IN",
    "owns": "OWNS",
    "calls": "CALLS",
    "indexes": "INDEXES_IN",
}

Create an additional helper function that identifies entity locations within the text.

def extract_relationships(nlp, documents):
    triples = []

    for text in documents:
        doc = nlp(text)
        entities = list(doc.ents)

        if len(entities) < 2:
            continue

        for verb_phrase, relation in RELATION_VERBS.items():
            if verb_phrase in text.lower():
                subject = normalize_entity_name(entities[0].text)
                object_ = normalize_entity_name(entities[1].text)

                triples.append(
                    {
                        "subject": subject,
                        "relation": relation,
                        "object": object_,
                        "source_text": text,
                    }
                )

    return triples

The above code has no universal way to determine which entity is the subject or object; however, when you know your input data is structured like controlled engineering documentation, this method provides a good working example for a baseline.

Run it:

if __name__ == "__main__": 
  nlp = create_pipeline() 
  
  triples = extract_relationships(nlp, documents) 
  
  for triple in triples: 
    print(triple)

Expected output:

{'subject': 'Checkout Service', 'relation': 'DEPENDS_ON', 'object': 'Payment API', 'source_text': 'Checkout Service depends on Payment API during payment authorization.'}
{'subject': 'Payment API', 'relation': 'STORES_IN', 'object': 'PostgreSQL', 'source_text': 'Payment API stores transaction metadata in PostgreSQL.'}
{'subject': 'Platform Team', 'relation': 'OWNS', 'object': 'Payment API', 'source_text': 'Platform Team owns Payment API and manages its deployment pipeline.'}

For many internal engineering knowledge bases, this kind of controlled extraction is a good first step before adding more complex models.

Step 6: Export JSON

The final step is to save entities and triples.

import json 


def write_json(path, data): 
  with open(path, "w", encoding="utf-8") as file: 
    json.dump(data, file, indent=2)

Use it:

if __name__ == "__main__": 
  nlp = create_pipeline() 
  
  entities = extract_entities(nlp, documents) 
  triples = extract_relationships(nlp, documents) 
  
  write_json("entities.json", entities) 
  write_json("triples.json", triples) 
  
  print(f"Wrote {len(entities)} entities to entities.json") 
  print(f"Wrote {len(triples)} triples to triples.json")

Complete Example

Here is the full script:

import json
import spacy

documents = [
    "Checkout Service depends on Payment API during payment authorization.",
    "Payment API stores transaction metadata in PostgreSQL.",
    "Platform Team owns Payment API and manages its deployment pipeline.",
    "Recommendation Service calls Catalog API to retrieve product details.",
    "Catalog API indexes product data in Elasticsearch.",
    "Search Team owns Catalog API.",
]

ALIASES = {
    "payment-api": "Payment API",
    "Payments API": "Payment API",
    "payment service": "Payment API",
}

RELATION_VERBS = {      
    "depends on": "DEPENDS_ON",
    "stores": "STORES_IN",
    "owns": "OWNS",
    "calls": "CALLS",
    "indexes": "INDEXES_IN",
}


def create_pipeline():
    nlp = spacy.load("en_core_web_sm")
    ruler = nlp.add_pipe("entity_ruler", before="ner")
    patterns = [
        {"label": "SERVICE", "pattern": "Checkout Service"},
        {"label": "SERVICE", "pattern": "Recommendation Service"},
        {"label": "API", "pattern": "Payment API"},
        {"label": "API", "pattern": "Catalog API"},
        {"label": "DATABASE", "pattern": "PostgreSQL"},
        {"label": "SEARCH_INDEX", "pattern": "Elasticsearch"},
        {"label": "TEAM", "pattern": "Platform Team"},
        {"label": "TEAM", "pattern": "Search Team"},
    ]
    ruler.add_patterns(patterns)
    return nlp


def normalize_entity_name(name):
    return ALIASES.get(name, name)


def extract_entities(nlp, documents):
    results = []
    for text in documents:
        doc = nlp(text)
        for entity in doc.ents:
            results.append(
                {
                    "text": normalize_entity_name(entity.text),
                    "label": entity.label_,
                    "source_text": text,
                }
            )
    return results


def extract_relationships(nlp, documents):
    triples = []
    for text in documents:
        doc = nlp(text)
        entities = list(doc.ents)
        if len(entities) < 2:
            continue
        for verb_phrase, relation in RELATION_VERBS.items():
            if verb_phrase in text.lower():
                subject = normalize_entity_name(entities[0].text)
                object_ = normalize_entity_name(entities[1].text)
                triples.append(
                    {
                        "subject": subject,
                        "relation": relation,
                        "object": object_,
                        "source_text": text,
                    }
                )
    return triples


def write_json(path, data):
    with open(path, "w", encoding="utf-8") as file:
        json.dump(data, file, indent=2)


if __name__ == "__main__":
    nlp = create_pipeline()
    entities = extract_entities(nlp, documents)
    triples = extract_relationships(nlp, documents)
    write_json("entities.json", entities)
    write_json("triples.json", triples)
    print(f"Wrote {len(entities)} entities to entities.json")
    print(f"Wrote {len(triples)} triples to triples.json")

Run it:

python extract_entities.py

You should now see two output files:

entities.json
triples.json

Why This Matters to GraphRAG Systems

GraphRAG systems rely on accurate entity and relationship information in order to perform well.

Poor extraction means poor graph retrieval. The Large Language Model (LLM), while capable of reasoning about the context provided, cannot correct the underlying data problems.

Therefore, entity extraction, entity normalization, relationship extraction, and source tracking are to be considered first-class engineering issues.

Production Considerations

While this tutorial uses a small set of rules to extract entities and relationships, larger-scale production systems require additional robustness through "guardrails".

Use a service catalog when possible.

If your organization has an existing internal listing of services, teams, repositories, and owners, then these items should be used as the sources of truth for extracting patterned entities.

Track confidence.

Do not treat all extracted relationships with equal weight. Inferences from unstructured documentation, such as a free-form document, likely have less confidence than those from structured documentation, such as a service catalog.

Keep the source sentence.

Each triple includes the original sentence text. This makes reviewing, debugging, and referencing answers much simpler.

Review low-confidence edges.

Graph errors grow rapidly. An error in one dependency can cause downstream traversal results to be misleading.

Start simple before using an LLM.

While LLMs can be helpful for extracting entities/relationships from unstructured data, they increase cost, latency, and variability. Rules + Domain Dictionary may be sufficient for predictable documentation like engineering documentation.

Key Takeaways

SpaCy is effective in both general NLP and domain-specific extraction of entities via custom entity rules.

Domain documents like engineering documentation will have custom entity names that include SERVICE, API, DATABASE, TEAM, etc.

The relationship triples represent an efficient path from the unstructured document to the knowledge graph.

Entity Normalization is NOT optional. If you do not perform Entity Normalization on your extracted entities, then the single "Real-world" system will be represented by multiple isolated/disconnected nodes in your graph.

In terms of performance with GraphRAG systems, quality of extraction is equal in importance as retrieval and prompting.

Try It Yourself

Add the following documents:

Billing Worker publishes events to Kafka. 
Data Platform Team owns Kafka.

Then add the following to the entity patterns:

Kafka
Billing Worker
Data Platform Team

Finally, add a new relationship type:

PUBLISHES_TO

Your goal is to produce the following triple:

Billing Worker --PUBLISHES_TO--> Kafka

This small exercise demonstrates the work that is needed to adapt entity and relationship extraction to your own engineering domain.