Engineering Production Agentic Systems: Part 1: The Pipeline

Context Engineering for Production Agentic Systems, or Why Pipeline Beats Prompt

This is Part 1 of a three-part field manual on engineering production agentic systems. Part 1 takes context engineering. Part 2 takes guardrails. Part 3 takes human-in-the-loop topology. The conviction underneath all three: production agentic systems are won on these three architectural disciplines — not on model choice.

The Opening Claim

The first time a production agent fails because of context-window collapse, the team’s instinct is to switch to a bigger model. The agent’s reasoning gets noisy at turn nine, the most important piece of context ends up buried in the middle of the prompt, the output comes back confident and structurally wrong. The team upgrades. Six weeks later, the same failure recurs at turn fourteen.

I have watched this loop repeat across production agentic systems in regulated industries — supply chain, banking, mainframe modernization — and the pattern is identical every time. A bigger context window does not fix context-window collapse. A bigger context window is what causes context-window collapse, because it lowers the cost of pushing in junk that the agent must then reason around.

The fix is upstream of the model. The fix is to engineer the context that enters the prompt: gather it from the right places, enrich it with the metadata that makes it self-describing, verify it before it becomes a hallucination vector, compress it down to the reasoning signal, and inject it at the position in the prompt where the model will actually attend to it.

That is the pipeline. Five stages, each preventing a specific failure mode. This article is about how the stages are built — with code from the implementation, design decisions worth defending, and the trade-offs you will hit when you build your own.

The Two Views

There are two ways to read the pipeline, and you need both.

The runtime view is what happens to a single context payload as it moves from raw data to injected prompt. Five stages, executed in order: Gather, Enrich, Verify, Compress, Inject. This is the operational lens — the one a debugger uses when a context payload comes out wrong.

The architectural view is what services exist in the platform to support that runtime flow. Four layers (plus Orchestration): Acquisition handles Gather; Refinement handles Enrich and part of Verify; Distribution handles Compress and Inject; Evolution closes the loop with feedback and memory; Orchestration coordinates all four. This is the structural lens — the one an architect uses when deciding where to extend the system.

The runtime view is sequential. The architectural view is concurrent. Both are correct; neither is sufficient on its own. The runtime view without the architectural view produces a fragile monolith — a single pipeline function that does too much. The architectural view without the runtime view produces an over-engineered platform where no one can answer “what exactly happens to a request?”

The mapping matters because most teams skip one of the two views. They build the runtime view as a Python script, then six months later cannot extend it to handle a new domain because there is no service boundary to extend at. Or they build a beautiful four-layer service architecture and cannot diagnose why a particular customer’s queries get bad context, because there is no end-to-end trace of a single payload through the layers.

The implementation that backs this article keeps both views as first-class concerns. The four context_*.py modules are the architectural layers. A single request through the FastAPI service traces the runtime flow across all four. The Orchestration layer’s state manager keeps the per-request trace addressable for debugging and audit.

Inside Gather

Gather is the stage where most teams over-rely on a single retrieval method. The default move is dense vector retrieval — embed the query, find the K nearest neighbors in a vector index, return them. This works for queries where the relevance signal is semantic. It fails for queries where the relevance signal is literal — a container ID, a customer SKU, a regulatory citation, a function name. Dense retrieval will confidently surface conceptually similar but literally wrong matches.

The fix is hybrid retrieval: run dense and sparse (BM25 or TF-IDF) in parallel, then fuse the two ranked lists with a tunable weight. The fusion weight alpha (how much to favor dense versus sparse) is the most important single parameter in the entire pipeline. In production, alpha = 0.5 is almost never the right answer. For a query class dominated by entity lookups (container IDs, account numbers, SKUs), alpha lives near 0.2. For semantic exploration (“similar past exceptions in cold-chain logistics”), it lives near 0.8.

Inside Gather — Hybrid Retrieval With Weighted Fusion

A snippet of how the fusion math actually runs:

# From context_acquisition.py — HybridRetriever.retrieve
dense_results  = self.dense_retriever.similarity_search(query, k=top_k * 2)
sparse_results = self.sparse_retriever.get_relevant_documents(query)[:top_k * 2]

# Normalize each retriever's results to rank-position scores in [0, 1]
for i, doc in enumerate(dense_results):
    doc_id = doc.metadata.get("doc_id", f"dense_{i}")
    all_docs[doc_id] = {"doc": doc,
                        "dense_score":  1.0 - (i / len(dense_results)),
                        "sparse_score": 0.0}

# Fuse: hybrid_score = α · dense + (1 − α) · sparse
for doc_id, info in all_docs.items():
    info["hybrid_score"] = (self.alpha       * info["dense_score"] +
                            (1 - self.alpha) * info["sparse_score"])

Three things worth defending about this design.

Score normalization by rank, not raw score. Dense and sparse retrievers produce scores on different scales — cosine similarity versus BM25 — that do not compose linearly. Normalizing to rank position before fusion strips out the scale mismatch and makes alpha a meaningful knob.

Retrieve top_k × 2 from each, fuse, then truncate. Truncating before fusion throws away documents that score low on one retriever but high on the other — exactly the documents hybrid retrieval is supposed to surface.

alpha as a per-query-class parameter, not a constant. In production, alpha is not a single value tuned once. It varies by the query intent classifier upstream — typically a small rule-based gate that routes queries by surface features: regex matches for container IDs, SKU patterns, and regulatory citations route to entity-lookup (alpha ≈ 0.2); absence of literal anchors combined with natural-language phrasing routes to semantic-exploration (alpha ≈ 0.8); a lightweight classifier-LLM call is the fallback when neither rule pattern fires confidently. Treating alpha as a constant is the single most common Gather mistake I see.

Inside Enrich

Gather produces a list of documents. Enrich turns those documents into reasoning-grade context by attaching the metadata that the model needs in order to reason well: type tags, provenance, timestamps, confidence scores, and the hierarchical position of each document in the broader context structure.

The single highest-leverage move in Enrich is hierarchical context representation. Rather than treating retrieved documents as a flat bag, organize them into a tree: a root summary at the top, mid-level synthesis nodes, and leaf-level raw facts. The agent can navigate the tree by depth — pulling a high-level summary for fast reasoning, descending into leaves only when a sub-decision requires it.

Inside Enrich — Hierarchical Context Tree

The implementation uses a recursive node type with parent/child relationships and a flatten operation that lets the Distribution layer downstream choose how deep to descend:

# From context_refinement.py — HierarchicalContextNode
class HierarchicalContextNode:
    def __init__(self, content, level=0, metadata=None):
        self.content   = content
        self.level     = level                  # 0 = highest abstraction
        self.metadata = metadata or {}
        self.children = []
        self.parent    = None

    def flatten(self, max_depth=None):
        documents = [Document(page_content=self.content,
                               metadata={**self.metadata, "level": self.level})]
        if max_depth is not None and self.level >= max_depth:
            return documents
        for child in self.children:
             documents.extend(child.flatten(max_depth))
        return documents

Two design choices worth surfacing.

Level as an integer, not a categorical type. Levels compose; categories don’t. A level-2 node beneath a level-1 node beneath a level-0 root is something you can reason about in a depth-limited traversal. A “summary node” beneath a “synthesis node” beneath a “root node” is a taxonomy that fights extension.

Metadata at every level, not just leaves. The summary nodes carry provenance too — which leaves contributed to them, what time window they span, what confidence the synthesis carries. The agent reasoning over a summary needs to know whether that summary was derived from five recent reliable sources or one stale one.

Inside Verify

This is the stage most production pipelines do not have. Teams add Verify after their first hallucination post-mortem, typically around month four. It is cheaper to add on day one.

Verify does three things. It cross-references retrieved facts against a ground-truth registry where one exists (regulatory citations, product specs, customer records). It flags sources known to be stale or untrusted, dropping them from the context payload or marking them with reduced confidence. And it applies contrastive ranking — a technique drawn from Stanford’s work on contextual representation learning — to demote documents that are conceptually adjacent but factually divergent from the query intent.

The hardest part of Verify is not the implementation. It is the policy: deciding what counts as ground truth, who maintains it, and how stale is too stale. In a supply chain context, the carrier’s tracking API is ground truth for container location — except when the carrier has not pushed an update in eight hours, at which point a manual log entry from operations becomes ground truth. The Verify stage encodes that policy explicitly. The agent should not be reasoning about which source to trust; the pipeline should have made that decision before the context entered the prompt.

In production, Verify is also the stage where compliance lives. Regulated industries — banking, healthcare, anything subject to data residency or PII handling — need a stage where context is checked for regulatory cleanliness before it gets injected into a prompt that might be logged, replayed, or sent to an external model. That check belongs in Verify, not in the prompt template, and not as a post-hoc audit step.

Inside Compress

Compression is where the “bigger context window will fix it” argument finally evaporates.

The naive approach to compression is truncation: keep the top-K by relevance score, drop the rest. This is wrong in a specific and dangerous way. Truncation by relevance throws away exactly the documents that provide contrast — the close-but-different cases that prevent the model from over-generalizing. The agent reasoning over only the top-K most relevant documents converges quickly to a confident wrong answer, because nothing in the prompt is pushing back on its first hypothesis.

The better approach is landmark attention, drawn from Meta’s research on efficient context windows. The idea: identify a small number of high-information tokens (landmarks) distributed across the full context, preserve those at high fidelity, and compress the filler around them. The agent retains the structural shape of the full context while paying token cost only for the landmarks plus a thin shell of surrounding context.

Inside Compress — Landmark Attention vs. Naïve Truncation

# From context_distribution.py — LandmarkAttention.extract_landmarks
combined_text = " ".join([doc.page_content for doc in documents])
words = combined_text.split()

if len(words) <= self.num_landmarks:
    return [(i, word) for i, word in enumerate(words)]

# Evenly spaced landmarks across the full context surface.
# Production replaces uniform spacing with TF-IDF + entity-boost + query-conditioned scoring.
indices   = np.linspace(0, len(words) - 1, self.num_landmarks, dtype=int)
landmarks = [(int(i), words[int(i)]) for i in indices]
return landmarks

The snippet above shows the evenly-spaced baseline. In production, the landmark selection function is richer — TF-IDF weighting, named-entity boosting, and query-conditioned selection all improve over uniform spacing. The principle is what matters: landmarks are positions in the context, not just the top-K documents.

KEY RESULT · LANDMARK ATTENTION

~70% token reduction. 16K-token payload compresses to roughly 4.8K tokens with the reasoning signal preserved end-to-end — measured against my own production workloads. That ratio is the difference between an agentic system that runs at one cent per turn and one that runs at three cents per turn — the difference between deployable and not deployable at any reasonable scale.

Inside Inject

The last stage is the one most teams do not even name. They retrieve, they format, they concatenate, and they append the result to the system message. That is not injection — that is appending, and it is responsible for a class of failures most teams misdiagnose as “the model is bad at long context.”

Injection is the deliberate placement of compressed context at specific positions in the prompt: some in the system message, some in a tool-result slot, some in a structured XML block earlier in the user turn, some in the function-calling schema itself. Different positions yield different attention. The model attends most strongly to the start and end of the prompt and least strongly to the middle. Inject puts the load-bearing context at the positions the model will actually read.

The implementation uses token-level fusion drawn from Google’s RAG-Token work: rather than concatenating context as one block, it interleaves context tokens with query tokens at fusion points the prompt template specifies. The Function Identifier component handles the special case of injecting context into function-calling schemas, so the agent’s available tool set is parameterized by the current context rather than being a static catalog.

# From context_distribution.py — abbreviated injection path
def distribute_context(self, context, process_step, role,
                        task_complexity, max_tokens, task_description):
    compressed = self.landmark_attention.apply(context)
    template   = self.template_manager.for_role(role, process_step)
    fused      = self.token_fusion.interleave(compressed, template,
                                               max_tokens=max_tokens)
    fn_schema  = self.function_identifier.constrain_tools(fused, role)
    return {"prompt": fused, "tools": fn_schema}

Three things to flag.

Role and process step both parameterize the template. A Logistics Analyst at Initial Assessment gets a different injection template than a Supply Chain Manager at Resolution Planning. Same underlying context tree; different slice, different position, different surrounding scaffolding.

Tools are constrained by context. The set of tools exposed to the agent in this turn is filtered by what the current context implies the agent should be allowed to call. This is the seam between the pipeline and the guardrails — Part 2 picks this thread up.

max_tokens is a runtime parameter, not a constant. Complexity calibration upstream decides how much budget this particular turn deserves. A high-stakes resolution-planning turn gets a larger budget than a routine status update. Constant token budgets are a tell for a pipeline that hasn’t been instrumented yet.

Measurement

The pipeline is observable, or it is not real. The Evolution Layer in the architecture exists primarily as the measurement-and-feedback surface.

Three signals are worth instrumenting from day one. Relevance feedback on every distributed context payload — was the context that the agent received useful for the decision it then made? Captured via a /feedback endpoint that the downstream agent or the human-in-the-loop fills in. Drift detection on the retrieval distribution — when the mix of documents returned by Gather shifts, that is a leading indicator of either upstream data changes or query-class drift, both of which need investigation before the agent’s accuracy starts degrading. Compression fidelity — periodic re-runs of the same query with and without compression, checking whether downstream agent decisions change.

The Evolution Layer feeds the signals back into the upstream stages. Bad relevance scores reweight the fusion alpha. Drift shifts the retrieval thresholds. Fidelity regressions trigger an alert on the landmark selection function. The pipeline does not just run — it improves the next time it runs, because the feedback ingestion is engineered into the pipeline itself rather than left as a manual exercise. This closes the runtime view back onto the architectural view; the loop is the system.

Closer

Context engineering is the unglamorous half of agentic system design. It is also the half that determines whether the system holds up in production under regulated-industry scrutiny.

The pipeline I have described is not exotic. The components are documented in published research from MIT, Google, OpenAI, Meta, IBM, and Stanford. What is exotic — and what I keep watching teams under-invest in — is the discipline of treating context as a managed pipeline rather than as a one-shot retrieval step. The five runtime stages and the four architectural layers compose. The composition is what makes the system extensible and what makes failure modes diagnosable when they happen.

This pattern is what underpins MoJoCo, the agentic modernization platform I have been designing hands-on for the last eighteen months. The deterministic reverse-engineering tools that sit underneath (ARC, MAM, CAST) produce the kind of raw output that demands a context pipeline — high-volume, mixed-modality, varying confidence — and the pipeline is what turns that raw output into reasoning-grade context the multi-agent layer can act on. The pipeline is the actual product. The tools are the surface.

Two failure modes I have not yet addressed in this part: tool-authorization sprawl and audit-trail opacity. Both are guardrails problems — the pipeline can produce the cleanest possible context, but if the agent then calls the wrong tool with the wrong parameters and no one can trace what happened, the cleanliness of the context did not save you. Part 2 takes that on.

Production agentic systems are won on context engineering, guardrails, and human-in-the-loop topology — not on model choice. We have closed out the first of the three. The pipeline is the moat the field is currently under-investing in. The next part takes the second.

References

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). “Lost in the Middle: How Language Models Use Long Contexts.” Stanford. Underpins the Inject-stage claim that the model attends most strongly to the start and end of the prompt and least strongly to the middle. arXiv:2307.03172

Mohtashami, A., & Jaggi, M. (2023). “Landmark Attention: Random-Access Infinite Context Length for Transformers.” EPFL. Source of the landmark attention technique used in the Compress stage; reduces token cost by preserving high-information landmarks at high fidelity while compressing surrounding context. arXiv:2305.16300

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Facebook AI Research. NeurIPS 2020. The original RAG paper; foundation for the token-level fusion approach in the Inject stage. arXiv:2005.11401

Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). “Dense Passage Retrieval for Open-Domain Question Answering.” Facebook AI Research. EMNLP 2020. The dense retrieval foundation for the Gather stage’s hybrid retrieval design. arXiv:2004.04906

Robertson, S., & Zaragoza, H. (2009). “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval, 3(4), 333–389. The sparse retrieval (BM25) foundation paired with dense retrieval in hybrid fusion. Publication link

Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” Proceedings of SIGIR 2009, 758–759. Justifies the rank-position-based fusion approach used in HybridRetriever; explains why fusing on rank is more robust than fusing on raw scores. Publication link

Part 2 — The Guardrails: tool surface design, authorization scopes, audit-trail engineering — drops next.