Agentic System Design in Practice: The Technical Debt in Enterprise Agentic Systems

Everybody wants to build agents these days  —  the internet is bombarded with stories of developers who built their own productivity agents in a single day. Can enterprises do the same? Maybe, but not at scale. Yes, agents bring unprecedented speed of operation, but they also bring the potential for sometimes hidden technical debt.

This reminds me of a chart my professors showed us in our machine learning class almost a decade ago. 

Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex. — Hidden Technical Debt in Machine Learning Systems, D. Sculley et. al, 2015

Replace ML with agents, and a lot of the truth in that statement remains the same 10 years later. Let’s dissect these blocks and delve deeper into how each of these concepts could apply to agentic systems and introduce technical debt. 

Configuration → Identity and Agent Configuration

Identity

An agent, at its heart, is a large language model capable of reasoning to call the right tools for a user task. Marketing spiels promise quick return on investment across business functions like HR, Procurement, and IT, among others. The promise of an agent autonomously performing tasks like applying for leave or applying for an employment letter for an end user. But it’s not as simple as adding a few Workday APIs behind an LLM, is it?

The configuration — especially setting up proper authorization and role-based access  —  becomes critical. 

Authentication (AuthN)

How does the tool in the backend know that an agent has authorization from an end user to invoke that action? Through concepts such as on-behalf-of, or OBO, flows. The chart below shows what that flow looks like. 

Ensuring that this is configured correctly is critical. Although most identity providers rely on standard protocols like OAuth 2.0, each has its own implementation details and configuration requirements. Likewise, enterprise APIs and backend tools may expect different token types, audiences, or authentication mechanisms. A mismatch at either end can prevent the agent from successfully acting on the user’s behalf.

Authorization (AuthZ): Another consideration at this stage is the authorization and access that go beyond the establishment of identity. Does every user get access to every tool, or are the tools available to an agent determined by the user’s access rights? Some general best practices in this area are enforcing least-privileged access and limiting scope based on user session and/or role. 

AuthN and AuthZ work together to enforce end-to-end context: can this agent act on behalf of this user, and can this user perform this task in this application? [2][3]

Agent Configuration

This is where we think about the agent itself. The first decision point is the model to be used. Are the tasks simple enough to be handled by a smaller model, or are there complex decisions that need a larger model? Do we need multiple models in a multi-agent architecture? 

Large language models also come with their own set of parameters. Does model reasoning need to be set to high, leading to higher latency, or can it be set to low? How much flexibility in responses are we willing to handle with the knob set to temperature? How long or short do we want our responses to be? What’s the prompt going into the system? There is so much configuration that sits on top of the base LLM that finding the right balance of parameters can take some trial and error. However, out of all the technical debts we’ll discuss in this article, this one is probably the most apparent to all developers. 

Data Collection → Context Curation 

Just because you don’t need data to train an LLM does not mean data is not needed to leverage an LLM. An agent is only as good as the context it receives to support its internal knowledge, especially for enterprise applications where you cannot rely on its internal knowledge to respond to user queries. You need it to respond to user queries using your in-house ground truth.

Between LLMs and agents, there was a time when everyone was promoting retrieval-augmented generation (RAG) as the solution to that challenge. And it is still a prevalent part of almost every agentic system being built by large enterprises. In fact, RAG has become so pervasive that it is now being applied across a wide range of knowledge sources  —  from traditional relational databases and document stores to graph databases and other structured knowledge representations.

But context curation doesn’t stop at the context fetched through RAG. The design of the agent memory plays a big role in how the user and conversation context are carried from one query to another. LLMs have limited context memory, so additional methods can be applied to the conversation history to efficiently compress and summarize it, and to extract and store valuable user and conversation facts into context variables. 

There’s also a fine balance that must be maintained between too much and too little context. Too much context can confuse the model, leading to issues such as hallucinations, latency, and increased costs. Too little context can also lead to hallucinations and poor response quality and decision-making for the agent. 

Imagine a scenario where you have 100s of tools in your backend systems. Would you load all those tools into your model context? That’s what many naive implementations can do if the number of tools is small, but it becomes risky in an enterprise setting. These are also the kinds of issues that good context curation can help address through features like progressive discovery [1]. Feature discovery provides your agent with tool information on demand, instead of overloading the context window upfront.

Feature Extraction → Tool Cataloging and Discovery

Similar to the data exploration phase of a machine learning modeling process, there should be a tool cataloging phase where the enterprise lists all the tools needed in their agentic system. 

This exercise influences the agentic system design more than meets the eye. In an ideal agentic system architecture process, the architect and/or lead developer already know which tools they need in their system, and that knowledge influences the number of agents in the system, the overall architecture design of the system (sequential, parallel, human-in-the-loop, among others), and the design of the agent payload. If these things are not outlined before the development process begins, the risk of rearchitecting the agentic system as new tools are onboarded increases. 

Setting up the Model Context Protocol (MCP) layer would also fall under this bucket. MCP is an open-source standard for connecting agentic (or AI) applications to external systems. Using MCP, these applications can connect to data sources, tools, and workflows in a standardized way, enabling them to access key information and perform tasks. Rather than building custom integrations for every backend system, developers can expose capabilities through MCP servers, allowing agents to discover available tools and their schemas dynamically. This reduces the implementation complexity and the development cycles required to connect agents with backend systems. 

Other components that fall into this area would be general best practices for tool design and exposure. Things like using a consistent naming convention, applying the right versioning policies, converting an existing OpenAPI spec into a tool only after an audit and cleanup, and ensuring schemas are defined and documented. Inconsistent tool descriptions, poor schema design, versioning changes, and the proliferation of overlapping or redundant tools can still lead to technical debt. 

Data Verification → Guardrails and Guidelines 

There are many stories online about people misusing online AI-based chatbots for non-malicious purposes. [4] However, there are bigger concerns surrounding the idea that nefarious user prompts can override a system prompt in an LLM-based application.

Data verification is still a critical building block of an agentic system, even if the data validation looks different from a traditional ML model. Guardrails need to be put in place to detect and counter any input containing hate, abuse, or profanity, flag and escalate input containing signs of jailbreak or prompt injection, and block inappropriate questions. PII redaction may need to be implemented before the agent responds to a user, considering deploying agents in a healthcare organization operating under HIPAA regulations, for instance. 

There may also be guidelines built around the agent to ensure it adheres to business policies. In a customer refund scenario, for instance, you do not want the agent to issue a refund without checking the rules defined in its guidelines. The ideal workflow here would be for the agent to take the request, first verify the customer’s eligibility, then process the information according to its guidelines, request additional input from the customer as needed, and finally either approve the refund for simple cases or pass the request to a human for complex cases. 

Guardrails also need to be established for the output of the agent. The agent needs to be protected against responding with any hateful language. Guidelines may be defined to validate the arguments returned from a tool, and conversion to structured outputs might need to happen before a response is generated for the end user.

Just as data verification ensures an ML model operates on trusted and well-formed inputs, guardrails and guidelines ensure an agent acts after verifying the user request (guardrails), gathering sufficient information, and following the organization’s prescribed decision-making process (guidelines). 

Machine Resource Management → Model & Resource Optimization

Some of the concepts that fall under this overlap with the previous sections, but now we examine them through a slightly different lens. These are only a handful of the top highlights; there may be other concepts that also fall under this umbrella. 

Model Selection

We briefly mentioned choosing between a large and a small model according to the needs of the use case. However, developers often gravitate toward larger models because they are more forgiving of ambiguity in prompts and edge cases. But larger models also come with their large price tag. Then there is a choice between reasoning vs. non-reasoning models. And a choice between high and low reasoning for models. 

Adaptive model routing can be another way to address this challenge  —  route the query to a small model for simple tasks and to a larger model for more complex tasks. One size never fits all, and excessive computation for simple tasks introduces unnecessary cost and latency, while underpowered models can create reliability and performance issues. Running experiments is critical at this time to ensure stable, cost-effective model selection and to avoid performance- or cost-related repercussions in production.

Prompt Caching

Many agentic applications repeatedly send the same instructions, system prompts, tool descriptions, and context to the LLM. Prompt caching avoids reprocessing these unchanged prompt prefixes, reducing both inference cost and latency. Prompt caching is not available by default; it depends on both the model architecture and the deployment platform. Before deploying your agentic application to production, verify whether prompt caching is available. It could save you both time and money.  

Parallel Tool Execution

This could be either an architectural decision or an implementation choice. And this is one of those things that we get better at with experience. Let’s say you’re building a travel agent that helps an end user book travel. It needs to check the weather and respond to the user with flight and hotel recommendations before being prompted to make certain reservations. A naive approach could run these three in sequence  —  first check the weather, then book the flight, then book the hotel. That’s what feels most natural because that’s how we’d do it as humans. But an agent does not have to be limited to the  parallel-processing capabilities of humans (or the lack thereof). The workflow can trigger the three tool calls in parallel, process the results, and respond to the user with recommendations. 

Inference Budgets

Each interaction with an LLM consumes input and output tokens, making token usage a shared resource that must be actively managed. Without clear token budgets, costs can scale unpredictably as the solution gets looser. Use of larger models where smaller models could’ve sufficed, lengthy prompts that should be steps in a workflow, ten passages retrieved for a RAG solution that only needs three, or LLMs trying to call the right tool ten times before returning an error, can all lead to the exponential growth of the token usage for a solution that did not need to be so bloated in the first place. Token limits encourage smarter design choices by creating a constrained environment, and help balance response quality with latency and cost while preventing runaway execution in production.

Analysis Tools → Observability & Tracing

In ML, analysis tools include scripts and/or notebooks that can help developers troubleshoot and debug the ML pipelines. In agents, debugging happens by investigating the agent traces. 

OpenTelemetry is a common term that you’ll hear when discussing agentic evaluation. OpenTelemetry is a vendor-neutral standard for collecting and exporting traces, metrics, and logs from agentic workflows, using OpenTelemetry SDKs, the OpenTelemetry Protocol (OTLP), and compatible monitoring backends. LangFuse is a popular open-source framework for tracing and evaluating agentic workflows. 

If your agentic workflows were built using LangChain or LangGraph, you can also use LangSmith to evaluate your agents using native methods such as execution graphs, conversation replay, prompt inspection, and tool call visualization. [5]

If you thought you were flying blind without the ability to troubleshoot your ML pipelines, you’d be flying blind in the dark in an undefined subspace without the ability to investigate the trace logs for your agentic systems. While ML pipelines were deterministic to some extent, agentic pipelines are much more autonomous, and the need to understand and debug the decision-making is even higher. 

Serving Infrastructure → Inference Providers

Many infrastructure providers give developers access to the same models, and many enterprises have access to multiple providers. The question then becomes  —  how do the developers choose? 

This can be broken down into a few areas from an inference standpoint. 

Latency

Some providers offer specialized inference hardware designed to minimize latency for supported open-weight models, while others optimize for broader model availability or advanced reasoning capabilities. Providers may also employ techniques such as model quantization to accelerate inference with minimal impact on quality. Finally, deployment topology matters: network distance, regional availability, and additional network hops can all contribute to end-to-end response latency.

Cost

Providers have different pricing models and per-token costs. 

Regional Compliance

Enterprises may need to choose providers based on data residency or regulatory requirements.

Scalability & Reliability

Service availability, rate limits, concurrency support, request throughput, load balancing, and autoscaling all influence how well a provider performs under production workloads. While these differences may not be apparent during development, they become increasingly important as agent adoption grows, determining whether a system can maintain consistent performance and reliability under sustained or burst traffic.

Deployment Choices

Enterprises also need to choose between options such as Software-as-a-Service, on-premises deployment, or a hybrid deployment architecture.

There are other considerations — provider routing (similar to model routing but for providers) and failover strategies — that would also fall under this umbrella. 

Monitoring → Evaluation

This is related to the concepts covered under Observability and Tracing, but unpacks more concepts in that area, especially as they relate to monitoring agents in production. 

Setting up a solid foundation for observability during the development period is critical to ensure success with evaluation during production. The first step is to determine the metrics you need to evaluate once your agent is in production. In agentic workflows, you’re not just monitoring the LLM; you’re monitoring prompts, tool calls, latency, costs, journey failures, journey successes, and some form of user feedback. 

If you’ve implemented a RAG tool, you’d also need to evaluate specific RAG metrics like retrieval quality, generation quality, and faithfulness. If you want to capture user feedback, you’d want to build in implicit metrics — user journey completion analytics, tool usage pattern analytics, goal achievement metrics, escalation triggers, error and confusion signals, complexity indicators, and explicit metrics — “thumbs up” and “thumbs down” feedback responses. 

Process Management Tools → Workflow Orchestration

Last but definitely not least (probably the first in order of execution as it relates to agentic system design) — this is where we take a step back and ask ourselves what really needs to be agentic and how we best design our multi-agent orchestration system. This is probably one of the loosest mappings to the original paper’s concept, but it’s an important topic to discuss nonetheless. 

Unlike traditional ML systems, agentic systems don’t just make predictions; they execute multi-step workflows, introducing an entirely new class of technical debt around coordination, state management, and recovery that the original paper didn’t have to consider.

The first key decision point is what needs to be an agent and what can be a deterministic flow for an agent to leverage.  The second key decision is how the state is maintained between different agents and tools. The third decision is the flow of information and branching of business processes and decisions. Then there are decisions around retries, checkpoints, timeouts, error handling, and the points for human escalation that need to be made at this stage. 

Let’s continue to adopt agentic design best practices so that this quote does not become a forewarning for agentic systems as well. 

Acknowledgment: All the opinions in this article are my own, not those of my employer. I leveraged AI tools for my research and to generate some of the graphics in the article.