DZone Spotlight

Anyone who has tried to build a small, realistic test database from a production schema knows the drill: you don't just want "all orders." You want a customer's orders, their order items, the products they reference — but not the entire payment history, not every audit log row, not the internal reporting tables three joins away. Every table you pull in for one reason drags in three more you didn't ask for, because your schema is a graph, not a list. Jailer solves the mechanical half of this problem: give it a starting table, a condition, and a set of rules for which relationships to follow, and it will walk the foreign-key graph and hand you back a consistent, referentially valid slice of the database. What it didn't solve, until recently, was the tedious half — sitting down and deciding, association by association, what to include and what to cut off. On a schema with a few hundred tables, that's not a five-minute job. This is where Jailer's AI Subsetting Assistant comes in. It's a narrower, more interesting problem than "generate me some SQL" — and the way it's built is a decent case study in what it takes to let an LLM safely edit a structured, rule-based model instead of just emitting text. What an Extraction Model Actually Is Before the AI part makes sense, the underlying model needs to be clear, because it's not just "a query." A Jailer extraction model has three parts: A subject table – the table you start from.A condition – a WHERE clause that picks the starting rows out of that table (aliased T).A set of per-associationrestrictions – an association being a foreign-key relationship (or a user-defined one) between two tables. The part that trips people up is what happens by default: starting from the subject rows, Jailer automatically follows every association in the data model, recursively, until nothing new is reachable. That's the whole point — it's how you get a referentially consistent snapshot instead of a set of orphaned rows. But it also means the default behavior is to over-include. If you don't explicitly tell Jailer to stop at the payments table, it won't stop on its own; it doesn't know your intent, only the graph. A restriction on an association is one of three things: false (don't follow it — exclude that branch entirely), a SQL predicate (follow it, but only for rows matching the predicate, with A/B aliasing the association's source/destination tables), or empty (follow it, no filtering — the default). Building a correct extraction model is really the exercise of walking every association reachable from your subject table and deciding which of these three it needs. The extraction model for "orders of customer 42, with items and products, no payments": one subject table and condition, plus one restriction decision per association. From Prose to a Reviewable Model, Not Just SQL The AI Subsetting Assistant lives in the Extraction Model Editor — reachable from the toolbar's "AI" button, the "AI Subsetting Assistant…" menu item (Ctrl+Shift+A), or directly from the startup wizard when you're creating a model from scratch. You type a description, for example: "All orders for customer 42, with order items and products, but without payment history." and the assistant doesn't hand back a SQL query. It hands back a structured proposal for the extraction model itself: JSON { "subject": "ORDER", "condition": "T.CUSTOMER_ID = 42", "restrictions": [ {"association": "payments", "condition": "false"}, {"association": "order_items", "condition": ""}, {"association": "products", "condition": ""} ], "explanation": "Extracts orders of customer 42. Order items and products are included. Payment history is explicitly excluded." } That JSON shape is the actual response contract, not a simplification — subject table, subject condition, a restriction decision for every named association, and a plain-language explanation. The system prompt that produces this is worth a closer look, because it's essentially a compressed spec of Jailer's own traversal semantics, written for an audience that has never seen the tool before: it explains that Jailer follows every association by default, that restrictions are the only way to stop it, what the A/B aliases mean in a restriction predicate, and where a filter belongs — on the subject condition versus on a restriction — depending on which table it constrains. Getting an LLM to produce a valid, minimal restriction set for an arbitrary schema depends entirely on it understanding that asymmetry between "included by default" and "excluded by default," and the prompt exists specifically to correct for the fact that most LLM training data assumes the opposite. Why This Is Safe to Point at a Real Schema Handing an LLM the power to add or remove restrictions on a data model — the same model that determines what a production extraction pulls out of your database — is not something you want to do on blind trust. A few things here are deliberate, not incidental: Nothing is applied automatically. The proposal is rendered in a preview pane — subject, condition, and a per-association list of "exclude" / "restrict: <sql>" / default — before you touch anything. Only clicking Apply to Editor commits it, and the whole change (subject, condition, and every restriction) is grouped into a single undo step, so one Ctrl+Z reverts it completely. Beyond the review step, the dialog runs two sanity filters over the model's own response before it's even shown to you: It strips any proposed restriction on an association where the destination has to be inserted before the source (a dependent/parent relationship) — restricting those would silently break referential integrity, so the assistant refuses to let the model do it regardless of what it proposed.It drops restrictions on associations that aren't even reachable from the proposed subject table's closure — harmless, but noise that would clutter the review with decisions that don't matter. The effect is that the LLM's output is treated the way you'd treat a junior engineer's pull request: useful, plausible, worth reviewing — but not trusted to bypass the tool's own consistency rules or to go live without a look. End-to-end flow of a request through the AI Subsetting Assistant — the two sanity filters run automatically before you ever see the proposal. Scaling to Real Schemas A schema description that includes every table, column, type, and foreign key gets expensive fast once you're past a few dozen tables — both in latency and in the literal risk of blowing past a model's context window. The assistant has two independent levers for this: Reduced Schema mode splits the work into two calls. A cheap first pass sends just the list of table names and asks the model to pick the single best subject table for the request. From that table, Jailer does a breadth-first traversal of the association graph up to a configurable table limit, and only that reduced neighborhood — not the full schema — goes into the second call that actually produces the restrictions.Omit Column Types trims the per-table description further, keeping table and column names, primary keys, and foreign keys, but dropping type information that the model rarely needs to decide on a restriction anyway. The dialog also estimates the request size in tokens before sending it and flags — in the status line, not with a blocking error — when the estimate is creeping past roughly 60% of the target model's known context window, which is a small but honest thing to surface rather than let you discover as an opaque API failure. Bring Your Own Model The assistant isn't tied to one vendor. It shares its provider plumbing with Jailer's other AI features, supporting Anthropic, any OpenAI-compatible endpoint (OpenAI itself, Azure OpenAI, Groq, and similar), OpenRouter (which includes several free models), and Ollama for models running entirely on your own machine. For anyone whose schema descriptions — table and column names, sometimes revealing plenty about a business on their own — shouldn't leave the building, Ollama's no-API-key, nothing-sent-externally mode is the relevant option, and it's a first-class citizen here, not an afterthought. Both system prompts used by the assistant — the main extraction-model instructions and the lightweight subject-table-detection prompt used in Reduced Schema mode — are user-editable, with a reset-to-default button, and persist across sessions. If your schema has naming conventions or domain quirks the default prompt doesn't account for, that's the place to teach it. A Sibling, Not a Duplicate Jailer's AI Assistant dialog, reachable from the SQL Console, does something related but distinct: it generates and refactors ad-hoc SQL from natural language, with an Advisor mode for explaining and rewriting existing queries. It's built on the same request/response infrastructure as the Subsetting Assistant, but it never touches the extraction model — it writes into the SQL editor for you to review and run yourself. The two features solve different problems (querying versus configuring a repeatable extraction), and the separation is deliberate rather than a gap. What makes the AI Subsetting Assistant a more interesting case than "yet another natural-language-to-SQL box" is that it isn't generating disposable output — it's proposing an edit to a persistent, rule-based model that later runs unattended against a real database. That constraint shapes everything: the strict JSON contract instead of free text, the system prompt that front-loads the tool's actual semantics instead of assuming the model already knows them, the two hard-coded sanity filters that override the model's own suggestions when they'd violate referential integrity, and the fact that every single proposal ends at a review screen with a one-keystroke undo. For anyone building an LLM feature that edits structured application state rather than just chatting, that pattern — teach the model your domain's actual rules, then don't fully trust it anyway — is the part worth stealing. Jailer is open source under the Apache 2.0 license: github.com/Wisser/Jailer.

Every mature engineering team has a Security Incident Response Plan, refined over years of postmortems. But when the incident involves AI — a toxic hallucination with real consequences, a discriminatory algorithmic output, PII leaking out of a RAG pipeline — the old playbook stops working. There's no CVE to patch, no clean indicator of compromise. What you get instead is a probabilistic failure paired with legal escalation happening in real time. One hallucination, a thousand claims: that's the shape of the risk nobody budgeted for. Anatomy of an AI Incident The case everyone cites is Air Canada's chatbot, which in 2024 invented a bereavement fare policy that didn't exist and promised a customer a discount based on it. According to the BC Civil Resolution Tribunal's decision in Moffatt v. Air Canada (2024 BCCRT 149), the airline argued that the bot was "a separate legal entity responsible for its own actions" — a defense the tribunal simply didn't accept, ordering the company to pay damages. 2026 has provided a sharper, messier reality. In May 2026, the Higher Regional Court of Hamm (OLG Hamm) ruled — in a judgment that is not yet final (I-4 UKl 3/25) — that a cosmetic clinic was liable for its website chatbot, which falsely attributed specialist medical titles to two directors. The court established a vital principle: if you deploy a chatbot on your site, you bear the risk of its hallucinations. Just weeks later, the Munich I Regional Court (LG München I) issued a preliminary injunction (26 O 869/26) prohibiting Google from repeating specific false claims about two publishers in its AI Overviews — reasoning that AI summaries are the operator's own content, not third-party search results. Air Canada in 2024 was the warning bell; by 2026, European courts transformed the exception into the rule: the operator owns the output. Consider a plausible scenario: an LLM-based fintech assistant confidently advises users on a grey-area loophole to dodge transaction fees. Engineers, spotting the bug, instinctively push a hotfix to the system prompt and wipe the logs from the affected sessions to "clean up the mess and move on." When consumer protection regulators eventually come knocking, the company cannot produce evidence of what the model actually told users. A technical mistake has, legally, become something that looks a lot like deliberate spoliation of evidence. Why This Isn't a Security Incident With a New Label When a database goes down or a SQL injection lands, the faulty component is obvious. AI systems don't give you that clarity. Reproducibility isn't guaranteed – run the same prompt again at a temperature above 0.0, and you might get a perfectly safe, correct answer the second time.The "guilty component" is blurred – did the base model degrade, did the vector database pull poisoned context, or did a user run a clever prompt injection that slipped past the guardrails? Nobody knows in hour one.Evidence is volatile – the state of the context window, hidden system prompts, and chained API calls disappear without a trace unless you've built AI-grade telemetry into the pipeline from the start. The Runbook: Hour by Hour The first 24 hours decide whether you end up writing an internal report or fielding calls from a regulator and a journalist at the same time. 0–1 Hour: Freeze The most expensive mistake in the first minutes is an engineer quietly trying to fix things on the fly. No rollback without preserving state first. Take a full snapshot of the model (if self-hosted), system prompts, weights, and pipeline configuration, including sampling parameters like top-k and temperature.Freeze the logs for the exact sessions involved in the incident — this is where solid logging architecture pays for itself.The goal is simple: preserve evidence for root cause analysis and for regulators. Without it, you can't prove you weren't negligent, even if you weren't. 1–4 Hours: Containment Don't try to debug the model live while users keep generating fresh risk. Stop the bleeding first. Kill switch: cut the AI functionality at the API gateway level if the risk is assessed as critical.Degrade to human-only: route affected traffic to support staff using the old non-AI fallback process.Feature isolation: hard-block the specific topic (say, all financial questions) using rule-based filters while the investigation continues. 4–12 Hours: Scope Assessment and Legal Notification This is where things get genuinely difficult because the incident now has to be classified against real regulatory deadlines. Article 73 of the EU AI Act sets out reporting obligations for serious incidents involving high-risk AI systems, and the timelines are unforgiving: Standard cases: report immediately after establishing a causal link, and in any event no later than 15 days after becoming aware of the incident.Widespread infringement (or incident types defined under Article 3(49)(b)): report immediately, and no later than two days after becoming aware.Death of a person: report immediately once a causal link is suspected, and no later than 10 days after becoming aware. If the incident involves the model surfacing personal data from training or retrieval, GDPR's 72-hour breach notification window kicks in alongside the AI Act obligations. The team also needs to work out how many users were affected and whether the incident maps onto the Manage and Measure functions of the NIST AI Risk Management Framework, which auditors and, increasingly, regulators expect to see referenced in your own process. 12–24 Hours: Communication Draft the public postmortem. This window is about being transparent with users and stakeholders without conceding legal liability in the wording. Stick to the facts: safety protocols triggered, the AI feature was moved into a safe fallback mode, and an investigation is underway. Why Legal Counsel Belongs in the War Room in Hour One, Not Day Three In a typical DevOps process, lawyers get looped in when it's time to draft an apology to angry customers. With an AI incident, legal and compliance need to be in the room from hour one — and the reason is legal privilege. If engineers are hashing out the incident on a public Slack channel with messages like "our model's spouting discriminatory nonsense again, we screwed up the dataset," those messages become discoverable in litigation and can be weaponized against the company. In my practice, the very first action I take during an escalation is establishing a privileged communication channel and issuing a strict directive: "Do not diagnose the legal fault in Slack; describe only the technical symptoms." The goal isn't to obscure the truth, but to ensure that a stressed engineer's rushed hypothesis isn't treated as a binding confession of corporate negligence in a courtroom. Legal counsel redirects that conversation early, and — just as importantly — makes sure the technical rollback doesn't look like spoliation of evidence once regulators or plaintiffs start asking what happened to the original logs. Runbook Template: First 24 Hours PhaseTimeEngineering & Product ActionsLegal & GTM ActionsExpected ArtefactDetectionT+0Flag the anomaly via telemetry, behavioural analytics, or user reports.Notify C-level stakeholders.Incident ticket (Jira/PagerDuty)FreezeT+1hSnapshot prompts, logs, RAG context, and API state.Initiate legal privilege protocol.Isolated system backupContainmentT+4hTrigger kill switch / roll back to fallback process.Assess against AI Act, GDPR, and NIST "serious incident" criteria.System moved to safe modeAssessmentT+12hRun RCA, test reproducibility hypotheses.Draft regulator notification (if the 2-day threshold applies).Preliminary technical reportCommsT+24hTest the patch (filters, updated prompt).Issue customer statement, aligned with PR.Public postmortem In generative AI, technical ambition has to be matched by operational discipline. The structural difference between an engineering team that survives an AI incident and one that sinks the company lies entirely in the actions taken during the first few hours: evidence discipline and immediate legal integration. Build this runbook, establish your privileged channels, and rehearse the freeze protocol before your model's output becomes the subject of a breaking news story, not after.

Modern organizations operate under a persistent tension: they must both discover the future and deliver the present. These two modes of work — exploration and exploitation — are fundamentally different in goals, incentives, risk tolerance, and execution style. Yet both are essential for long-term success. The challenge is that most systems, teams, and incentives are not naturally designed to handle both well at the same time. Organizations that fail to balance these modes tend to collapse in predictable ways. Some become overly focused on optimization, refining existing products while missing shifts in technology or user behavior. Others become addicted to experimentation, constantly building new ideas without the discipline required to scale or sustain them. Sustainable companies learn to do both deliberately. What Exploration and Exploitation Really Mean Exploration is the process of discovering new opportunities. This includes new products, technologies, user behaviors, and markets. It is inherently uncertain. Success is measured not by stability or scale, but by learning. Exploration favors speed over perfection, and reversibility over permanence. It thrives in environments where failure is expected and inexpensive. Exploitation, on the other hand, is about scaling what already works. It is the phase where systems are hardened, performance is optimized, reliability is improved, and operational excellence becomes the focus. Exploitation favors predictability, consistency, and efficiency. It assumes that the underlying idea has already been validated and is worth investing in for long-term use. The key insight is that neither mode is superior. They are complementary, and the health of an organization depends on how well it can transition between them. Why Balance Is Difficult The difficulty arises because exploration and exploitation demand opposing behaviors. Exploration rewards experimentation, tolerance for ambiguity, and willingness to discard work. Exploitation rewards discipline, stability, and careful optimization. Teams often struggle because they try to apply the same engineering standards to both modes. If everything is treated as production-grade from day one, exploration slows down and innovation dies. If everything is treated as experimental, systems become unstable and difficult to maintain. Organizations that fail in this balance typically fall into one of two traps: Over-exploitation: Companies focus on improving existing systems until they become rigid and blind to change.Over-exploration: Companies generate many ideas but fail to turn them into reliable, scalable systems. The most successful organizations maintain what is often called organizational ambidexterity: the ability to explore and exploit simultaneously without letting one destroy the other. How Engineers Enable Exploration Engineers play a central role in making exploration safe and productive. During exploration, the goal is to maximize learning per unit of effort. This requires different design choices than those used in production systems. Key engineering principles for exploration include: 1. Optimize for Speed and Learning Early systems should prioritize rapid iteration. The goal is not correctness at scale, but fast validation of assumptions. 2. Keep Systems Lightweight and Reversible Exploration work should be easy to discard or rewrite. Heavy architecture decisions too early can slow learning and lock teams into premature constraints. 3. Use Isolation Mechanisms Feature flags, sandbox environments, and isolated services allow experimentation without risking core systems. 4. Limit Blast Radius Experimental work should be contained so failures do not cascade into production instability. 5. Treat Code as Temporary Exploration code should be written with the expectation that it may be replaced or removed entirely. The most important mindset shift is accepting that exploration is about learning, not longevity. How Engineers Enable Exploitation Once a direction is validated, the focus shifts from learning to scaling. This is where engineering discipline becomes critical. Exploitation requires different priorities: 1. Raise Quality Standards Reliability, performance, security, and maintainability become central concerns. Systems must now behave predictably under real-world conditions. 2. Simplify and Stabilize Complex experimental structures should be reduced or refactored into stable designs. What was once acceptable for speed may become unnecessary overhead. 3. Pay Down Technical Debt Shortcuts taken during exploration must be revisited. Debt that is ignored compounds and eventually slows down future progress. 4. Standardize and Automate As systems scale, consistency becomes critical. Automation, observability, and standardized patterns reduce operational burden. 5. Design for Longevity Exploitation systems should assume long-term operation. This means careful attention to interfaces, dependencies, and evolution paths. The transition from exploration to exploitation is one of the most important engineering inflection points. Many systems fail not because the idea was wrong, but because the transition was never properly completed. The Core Engineering Discipline At the center of this balance is a deceptively simple question: Are we exploring or exploiting right now? This question matters because it determines everything else — architecture, testing strategy, deployment rigor, and even communication style. When this intent is clear: Engineers can apply the right level of rigorTeams can consciously accept or reject technical debtTrade-offs become explicit instead of accidentalSystems evolve without losing coherence When this intent is unclear, teams often apply mismatched expectations. Experimental systems become over-engineered too early, or production systems remain under-documented and fragile. Clarity of intent is what enables disciplined flexibility. The Role of Product and Engineering Together The balance between exploration and exploitation cannot be managed by engineers alone. It requires close alignment with product thinking. Be Explicit About Intent Teams should clearly label work as exploratory or exploitative. This avoids confusion about expectations and quality standards. Define Success Appropriately Exploration should be evaluated based on learning outcomes: validated hypotheses, user insights, or technical feasibility. Exploitation should be evaluated based on reliability, efficiency, and scalability. Manage Technical Debt Intentionally Speed during exploration often introduces debt. The key is not to avoid it, but to make it visible and intentional, with a plan for when it will be addressed. Protect Capacity for Both Modes Healthy organizations allocate time for experimentation, operational improvement, and debt reduction. Without this balance, either innovation or reliability suffers. Make Transitions Explicit When an experiment proves successful, it should be consciously transitioned into a production system. Likewise, failed experiments should be retired decisively to avoid long-term clutter. The Bottom Line The long-term success of engineering organizations depends on their ability to explore new possibilities while reliably exploiting proven systems. This balance is not accidental — it must be designed. When exploration is clearly separated from exploitation, teams can move quickly without fear and scale confidently without chaos. Technical debt becomes a managed tool rather than an unintended burden. Systems evolve in a controlled way rather than accumulating uncontrolled complexity. Ultimately, the goal is not to choose between exploration and exploitation, but to build the discipline and systems that allow both to coexist. That is what enables continuous innovation while still delivering dependable value at scale.

The DORA framework is built on a premise so foundational that it rarely gets examined. When the research team behind it studied thousands of engineering organizations and identified the metrics that predict software delivery performance, they were measuring what pipelines report. Deployment frequency from deployment logs. Lead time from commit timestamps. Change failure rate from incident records correlated with deployment events. Failed deployment recovery time from incident resolution timestamps. Deployment rework rate from the proportion of deployments consumed by fixing previously shipped work. Every one of these metrics is a faithful record of what the pipeline said happened. None of them have a mechanism to assess whether what the pipeline said happened was accurate. This is not a criticism of DORA metrics. The framework measures what it was designed to measure. The problem is the assumption that sits underneath the measurement: that a pipeline reporting green is reporting something true about the system it is validating. In a significant number of engineering organizations, this assumption is not holding. What the Pipeline Is Actually Reporting A CI pipeline produces results based on what it is configured to check. Unit tests check whether individual functions behave as their developers specified. Integration tests verify that services interact as expected when the tests were written. The pipeline aggregates these results and reports a pass or fail. The pass or fail tells you whether the code being deployed is consistent with the assumptions encoded in the tests. It does not tell you whether those assumptions are still accurate. This distinction is invisible in normal operations. When a pipeline passes, it looks the same regardless of whether the tests are checking against current system behavior or against a snapshot of how the system behaved six months ago. The green is green either way. The DORA metrics that depend on pipeline accuracy are the ones most directly affected by this distinction. Change failure rate measures the percentage of deployments that cause an incident requiring remediation. If tests are passing against stale assumptions about how downstream services behave, changes that will cause production incidents are getting through the pipeline and being counted as successful deployments- right up until the incident that reclassifies them. The change failure rate reflects what the pipeline caught. It does not reflect what the pipeline missed because it was not checking against current reality. Deployment rework rate, the fifth DORA metric introduced in the 2024 research, captures the proportion of deployment activity consumed by fixing previously shipped work. It is the metric that most directly surfaces when change failure rate is underreported. When teams notice their deployment rework rate is high while their change failure rate appears controlled, the gap between those two numbers is often a signal that failures are occurring after the immediate post-deployment window- in the period when stale test coverage cannot catch them but real usage can. "A note on reliability: the 2022 State of DevOps Report introduced reliability as an additional outcome dimension assessed through SLOs and SLIs. It is tracked separately from the five core delivery metrics rather than as part of the primary measurement framework. The five metrics that form the core DORA framework are the four original metrics plus deployment rework rate. Where the Staleness Comes From The mechanism that causes test coverage to drift from reality is well understood by any team that has run distributed systems at scale. It is less well understood as a systematic source of inaccuracy in DORA metrics. In systems where services deploy independently on their own schedules, the mock files and integration assumptions that tests run against were accurate when they were written and become less accurate with every deployment of a downstream service. A service changes its error response format. A dependency updates its authentication behavior. An upstream API adds a required field that existing integrations do not send. Each of these changes is correct from the deploying service's perspective. Each of them potentially invalidates assumptions in the test suites of services that depend on them. The consuming service's tests keep passing because they are not running against the updated service. They are running against a mock that reflects the updated service's behavior as of the last time someone thought to update it, which may have been before several of the changes that have since occurred. In this state, the pipeline is not checking whether the deployment is compatible with the current system. It is checking whether the deployment is compatible with a historical snapshot of the system. The two checks are not the same, and DORA metrics cannot distinguish between them. What This Does to Each Metric The effect on individual DORA metrics is specific enough to be worth tracing through. Deployment frequency is the metric least affected by pipeline accuracy. It measures how often deployments happen, which is an observable fact regardless of whether those deployments were properly validated. Lead time for changes is similarly unaffected. The time from commit to production is a timestamp measurement that does not depend on the accuracy of what was validated during that time. Change failure rate is where the inaccuracy concentrates most visibly. A deployment that passes a pipeline running against stale assumptions but causes a production incident days later gets counted as a failure. However, deployments that cause subtle degradation- service interactions that are slightly wrong, error conditions that are handled incorrectly because the error format changed- may not generate an incident that gets correlated with the deployment at all. They surface as unexplained production issues or as elevated user error rates that get investigated independently. These do not enter the change failure rate calculation. The metric underreports the actual failure rate in proportion to how far the test coverage has drifted from current system behavior. Failed deployment recovery time measures recovery after incidents are declared. If incidents that originate in stale test coverage are not recognized as deployment-related, they also do not enter this calculation. The metric stays clean while the underlying reliability erodes. Deployment rework rate is the most sensitive indicator of this pattern. It catches the work that the other metrics do not- the fixes deployed in response to issues that passed the pipeline, the hotfixes for behavior that tests did not catch, the rollbacks for failures that only manifested under real usage patterns the test suite never encountered. When the deployment rework rate rises while the other four metrics remain stable, the most common explanation is that the change failure rate measures a subset of actual failures rather than the whole. What Accurate Pipelines Actually Look Like A pipeline that is genuinely telling the truth about deployment safety shares a property that is straightforward to describe and requires deliberate investment to achieve: the assumptions it validates against reflect how the system currently behaves. For unit tests, this is relatively automatic. The code is the specification, and the tests validate against the code. When the code changes, the tests break and require updating. The feedback loop is tight. For integration tests that span service boundaries, this is structurally harder. The specification is a mock file that someone wrote to represent a downstream service's behavior. The downstream service continues to change on its own schedule. The mock does not update automatically. The gap between specification and reality accumulates silently. The teams whose pipelines are telling the truth about integration behavior have addressed this gap architecturally rather than through process discipline. Instead of maintaining static specifications of how downstream services should behave, they derive their integration test coverage from observed real behavior. When a downstream service changes, new observations automatically update what the integration tests run against. The coverage stays grounded in how services currently communicate rather than in how they communicated when someone last thought to update a mock file. This is the approach modern tools like Keploy take for API-driven systems. Rather than asking engineering teams to maintain mock files that represent downstream service behavior, it captures real traffic between services and generates test cases and dependency mocks from those actual interactions. When a downstream service changes its behavior after a deployment, the next round of captured traffic reflects that change. The pipeline validation running against Keploy-generated coverage is validating against current reality rather than against a historical specification. The change failure rate it contributes to reflects actual deployment safety rather than deployment safety as measured against assumptions that may have become outdated between the time they were written and the time the deployment ran. The distinction matters for DORA metrics specifically because DORA metrics are only meaningful relative to the accuracy of the pipeline they are measuring. Deployment frequency, lead time, change failure rate, failed deployment recovery time, and deployment rework rate are all accurate representations of delivery performance when the pipeline is checking current system behavior. They are flattering but incomplete representations of delivery performance when the pipeline is checking historical assumptions. The Metric That Tells You Which Situation You Are In Teams that want to assess whether their DORA metrics are reflecting reality rather than pipeline assumptions have a relatively direct way to check. Compare change failure rate against deployment rework rate over a rolling window. If change failure rate is low and deployment rework rate is also low, the pipeline is catching problems before they become incidents, and the rework burden is proportionally small. This is the pipeline pattern that accurately assesses deployment safety. If change failure rate is low but deployment rework rate is elevated, the gap between them is the most direct signal available that the pipeline is missing failures that real usage is finding. The deployments look clean by the metric that catches immediate post-deployment failures. The rework burden reveals that the failures are occurring on a delayed timeline that the pipeline was not designed to detect. The action this pattern calls for is not optimizing the four deployment metrics individually. It is examining where the pipeline's coverage is making assumptions that the system has since violated. The improvements to DORA metrics that come from closing that gap are not optimizations of the measurement. They are improvements to the actual delivery performance the metrics are supposed to be measuring. DORA metrics are a reliable indicator of delivery performance when the pipeline they are measuring is reliable. Making the pipeline reliable requires more than fast feedback loops and automated deployments. It requires that the feedback the pipeline provides reflects the system as it currently exists rather than as it existed when the tests were written. That is the assumption worth examining before concluding that a green pipeline is telling the truth.

Picture a business-critical SQL query crawling for seven hours. Nearly a full workday. The system keeps grinding through data, the business keeps losing time and money, and users are stuck waiting. Then a performance engineer steps in. After a few hours of careful analysis and a handful of precise code changes, the same query finishes in two minutes. Situations like this are not unusual in performance engineering. Turning hours into minutes is exactly the kind of work that makes this discipline valuable. In modern DevOps environments, where systems are deployed continuously and workloads change quickly, this type of work becomes part of everyday engineering practice. Who Are Performance Engineers? In simple terms, a performance engineer (PE) is responsible for making IT systems run better: faster, more reliably, and more efficiently. Behind this simple definition, however, lies a complex and multifaceted discipline. The bottleneck can appear almost anywhere in the stack: in application code, database configuration, network communication, or even the underlying hardware. And sometimes the bottleneck is not in the database or the application, but in the operating system. When systems handle thousands of concurrent network connections, limits may appear in the OS network stack or in kernel parameters. There are well-known cases in the history of database systems where the same database engine showed dramatically different performance on different operating systems, such as Windows, FreeBSD, or Linux. These differences were often caused by variations in filesystem behavior, networking stacks, and kernel-level I/O scheduling rather than by the database software itself. Once the root cause is identified, the performance engineer must understand the underlying mechanism behind it and propose an effective solution. Sometimes it means tuning the configuration. Sometimes it means rewriting a query or changing application behavior. Sometimes it points to a deeper architectural flaw that was hidden until the load exposed it. That is why the job often feels less like optimization in the abstract and more like investigation under pressure. And it sits somewhere between development, systems administration, and deep system analysis. It is important to note that many performance engineering tasks overlap with the responsibilities of a database administrator. Query optimization, lock analysis, and tuning parameters such as WAL settings are traditionally part of a DBA’s role. The difference is that a performance engineer usually operates at a broader level. They analyze the performance of the entire system, including the application, database, operating system, network communication, and underlying hardware. While a DBA focuses on a specific database platform, a performance engineer evaluates the system as a whole production pipeline. In cloud environments, this broader view may also touch tools when workload, container, or configuration findings overlap with production behavior. A Practical Example Performance problems rarely have a simple playbook. The same symptom can appear in different environments while the underlying cause is completely different. Engineers, therefore, rely on ongoing microlearning, hands-on experimentation, and careful analysis of system metrics to expand their troubleshooting knowledge as part of everyday engineering practice. One example illustrates how these investigations unfold. A client was migrating data from Oracle to PostgreSQL. The migration process relied on massive parallel data loading using COPY. At first, everything seemed to work normally, but eventually the process slowed dramatically. The investigation showed that the bottleneck was due to WAL (Write-Ahead Log) writes. In PostgreSQL, every change generates a WAL record that must be flushed to disk before the transaction commits. This mechanism guarantees durability and crash recovery, but under heavy write workloads, it can become a limiting factor. Initially, the team suspected that disk throughput was the problem. Developers even suggested a patch intended to speed up WAL writing. The patch did not improve performance. The client’s internal specialists were also unable to find a clear explanation. At that point, the performance team started analyzing the system in more detail. They noticed that many database sessions were waiting on the PostgreSQL wait event LWLock:WALInsert. That observation changed the direction of the investigation. It meant the system was not actually saturated by CPU or disk throughput. Instead, multiple processes were competing for internal synchronization while inserting WAL records. The migration workload involved hundreds of concurrent COPY operations. Each process attempted to reserve space in WAL buffers, which created contention around WAL insertion locks. The team experimented with several configuration parameters and eventually increased wal_buffers and wal_writer_flush_after. This allowed PostgreSQL to accumulate larger WAL batches in memory before flushing them to disk. The result was a significant reduction in contention around WAL insertion and about a 30 percent improvement in migration throughput. It is important to note that these changes are not universally safe defaults. Larger WAL buffers and less frequent flushing can increase the amount of data at risk during an unexpected crash. In this case, the workload was a migration. If the process stopped, it would have to be restarted anyway. Under those conditions, the temporary trade-off between reliability and performance was acceptable. The real lesson from this case is not a specific configuration value but the investigation process: identify where the system is actually waiting, test hypotheses, and verify improvements with measurements. Of course, real PE cases are often more complex than the simplified example shown here. In practice, investigations can take days and may involve analyzing internal database behavior, operating system limits, and network interactions to identify the true bottleneck. Common Performance Engineering Rules In performance engineering, there is an informal set of principles that experienced engineers tend to follow: 1. Proactivity Is the Best Prevention A performance engineer does not wait for a system to fail under load. The work starts earlier: analyzing the architecture of new services, anticipating how the system will behave as load grows, and identifying potential bottlenecks before they become production incidents. 2. Trust Metrics A common mistake, especially among less experienced engineers, is optimizing by eyeballing results. Someone changes a configuration or piece of code and says, “It seems to run three to five seconds faster.” That is not acceptable. Improvements must be confirmed with measurable data. Engineers compare metrics before and after a change: transactions per second (TPS), latency, CPU utilization, disk and memory usage, and queue lengths. Only these measurements can demonstrate whether performance has actually improved. Metrics matter more than subjective impressions, although experience still plays a role. Experienced engineers often use intuition to form an initial hypothesis about the cause of a problem. However, every hypothesis must be verified with measurements. Intuition helps guide the investigation, while metrics confirm the correctness of the solution. 3. Be Careful With Quick Fixes Sometimes incidents must be resolved immediately. A common example involves the max_connections parameter in PostgreSQL, which defines the maximum number of concurrent connections. When a system slows down under load, some developers try to increase max_connections. This can help temporarily, but it often creates new problems. A sudden increase in connections raises contention for internal database resources such as shared memory structures and locks. As contention grows, performance can degrade significantly due to locking and resource pressure. A quick fix can easily turn into a larger failure. A good performance engineer will highlight these risks and recommend a more systematic solution. Becoming a Performance Engineer Few people start their careers aiming for performance engineering. More often, they drift into it through a difficult problem that refuses to stay contained. That is how it happens in practice. A developer helps compare database options for an important project. Then the questions start multiplying. What should be measured? On physical servers or virtualized infrastructure? Which metrics matter? What changes under load? What changes only in production? One question leads to ten more. Before long, the person who thought they were helping with a tactical decision is working at the boundary between software, systems, and operational behavior. That path is common. People grow into it from development, systems administration, or operations. Nobody really graduates as a ready-made performance engineer, yet those who grow into the role often reach compensation levels that can support stronger long-term financial outcomes than many other career paths. Core Skills of Performance Engineers Because performance engineering sits at the intersection of software and infrastructure, practitioners usually combine skills from several technical disciplines. What does it take to move into this field? Programming A performance engineer needs to understand how software is written, how developers think, and what challenges they face. In many cases, the engineer works with tools built for other developers. Linux Strong Linux knowledge is very important: how the kernel works, how the user space operates, how processes are managed, and which operating system metrics can be measured. Algorithms Understanding algorithms and their complexity is essential for proposing efficient solutions. Math Another important but often missing skill is mathematical statistics. When an improvement is not dramatic but only one to two percent, engineers must prove that the change is meaningful and not just measurement noise. Concepts such as quantiles, percentiles, data distributions, and multimodal behavior help separate real improvements from measurement noise. Communication Performance engineers must clearly and carefully communicate findings to developers, testers, and business stakeholders. Explaining that a problem originates in someone’s code can be sensitive, so it must be done constructively. Attention to Detail Attention to detail is critical. An unusual spike in a graph or a repeating system pattern may point to the root cause of a problem. Persistence also matters. Test results can fluctuate due to environmental factors, so identifying the real issue often requires patience and careful investigation. The same applies to client-side performance work, where telemetry, crash analytics, and app data collection can help explain how the product behaves on real devices, networks, and usage patterns. The Future of Performance Engineering Systems are becoming more complex, and no single engineer can be an expert in every layer. As a result, the field is moving toward greater specialization. We are likely to see performance engineers focused on specific areas: application-level performance, operating system behavior, or hardware-level optimization, such as selecting the right CPU for a workload and tuning CPU frequency settings. What about AI? So far, there are no real tools capable of replacing performance engineers. AI can help engineers find information faster, although its output still needs verification. It does not yet solve complex analysis and optimization tasks. Automated tuning systems also do not currently appear capable of replacing human expertise. There is some expectation that AI will at least automate routine work. For now, most performance engineers see AI as an assistant rather than a threat. Final Thoughts: The Hunt Continues Performance engineering is a constant challenge. It is an intellectual puzzle with real operational impact. The work involves identifying hidden patterns, uncovering non-obvious relationships, and finding effective solutions where others see only complexity or system limits. Many engineers remember their first major optimization success. A query that once ran for minutes or even hours suddenly runs hundreds of times faster. Moments like this leave a lasting impression and often keep engineers in this field for many years. These experiences are what make performance engineering a difficult yet highly engaging profession, where the hunt for CPU cycles and response-time improvements never really ends.

By Alex Vakulov CORE

When I first started building enterprise applications with Large Language Models (LLMs), I fell into a trap that almost every developer encounters. I thought that scaling an AI system simply meant refining a single, massive prompt. I wrote complex system instructions, packed the context window with rules, and expected a single stateless API call to act as a researcher, analyst, and copywriter all at once. In production, this monolithic approach failed repeatedly. When processing dynamic data streams, the model flattened nuanced details, skipped critical execution steps, and regularly generated highly confident hallucinations. Through these failures, I realized the core problem: we are expecting a single inference step to manage an entire engineering workflow. To build predictable, production-grade software, I had to redesign my architecture. I moved away from monolithic prompts and began decoupling complex tasks into role-based, multi-agent frameworks in Python. My Breaking Point: The Competitive Intelligence Engine Failure Problem The necessity of this architectural shift became clear to me during a deployment for an enterprise technology firm. My team was tasked with building a competitive intelligence engine to track daily competitor product launches, analyze changing pricing sheets, and generate technical battlecards for our global sales team. My first iteration used a single, closed-source model wrapper. The prompt instructed the LLM to read raw HTML fragments from target URLs, extract feature updates, compare them against our internal capabilities matrix, and output a structured battlecard. During local testing with a few static URLs, it worked well. But when I went live against a shifting market, the system kept breaking without much notice: The Production Vulnerabilities I Encountered Context flattening: When parsing multiple long competitor pricing tiers, the model routinely dropped nuanced constraints, such as specific seat-count thresholds. It simply averaged out the data. Severe information loss: Instead of extracting the live web data provided in the context window, the model slipped back into its static pre-training data, hallucinating older features that the competitor had deprecated months prior. Prose without substance: Because the model had to handle data extraction, comparative reasoning, and copy editing simultaneously, it prioritized linguistic fluency over technical depth. The output looked like excellent marketing prose, but it was factually useless to our sales engineers. To fix this, I completely dismantled the monolithic prompt. I decoupled the system into three distinct programmatic agents, creating a clear engineering pipeline: Step 1: Establishing a Model-Agnostic Execution Boundary When I design multi-agent systems, my first rule is that agents must be decoupled from specific model providers. A production agent should depend on a stable, programmatic interface. This approach allows me to swap a cloud API like OpenAI for a local, open-weights model running via Ollama without changing a single line of business logic. Here is the standardized execution node I developed for this framework: Python import os from openai import OpenAI # I initialize the client container using environment boundaries client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) def execute_agent_inference(messages: list, target_model: str = "gpt-4o-mini") -> str: """ Provides a standardized execution node for all upstream agents to communicate with the designated model endpoint. """ response = client.chat.completions.create( model=target_model, messages=messages, temperature=0.1 # Low temperature enforces deterministic reasoning ) return response.choices[0].message.content Step 2: The Strategist Agent (Task Decomposition) The execution loop begins with the Strategist Agent. I isolated this node to handle a single cognitive task: ingestion and planning. Its sole job is to break down a broad user request into a chronological sequence of distinct tasks. Python def strategist_agent(user_objective: str) -> list: """ Ingests a broad objective and returns a structured execution plan. """ system_prompt = """ You are a project strategist. Your job is to break down a broad research objective into an ordered, numbered list of specific, non-overlapping data requirements. Do not summarize the topic. Output only the numbered steps. """ messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": f"Objective: {user_objective}"}, ] raw_plan = execute_agent_inference(messages) # Parse the numbered rows into a clean Python list return [line.strip() for line in raw_plan.split("\n") if line.strip()] By forcing the system to map out its roadmap before running any resource-heavy tasks, I ensure the application maintains a strict operational scope. Step 3: Integrating External Tools With the Extraction Agent An agent is only as good as the data it consumes. I designed the Extraction Agent to never guess or extrapolate. Instead, I equip it with specific Python functions that fetch live, real-world data before it runs an inference cycle. Here, I define a simulated web search utility and an internal vector store look-up tool: Python def fetch_live_web_data(query: str) -> str: """ Simulates a live web lookup via external search providers like Tavily or SerpAPI. """ return f"[Live Web Match] Found current market documentation regarding: {query}" def query_internal_vector_store(query: str) -> str: """ Simulates a vector database query for internal technical specifications. """ return f"[Vector DB Match] Internal baseline spec data for: {query}" def extraction_agent(allocated_task: str, running_context: str) -> str: """ Gathers factual data using external retrieval tools before forming response notes. """ # Execute the tools first to ground the agent's context in real data web_insights = fetch_live_web_data(allocated_task) internal_insights = query_internal_vector_store(allocated_task) system_prompt = """ You are a data extraction agent. Your job is to analyze tool outputs and compile precise, evidence-dense technical notes. Strictly ground your response in the provided tool outputs. Do not extrapolate. """ user_payload = f""" Current Task: {allocated_task} Prior Context: {running_context} Tool Outputs: - {web_insights} - {internal_insights} """ messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_payload}, ] return execute_agent_inference(messages) Step 4: The Technical Reviewer Agent (Synthesis and Audit) The final step in my pipeline is the Technical Reviewer Agent. I do not use this agent as a passive text formatter. Instead, I design it to act as an internal critic that actively checks the gathered research for missing technical data. Python def technical_reviewer_agent(compiled_research_notes: str) -> str: """ Audits research materials and synthesizes a structured final technical report. """ system_prompt = """ You are a technical reviewer. Synthesize a clean report from the provided research notes. CRITICAL RULES: 1. Organize your output using clear Markdown headings and bullet points. 2. Do not introduce general knowledge or unverified claims. 3. If the data contains gaps, note them explicitly instead of smoothing over them. """ messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": f"Research Notes:\n{compiled_research_notes}"} ] return execute_agent_inference(messages) Step 5: Constructing the Orchestration Loop With all my agents built, I put them together using a central orchestrator function. This loop manages the execution sequence, updates the running memory context between steps, and passes state across agent boundaries. Python def run_intelligence_engine(target_topic: str) -> str: """ Coordinates the execution sequence, updates persistent memory boundaries, and returns the finalized asset. """ print(f"[*] Initializing Strategy Phase for: {target_topic}") execution_steps = strategist_agent(target_topic) accumulated_notes = [] persistent_memory = "" for idx, step in enumerate(execution_steps, 1): print(f"[>] Executing Phase {idx}: {step[:50]}...") # Pass the running context so the agent knows what has been researched so far step_output = extraction_agent(step, persistent_memory) accumulated_notes.append(step_output) # Update the persistent memory to prevent duplicate work in later steps persistent_memory += f"\n[Completed Phase {idx} Info]: {step_output}\n" print("[*] Compiling and Reviewing Final Deliverable...") final_report = technical_reviewer_agent("\n".join(accumulated_notes)) return final_report if __name__ == "__main__": report_output = run_intelligence_engine( "Analyze competitor pricing models for cloud infrastructure shifts" ) print("\n--- Final Report Output ---\n") print(report_output) Resolving the Hidden Challenge: Information Degradation When I first launched a framework like this, I noticed a subtle engineering issue: the information handoff problem. When multiple agents pass unstructured text back and forth, the data risks losing clarity at each step. If the Strategist designs broad steps, the Extractor returns summarized notes, and the Reviewer formats them aggressively, the final output loses its technical precision. To keep your multi-agent networks at their best in production, I recommend implementing these two programmatic practices: 1. Maintain Strict Structural Memory Controls Never pass a raw conversational history across agent boundaries. Instead, require your extraction nodes to return explicit, structured technical updates (such as clear key-value maps or clean markdown bullet records). This approach preserves specific variables like precise pricing values or hardware specs, all the way to the final synthesis step. 2. Implement Automated Validation Gates Do not use an LLM to check if its own output is correct. Instead, place deterministic variable Python validation gates between agent handoffs. I write small programmatic checks to verify that the text matches a required schema, meets minimum character counts, or contains key terms extracted from the retrieval tools before letting the pipeline proceed. Measurable Production Outcomes Transitioning our enterprise tracking engines from monolithic prompt templates to this decoupled multi-agent architecture delivered immediate, verifiable improvements across our core operational metrics: Drastic reduction in hallucination rates: By isolating the extraction agent and grounding its context entirely in live tool calls, our documented hallucination rate fell from 7.2% to less than 0.2%.System traceability: When an output degrades, my team and I no longer dig through thousands of lines of a single prompt history. We simply look at the independent logs of each agent to find exactly where the data chain broke, reducing our Mean Time to Resolution (MTTR) from hours to minutes.Operational maintainability: I can update, optimize, or replace individual components, such as updating a web scraping API or refining the Reviewer's styling guide, without breaking or re-testing the rest of the application ecosystem. Conclusion The true test of an enterprise AI application is not how well it runs a basic query on a local development machine. Real success is defined by how reliably the application handles messy, dynamic data in production over time. By separating monolithic prompts into a coordinated pipeline of role-based agents, I turned unpredictable model outputs into stable, dependable software infrastructure. A perfect framework lies in distributing cognitive responsibility, creating clear interfaces, and engineering strict control boundaries around your models. Thank you for reading. Designing multi-agent AI systems for enterprise LLM workflows goes beyond calling powerful models; it requires thoughtful system design, coordination between agents, strong observability, and scalable architecture that can operate reliably in production.

A humanoid robot you can build with a desktop 3D printer is lowering the barrier to experimenting with machines that usually cost far more. Researchers at the University of California, Berkeley, have developed the Berkeley Humanoid Lite, an open-source humanoid robot designed to give students, hobbyists and researchers a cheaper way to experiment with robotics. The roughly 1-meter robot weighs about 16 kilograms and costs less than $5,000 in hardware, according to UC Berkeley Engineering. That is still a serious expense, but far below the cost of many commercially built humanoid platforms. More importantly, Berkeley is not selling it as a finished robot. The project is meant to be a starting point that people can build, modify, and learn from. The robot uses a modular actuator built around a brushless DC motor, magnetic encoder and 3D-printed cycloidal gearbox. The largest printed components fit within a standard 200-by-200-by-200-millimeter desktop 3D printer, while the rest can be sourced from common online suppliers. The hardware is only half the story The Berkeley team has also released the robot's hardware design, embedded code, training and deployment frameworks as open source. That makes the project potentially useful beyond humanoid robotics. The modular actuators can be used individually and adapted to different configurations, including bipedal and quadruped designs, according to Interesting Engineering. The researchers tested the actuators for efficiency and durability. Interesting Engineering reports that the gearbox reached about 90% mechanical efficiency under most conditions, while a 60-hour endurance test showed gradually increasing backlash as the printed components wore. The robot has also demonstrated basic walking and object manipulation. Researchers used reinforcement learning for locomotion and a VR-based teleoperation system for tasks including moving objects and solving a Rubik's Cube. The Berkeley team, however, acknowledges that its walking remains imperfect, while the long-term effects of heat and wear on the 3D-printed structure require more study. What eWeek found: The real opportunity is the actuator The most interesting part of Berkeley Humanoid Lite may not be the humanoid at all. Its modular actuator could become the project's most useful research and educational building block because developers can experiment with a single robotic joint before committing to an entire machine. That lowers both the financial and technical risk of getting started. The open-source design removes another barrier, but hardware ecosystems do not grow from design files alone. Berkeley still needs a community that builds the robot, documents failures, improves components, and makes those changes useful to the next person. If that happens, Berkeley Humanoid Lite could become more valuable as a platform than as a single robot. Its biggest contribution may ultimately be creating a repeatable way for students and smaller robotics teams to learn how humanoids are built from the joint up. Editor’s note: This article originally appeared on our sister publication, eWeek.

A few months into any serious RAG deployment, most engineering teams tend to hit the same wall. Our AI-powered payment assistant demo worked beautifully. The pilot with twenty internal users worked beautifully. Then it goes live, someone asks a question about their medical benefits or a wire transfer that failed, the model confidently makes something up, and suddenly "chatbot" is a word nobody on the team wants to hear in a postmortem. The fix isn't a smarter prompt. It's an architecture that assumes the model will occasionally be wrong, occasionally be asked something it shouldn't answer, and occasionally need to get out of the way entirely and hand the person off to a human. Here's how I'd structure that system if I were building it today. The Shape of the System Architecture for a Chatbot At a high level, you want to keep the stateless parts of your stack (the API gateway, the UI) separate from the stateful parts (retrieval, guardrails, escalation). That separation is what lets you scale, debug, and replace pieces independently later. Four layers do the real work: Ingestion and the vector pipeline – turning your documentation into something searchable.Safety and policy guardrails – a layer that inspects both what goes into the model and what comes out, before either reaches the user.Orchestration and retrieval – the part that actually assembles a grounded, relevant prompt.Deterministic escalation – a rules-based off-ramp to a human that doesn't depend on the LLM deciding it's confused. That last point matters more than it sounds like it should. If the only thing standing between a distressed user and a real person is a language model's judgment, you don't have a safety system — you have a hope. System Architecture Overview To maintain responsiveness and scalability, document ingestion pipelines are decoupled from the query-and-response execution flow. Core System Design Decisions Asynchronous framework (FastAPI): Utilizing Python’s async/await primitives prevents blocking the event loop during network I/O operations to external embedding models and LLM providers.Persistent vector database (Chroma DB): Indexing embeddings directly to disk (./chroma_db) guarantees data durability across service deployments without requiring complete document re-ingestion.History-aware query reformulation: User follow-up prompts (e.g., "How much does it cost?") rely on implicit conversation state. A specialized sub-chain evaluates the chat history to rewrite ambiguous inputs into fully standalone semantic queries ("What is the cost of the Senior Care Essential plan?") prior to executing similarity retrieval. Retrieval: Getting the Right Chunks in Front of the Model Most RAG failures aren't generation failures. They're retrieval failures wearing a generation costume — the model looks like it hallucinated, but really it just never saw the right paragraph. A few things that consistently move the needle: Incremental crawling over your actual documentation and service catalogs, rather than a one-time dump that goes stale within a month.Semantic chunking around roughly 1,000 characters with a 150-character overlap, so you're not slicing a procedure in half at a chunk boundary and losing the context that made it make sense.Metadata-aware vector storage (Pinecone, Qdrant, or pgvector if you want to stay inside Postgres) so you can filter by tenant, region, or document version instead of searching your entire corpus for every query.Hybrid search — sparse BM25 alongside dense cosine similarity — followed by a reranking pass (Cohere Rerank works well here). Pure vector search is good at "similar meaning" and bad at "exact term," which is a problem when someone types a SKU number or an error code. None of this is exotic. It's just the difference between a retrieval layer that was tuned once and one that's actually maintained. Guardrails: The Layer That Should Scare You a Little This is where the architecture earns its keep, and where most home-grown chatbots quietly cut corners. PII masking needs to happen on the way in, before anything — credit card numbers, SSNs, anything identifying — touches a vector lookup or gets shipped to a third-party model API. Critical trigger overrides are the part people underestimate. If someone's message contains language suggesting a safety threat, active fraud, or physical harm, that should never be routed through the LLM's judgment first. A rule-based classifier catches it and serves a static, pre-approved protocol response immediately — no generation, no ambiguity, no chance of the model getting creative at the worst possible moment. Regulated-domain disclaimers — medical, legal, financial — should be inserted automatically whenever intent classification detects the conversation drifting into that territory, and the system prompt should explicitly forbid the model from offering advice in those categories rather than relying on it to remember. Tools like NeMo Guardrails or Llama Guard are built for exactly this: sitting between the user and the model, and between the model and the user again, checking both directions. Ingestion Engine and Vector Search Pipeline A key factor in retrieval quality is selecting proper document chunk boundaries. Overly large chunks dilute document embedding clarity, whereas tiny chunks fail to retain operational context. A balance of 1,000 characters per chunk with a 150-character sliding overlap ensures contextual continuity across document splits. Python # rag_engine.py import os from dotenv import load_dotenv from langchain_community.document_loaders import PyPDFLoader, TextLoader from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_openai import OpenAIEmbeddings, ChatOpenAI from langchain_chroma import Chroma from langchain.chains import create_history_aware_retriever, create_retrieval_chain from langchain.chains.combine_documents import create_stuff_documents_chain from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder load_dotenv() class RAGEngine: def __init__(self, persist_dir: str = "./chroma_db"): self.embeddings = OpenAIEmbeddings(model="text-embedding-3-small") self.llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1) self.vector_store = Chroma( persist_directory=persist_dir, embedding_function=self.embeddings ) self.rag_chain = self._compile_chain() def ingest_document(self, file_path: str) -> int: """Parses, splits, and embeds documents into the persistent Chroma store.""" loader = PyPDFLoader(file_path) if file_path.endswith(".pdf") else TextLoader(file_path) docs = loader.load() splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150) splits = splitter.split_documents(docs) self.vector_store.add_documents(documents=splits) self.rag_chain = self._compile_chain() # Hot-reload retriever chain return len(splits) def _compile_chain(self): """Constructs a two-stage conversational retrieval pipeline.""" retriever = self.vector_store.as_retriever(search_kwargs={"k": 4}) # Stage 1: Contextualize Query context_prompt = ChatPromptTemplate.from_messages([ ("system", "Given a chat history and the latest user query, reformulate it into a standalone query. Do NOT answer the question."), MessagesPlaceholder("chat_history"), ("human", "{input}"), ]) history_retriever = create_history_aware_retriever(self.llm, retriever, context_prompt) # Stage 2: Grounded Generation qa_prompt = ChatPromptTemplate.from_messages([ ("system", "Answer strictly using the retrieved context below. If the answer is not present, state that you do not know.\n\nContext:\n{context}"), MessagesPlaceholder("chat_history"), ("human", "{input}"), ]) doc_chain = create_stuff_documents_chain(self.llm, qa_prompt) return create_retrieval_chain(history_retriever, doc_chain) def query(self, question: str, chat_history: list = None): """Executes retrieval-augmented generation across historical session state.""" response = self.rag_chain.invoke({ "input": question, "chat_history": chat_history or [] }) sources = list({doc.metadata.get("source", "Unknown") for doc in response.get("context", [])}) return {"answer": response["answer"], "sources": sources} REST API Service Layer The application server manages HTTP request parsing, payload validation via Pydantic, temporary file execution, and state translation between native JSON payloads and LangChain Message primitives (HumanMessage, AIMessage). Python # main.py import shutil, os from fastapi import FastAPI, UploadFile, File, HTTPException from pydantic import BaseModel from typing import List, Optional from langchain_core.messages import HumanMessage, AIMessage from rag_engine import RAGEngine app = FastAPI(title="Production RAG Engine", version="1.0") engine = RAGEngine() class MessagePayload(BaseModel): role: str # "user" or "assistant" content: str class QueryPayload(BaseModel): question: str chat_history: Optional[List[MessagePayload]] = [] @app.post("/upload") async def handle_upload(file: UploadFile = File(...)): """Ingests a document file (.pdf or .txt) into the vector space.""" if not (file.filename.endswith(".pdf") or file.filename.endswith(".txt")): raise HTTPException(status_code=400, detail="Unsupported file format.") temp_path = f"./temp_{file.filename}" with open(temp_path, "wb") as buffer: shutil.copyfileobj(file.file, buffer) try: chunks = engine.ingest_document(temp_path) return {"status": "success", "filename": file.filename, "chunks_indexed": chunks} finally: if os.path.exists(temp_path): os.remove(temp_path) @app.post("/chat") async def handle_chat(payload: QueryPayload): """Processes conversational questions against the context store.""" history = [ HumanMessage(content=m.content) if m.role == "user" else AIMessage(content=m.content) for m in payload.chat_history ] return engine.query(question=payload.question, chat_history=history) Managed Platform or Custom Build? This is a real tradeoff, not a formality: DimensionManaged (Voiceflow, CustomGPT)Custom (FastAPI + LangChain + pgvector)Time to launchDaysWeeksData privacy / on-premBounded by vendor's SOC2/HIPAA postureFull control, self-hostedIntegrationWebhook-basedNative RPC/gRPC/DB driversRouting logicPre-built rule UIFull graph state control (LangGraph, LlamaIndex) If you're regulated, handling sensitive data, or need routing logic more complex than "if sentiment negative, escalate," the custom path pays for itself. If you need something live for a trade show next week, it doesn't. What to Actually Watch in Production Environment Three metrics matter more than the rest combined: Faithfulness – is every claim in the output actually supported by a retrieved chunk? This is your hallucination canary.Context precision and recall – is the retriever pulling relevant chunks, or padding the prompt with noise that dilutes the model's attention?Time-to-first-token – stream responses over SSE and aim to get under 800ms. Users forgive a slightly slower complete answer far more readily than they forgive a UI that looks frozen. The Real Point AI is most useful and powerful when it adapts to people-not the other way around. None of these pieces — retrieval, guardrails, escalation — are individually hard to build. What's hard is remembering that a conversational chatbot handling real user problems needs all three working together, with the guardrails and escalation path treated as first-class citizens rather than an afterthought bolted on after the first bad headline. Build it that way from the start, and the postmortem you're avoiding is your own.

A large API response becomes a client problem long before it becomes a network problem. A browser can receive hundreds of megabytes and still become unresponsive while buffering bytes, parsing one enormous JSON document, retaining duplicate object graphs, and rendering too much state on the main thread. The reliable solution is not a larger timeout. It is to stop treating the response as a synchronous document and start treating it as a durable, observable job whose data arrives in bounded pieces. Browser streams support incremental consumption and backpressure, while background workers allow long-running processing to remain independent of user-interface scripts. The Response Becomes a Job, Not a Payload The public API should acknowledge work quickly and return a stable job identifier rather than hold an HTTP connection open until every upstream page has been fetched. A 202 Accepted response establishes that contract without implying completion. The client can then subscribe to progress events, request a partial view, or retrieve a final artifact when the job reaches a terminal state. RFC 9110 defines 202 Accepted specifically for requests accepted for processing when processing has not necessarily completed. Java @PostMapping("/reports") public ResponseEntity<JobAccepted> create(@RequestBody ReportRequest request) { String jobId = UUID.randomUUID().toString(); workflowClient.start(reportWorkflow::run, jobId, request); return ResponseEntity.accepted() .header("Location", "/reports/" + jobId) .body(new JobAccepted(jobId, "QUEUED")); } This endpoint performs no large download or expensive transformation. It creates an addressable unit of work and returns immediately. The browser remains responsive because the initial response is tiny, while server capacity is protected from long-lived request threads. The job record should expose states such as queued, fetching, indexing, ready, failed, and canceled, with progress kept monotonic and coarse enough to remain trustworthy. Temporal Owns the Long-Running Control Flow Temporal fits the control plane because Workflow state survives process crashes and worker restarts, while failure-prone operations such as remote API calls belong in Activities with explicit timeouts and retry policies. Temporal documentation distinguishes deterministic Workflow logic from non-deterministic Activities and provides retry, timeout, heartbeat, and message-passing mechanisms for long-running execution. Java @WorkflowMethod public ResultRef run(String jobId, ReportRequest request) { String cursor = null; int sequence = 0; do { PageRef page = activities.fetchAndStore(jobId, cursor, sequence); activities.publishChunkReady(jobId, page); cursor = page.nextCursor(); sequence++; } while (cursor != null && !canceled); activities.buildIndex(jobId); activities.publishCompleted(jobId, sequence); return new ResultRef(jobId, sequence); } @SignalMethod public void cancel() { canceled = true; } Only references and counters should cross Workflow boundaries. Passing raw pages through Temporal causes every Activity input and result to accumulate in Event History. Temporal warns that large histories increase Workflow Task latency, documents a 50 MB or 51,200-event history limit, and recommends external storage plus Continue-As-New for large or long-running executions. The response body therefore belongs in object storage, while Temporal retains keys, checksums, cursors, and status. The fetching Activity should checkpoint often enough to support retries without restarting the transfer. Heartbeat details can carry the last committed cursor or byte range. Temporal recommends heartbeats for long-running Activities because missed heartbeats can trigger failure detection and retry. Java public PageRef fetchAndStore(String jobId, String cursor, int sequence) { UpstreamPage page = upstream.fetch(cursor); String key = storage.put(jobId + "/" + sequence, page.bytes()); Activity.getExecutionContext().heartbeat( new FetchCheckpoint(sequence, page.nextCursor()) ); return new PageRef( key, sequence, page.nextCursor(), page.sha256() ); } Kafka Carries Facts, Not Giant Documents Kafka is most effective as the event backbone, not as a substitute for object storage. Events should describe what happened and point to durable data, ChunkStored, ChunkIndexed, JobProgressed, JobCompleted, or JobFailed. Kafka enforces record-size limits at both producer and broker levels, so pushing multi-megabyte fragments into records creates brittle configuration coupling and expensive retries. Every event should use jobId as the key. Kafka partitions are ordered logs, and records sharing a key normally land in the same partition, preserving per-job sequence while allowing unrelated jobs to scale across partitions. Consumer groups distribute partitions across workers and rebalance them when membership changes. Java public void publishChunkReady(String jobId, PageRef page) { ChunkReady event = new ChunkReady( jobId, page.sequence(), page.storageKey(), page.sha256() ); kafkaTemplate.send("report-events", jobId, event); } Duplicate delivery must be assumed at every boundary. Kafka producer idempotence prevents duplicate writes caused by producer retries when compatible acknowledgment and in-flight settings are used, but downstream side effects still require idempotent consumers. An indexer can enforce uniqueness with (jobId, sequence, checksum) and commit its database transaction before acknowledging the Kafka offset. Backpressure should be expressed through bounded concurrency rather than hidden in memory. An Activity can publish one stored chunk at a time, while indexer lag indicates downstream pressure. Temporal can pause between pages when lag crosses a threshold, or consumers can scale until partition count becomes the limit. The Client Receives Progress and Bounded Content Server-sent events are sufficient when communication is primarily server-to-client. The protocol uses text/event-stream, keeps a persistent HTTP connection, and represents each notification as a small text block. A projection service can consume Kafka events, maintain the latest job state, and expose a resumable stream using application event IDs Java @GetMapping( value = "/reports/{jobId}/events", produces = MediaType.TEXT_EVENT_STREAM_VALUE ) public Flux<ServerSentEvent<JobEvent>> events( @PathVariable String jobId) { return eventProjection.stream(jobId) .map(event -> ServerSentEvent.<JobEvent>builder() .id(event.sequence().toString()) .event(event.type()) .data(event) .build()); } The client should render status changes and small previews, not append the full raw response into application state. When direct streaming is required, the Fetch API exposes the response body as a ReadableStream, allowing chunk-by-chunk processing rather than waiting for completion. Parsing should occur incrementally, with CPU-heavy decoding or transformation moved to a Web Worker, whose execution remains separate from user-interface scripts. Final delivery should usually be a paginated query API, a range-readable artifact, or a signed download URL. A giant JSON reconstruction endpoint merely recreates the original failure at the last step. RAG Turns Stored Volume Into a Useful Interface RAG becomes valuable after chunks are durably stored. Each chunk can be normalized, split along semantic boundaries, embedded, and indexed with metadata containing the job identifier, source sequence, object key, and byte range. The original RAG formulation combines parametric generation with retrieved non-parametric memory, grounding generation in selected passages rather than the entire corpus. Java @KafkaListener( topics = "report-events", groupId = "rag-indexers" ) public void onChunkReady(ChunkReady event) { if (index.exists( event.jobId(), event.sequence(), event.checksum())) { return; } byte[] payload = storage.get(event.storageKey()); chunker.split(payload).forEach(chunk -> index.upsert( event.jobId(), event.sequence(), chunk ) ); progress.markIndexed( event.jobId(), event.sequence() ); } The query path retrieves only the most relevant chunks and sends those bounded passages to the model. Raw object references remain attached so generated statements can link back to source material. RAG should not conceal incomplete ingestion; the query service must expose index coverage and reject complete-report requests until all expected chunks are indexed. Java public Answer answer(String jobId, String question) { List<Passage> context = index.search(jobId, question, 8); return generator.generate(question, context); } This layer changes the client experience from downloading everything before anything is useful to inspecting progress, searching partial results, and retrieving only relevant evidence. It also keeps model context bounded when the source response is extremely large. A Responsive System Is Built From Explicit Boundaries The essential boundary is simple: Temporal owns durable intent and recovery, Kafka distributes compact facts, object storage holds large bytes, RAG builds a searchable semantic view, and the client receives only bounded updates or explicitly requested slices. Each component solves a different failure mode, and none is forced to carry the complete response through an interface designed for small messages. The resulting architecture prevents UI freezes, survives retries and restarts, supports cancellation and replay, and makes large upstream results useful before a monolithic download could finish. Large-response handling becomes reliable when completion is modeled as a process rather than a payload.

By Uthej Mopathi CORE

Retrieval-augmented generation solved a real problem: it grounded LLM outputs in facts the model was never trained on. But classic RAG has a ceiling. It retrieves once, stuffs the results into a prompt, and hopes the top-k chunks happen to contain the answer. There's no self-correction, no multi-step reasoning, and no way to recover when the first retrieval misses. Agentic RAG removes that ceiling by putting an LLM-driven agent in the loop — deciding what to retrieve, when to retrieve again, whether the retrieved context is actually good enough, and how to combine multiple sources before answering. This article walks through building one from scratch, step by step, with working code you can adapt to your own stack. Why "Agentic" Changes the Architecture In naive RAG, the flow is linear: Plain Text Query → Embed → Vector Search → Stuff Context → Generate Answer In agentic RAG, retrieval becomes a tool the agent chooses to call, possibly more than once, possibly against more than one source, with a reasoning step wrapped around every hop: Plain Text Query → Agent Plans → Calls Retrieval Tool(s) → Grades Results → (Insufficient? → Reformulate → Retrieve Again) → Sufficient? → Synthesize → Self-Check → Answer That loop is the entire value proposition. It costs more tokens and more latency per query, but it converts a system that silently fails on hard questions into one that visibly tries harder before giving up. Step 1: Define the Tools, Not Just the Index The first mistake teams make when porting from classic RAG is treating the vector store as the only retrieval surface. An agent needs a toolbox, and the toolbox should reflect the actual shapes of knowledge in your domain: Vector search tool – semantic similarity over unstructured docsKeyword/BM25 tool – exact term and code/identifier matches vector search missesStructured query tool – SQL or API calls against systems of recordWeb search tool – for anything outside your corpus, if permitted Each tool gets a clear name, a docstring the agent can reason over, and a narrow, single-purpose contract. This is also exactly where Model Context Protocol (MCP) earns its keep — it standardizes how these tools are described and invoked, so the same retrieval tool can be reused across agents and orchestration frameworks instead of being reimplemented per project. Python from typing import List, Dict def vector_search(query: str, top_k: int = 5) -> List[Dict]: """Semantic search over the document vector store. Best for conceptual questions, paraphrased queries, and 'how does X work' style requests.""" embedding = embed(query) return vector_db.query(embedding, top_k=top_k) def keyword_search(query: str, top_k: int = 5) -> List[Dict]: """Exact/BM25 search. Best for error codes, identifiers, config keys, and anything where wording must match verbatim.""" return bm25_index.search(query, top_k=top_k) def sql_lookup(question: str) -> Dict: """Structured lookup against systems of record (claims status, account data, ticket state). Use when the question asks for a current, specific fact rather than an explanation.""" query = nl_to_sql(question) return db.execute(query) Step 2: Give the Agent a Retrieval Plan, Not Just Tool Access Handing an LLM a list of tools and hoping it calls them well is how you get expensive, undisciplined agents. Instead, prompt for an explicit plan before any tool call happens: Python PLANNER_PROMPT = """ You are a retrieval planner. Given the user's question, decide: 1. What sub-questions need to be answered 2. Which tool(s) best fit each sub-question 3. Whether this requires one retrieval pass or several sequential ones Return a JSON plan: { "sub_questions": [...], "tool_calls": [{"tool": "...", "query": "..."}], "requires_iteration": true/false } """ This planning step is where agentic RAG earns its name — the system is reasoning about the retrieval strategy itself, not just executing a fixed pipeline. For a multi-part question ("compare our Q3 claims volume to Q2 and explain the driver"), the plan might route one sub-question to sql_lookup and another to vector_search, then merge both before generating. Step 3: Retrieve, Then Grade Before You Generate This is the step classic RAG skips entirely, and it's the single highest-leverage addition you can make. After retrieval, insert a grading pass that checks relevance before the results ever reach the generation prompt: Python GRADER_PROMPT = """ Question: {question} Retrieved chunk: {chunk} Is this chunk relevant and sufficient to help answer the question? Answer strictly: RELEVANT, PARTIALLY_RELEVANT, or IRRELEVANT. """ def grade_chunks(question: str, chunks: List[Dict]) -> List[Dict]: graded = [] for chunk in chunks: verdict = llm_call(GRADER_PROMPT.format( question=question, chunk=chunk["text"] )) graded.append({**chunk, "grade": verdict}) return [c for c in graded if c["grade"] != "IRRELEVANT"] If everything comes back IRRELEVANT, that's a signal, not a dead end — it routes back into Step 4. Step 4: Reformulate and Retry on Weak Retrieval When grading fails to produce enough relevant context, the agent should rewrite the query rather than silently generating from thin evidence: Python def agentic_retrieve(question: str, max_attempts: int = 3) -> List[Dict]: query = question for attempt in range(max_attempts): raw_results = vector_search(query) good_results = grade_chunks(question, raw_results) if good_results: return good_results query = llm_call( f"The search for '{query}' returned nothing useful for " f"the question '{question}'. Rewrite the search query to " f"use different terms or a narrower/broader scope." ) return [] # exhausted attempts — surface this honestly downstream This is the difference between a system that degrades gracefully and one that hallucinates confidently. Three attempts is a reasonable default; tune it against your latency budget. Step 5: Synthesize Across Sources, Not Just Chunks Once you have graded, relevant context — possibly from more than one tool — the generation prompt should make the agent explicitly reconcile sources rather than concatenate them: Python SYNTHESIS_PROMPT = """ Question: {question} You have retrieved information from multiple sources. Synthesize an answer that: - Cites which source supports each claim - Flags any contradictions between sources explicitly - States clearly if the retrieved context is insufficient, rather than filling gaps with unsupported assumptions Sources: {sources} """ Explicitly asking the model to flag contradictions and insufficiency here reduces silent hallucination more than almost any other prompt-engineering change in the pipeline. Step 6: Self-Check Before Returning an Answer A final verification pass — cheap relative to the rest of the pipeline — catches cases where the synthesis drifted from the retrieved evidence: Python VERIFY_PROMPT = """ Answer: {answer} Source context used: {sources} Does every factual claim in the answer trace back to the source context? List any claim that does not. If all claims are supported, say VERIFIED. """ If verification fails, route back to Step 3 with the flagged claim as a new sub-question, rather than returning an answer you can't trace. Step 7: Orchestrate the Loop Tie the steps together with an explicit state machine rather than a single giant prompt. A minimal LangGraph-style graph looks like: Plain Text plan → retrieve → grade → (insufficient? → reformulate → retrieve) → synthesize → verify → (unsupported claim? → retrieve) → return answer Keeping this as an explicit graph — rather than trusting one long agent prompt to "figure it out" — is what makes the system debuggable in production. Each node logs its own decision, so when an answer is wrong, you can see exactly which stage introduced the error: a bad retrieval, a bad grade, or a synthesis that overreached its sources. What This Costs You, Honestly Agentic RAG is not free. Expect: 2–5x the token spend of single-pass RAG on the same question, since grading, reformulation, and verification all consume LLM callsHigher latency — a multi-hop query can take several seconds longer end to endMore moving parts to monitor — retrieval, grading, and verification all need their own observability, not just the final answer The payoff is a system that fails visibly and recovers automatically on hard, multi-part, or ambiguous questions — instead of quietly returning a confident answer built on the wrong three paragraphs. For high-stakes domains (claims processing, compliance, clinical or legal contexts), that trade is usually worth it. For simple FAQ-style lookups, classic RAG is often still the right call — agentic RAG is a scalpel for hard questions, not a universal upgrade.

By Balaji Venkatasubramaniyar CORE