AI/ML

Traditional vector RAG retrieves by embedding the question and finding semantically similar chunks, often augmented with lexical search, filtering, or reranking. This approach works when the answer is explicitly described in one or more chunks. However, it breaks down when the answer depends on relationships between facts. The question "Does my application depend on a compromised package?" illustrates this limitation. The vulnerable package may be several layers deep in the dependency tree, and no single chunk contains the answer. The answer emerges by following a chain of dependencies, but similarity search can struggle because the answer is distributed across multiple relationships rather than represented as a single semantic concept. GraphRAG addresses this issue by retrieving from a knowledge graph, where entities are connected through explicit relationships. However, GraphRAG is not a replacement for vector RAG. It is a retrieval paradigm for problems where relationships, graph structure, and provenance matter. For many applications, vector retrieval remains the fastest and most effective way to find semantically relevant information, while graph retrieval adds value when the answer depends on connected facts. But graph retrieval isn't a single technique; it requires deciding what to retrieve, how to retrieve it, and whether retrieval should happen in one pass or multiple stages. If these decisions are made correctly, GraphRAG can answer questions that vector search cannot; otherwise, it may result in a system that's slower and no more accurate than a well-designed vector RAG. This article focuses on retrieval. Graph construction is a separate issue, and schema quality affects everything. Here, we assume a well-built graph. The examples are based on a small software supply-chain graph. It's fictional, but the shape is from a real incident. In 2018, the npm package event-stream was compromised after a malicious dependency, flatmap-stream, was introduced into its dependency chain. The challenge is discovering the path shopping-app -> analytics-js -> event-stream -> flatmap-stream and connecting it to CVE-2024-1. No single text chunk contains this chain, and none of those package names indicate "compromised package." Scanners such as npm audit answer this question easily because they're built for this structure. GraphRAG can answer broader relationship questions around the same graph structure, especially when the answer requires combining multiple sources. Decision 1: Granularity Graph retrieval returns one of four units, from finest to coarsest. Node (for example, flatmap-stream and its attributes). Triplet (for example, flatmap-stream -[HAS_VULNERABILITY]-> CVE-2024-1). Path: like the red route. Paths are good for multi-hop and provenance questions. Subgraph: a connected region, like the payment component. Finer units are precise, but coarser units have more context, and more noise. Choose a unit that matches the question. Each mechanism, described in the next section, produces some units more naturally than others. Decision 2: The Six Mechanisms 1. Similarity Embed graph elements, retrieve the ones nearest the question vector. Most systems use this to find starting points. It answers "find things like this" questions on its own. Cypher CALL db.index.vector.queryNodes('pkgEmbeddings', 5, $questionVector) YIELD node, score RETURN node.name, score A simple query like "Which of our packages resemble this known-bad one?" is all that's needed. However, similarity over plain text embeddings has its limitations; it's blind to structure. So it only finds nodes that match the query and misses connected facts that don't resemble it. This method is useful for locating entry points and serving semantic lookups, but for anything that requires multiple hops, it's best to hand off to a structural mechanism. 2. Structural Traversal To get around the limitations of similarity searches, walk outward from the entry points, using techniques like neighbor expansion, breadth-first or depth-first search, and pathfinding. The granularity of what you collect depends on the approach; collecting neighbors gives you a local subgraph, while tracing routes between two entities gives you paths. Cypher // Entity-centric: what does the app build on? (a neighborhood) MATCH (:Package {name: 'shopping-app'})-[:DEPENDS_ON*1..2]->(dep:Package) RETURN DISTINCT dep // Connection question: how does the app reach vulnerable packages? MATCH p = (:Package {name: 'shopping-app'})-[:DEPENDS_ON*..5]->(bad:Package) WHERE (bad)-[:HAS_VULNERABILITY]->(:CVE) RETURN p For instance, the second query might return shopping-app -> analytics-js -> event-stream -> flatmap-stream, along with any other route to a vulnerable package. When the question is about risk, you want every possible path, not just the shortest one, because a second route is a second exposure. For large graphs, it is often more efficient to start from known vulnerable nodes and traverse backward, or constrain the search from the application side, depending on the query. Traversal is the cheapest, fastest, and easiest mechanism to explain, since you can read the route. But it has a weakness: fan-out. The practical depth limit depends on how constrained the walk is; unconstrained neighbor expansion grows rapidly, so shallow expansion is often preferred. A typed, direction-constrained path search, like the one mentioned earlier, prunes most of that growth and remains tractable deeper, which is why a four-hop dependency chase basically works in this case, but a generic four-hop expand-everything doesn't. Use traversal when the question is anchored on specific entities, and it's the best approach. 3. Graph Algorithms Two things are needed for traversal: a starting point, and a rule for edge selection. But what if you're missing one or both? There are two graph algorithms that can help. Personalized PageRank is useful when you have a starting point but no rule. It assigns a score to the mentioned entities, lets the score spread across the edges, and ranks nodes by the score they receive. Nodes that are highly reachable from the seeds through many strong paths receive higher scores. This helps find relevant nodes even if they're far away and don't share any words with the question. HippoRAG uses this for retrieval. Cypher CALL gds.pageRank.stream('supplyChain', { sourceNodes: $seedEntities, dampingFactor: 0.85 }) YIELD nodeId, score RETURN gds.util.asNode(nodeId).name AS entity, score ORDER BY score DESC LIMIT 10 For example, on our graph, if we seed analytics-js and CVE-2024-1, flatmap-stream receives a high score because it is strongly connected to both seed regions. Nothing in the query named it, though. Furthermore, directionality matters. For dependency graphs, reverse traversal or an appropriate projection is often required because vulnerabilities may be sink nodes. Community detection is especially useful for broad exploratory questions where no specific entity is known. A question like "what are the main risk areas across our dependencies?" is about the whole graph. The approach is to cluster the graph into communities, have an LLM summarize each community, and answer broad questions from those summaries. This is a key component of the Microsoft GraphRAG approach. On our graph, clustering gives us three communities: a payment-and-media stack, a web-framework stack, and the analytics subsystem with the compromised package. Cost is the main difference between the two. PageRank is expensive at query time because its scores depend on the seeds and can't be precomputed. On a large graph, you need to bound the projection to a region found by a similarity pass first. Community detection is expensive at index time because the LLM summaries are costly to compute and need to be redone when the graph changes. But queries after that are cheap. So pick PageRank when you have entities but no target. Pick communities when there's no starting entity. 4. Declarative Query With exact structural constraints in the question, it's best to query the graph directly. The model translates the question into a query language, like Neo4j's Text2Cypher. Schema grounding and validation significantly improve generated queries. Cypher // "Which volunteer-maintained packages also have a known CVE?" MATCH (pkg:Package)-[:MAINTAINED_BY]->(:Maintainer {kind: 'volunteer'}) MATCH (pkg)-[:HAS_VULNERABILITY]->(cve:CVE) RETURN pkg.name, cve.id This returns results like flatmap-stream / CVE-2024-1, combining a maintainer condition and a vulnerability condition that fuzzy mechanisms can only approximate, resulting in an exact and auditable outcome. This risk lies in the generation step, with models often inventing relationship types that sound right but don't exist in the schema; a well-formed query over imaginary edges returns zero rows without error. To mitigate this, ground the generation in the actual schema, validate the query before running it, and treat an empty result as a signal to fall back to another mechanism. Use this approach for questions that reduce to filters, counts, or joins across relationship types. 5. Generative Retrieval This mechanism works in two stages: first generating a retrieval plan that defines the relationship pattern to follow, and then translating that plan into a graph query. Reasoning on Graphs (RoG) works this way, with an LLM generating planning paths and the system retrieving the concrete paths that satisfy them. Cypher plan ← LLM("what relation path answers this?", schema) → "DEPENDS_ON* , then HAS_VULNERABILITY" paths ← graph.match(seed='shopping-app', pattern=plan) The model infers the shape of the traversal, and the graph supplies the instances. This approach fits questions where the right pattern isn't known in advance, and you don't want to hand-write a template for it. It relies on the model understanding the schema, and a one-shot plan can't correct itself unless you make the retrieval iterative. 6. Learned Retrieval A model can be trained to do the selecting, with a graph neural network scoring nodes for relevance to the question. Approaches such as G-Retriever formulate subgraph selection as an optimization problem, including variants inspired by Steiner tree formulations. The retriever is a trained component; it embeds the question, scores candidate nodes and edges, and returns the highest-value connected subgraph as evidence. This approach has demonstrated strong accuracy on hard multi-hop benchmarks, but it comes with training, serving infrastructure, and transparency costs. They're suitable for accuracy-critical question answering over a stable schema, but they're rarely the first build. Decision 3: The Paradigm You've still got to decide how many times to go to the graph. This affects both latency and accuracy. A simple approach is one retrieval, gathering everything in a single pass, which keeps latency low and is suitable for real-time answers. Iterative retrieval is another option, involving multiple passes that build on each other, useful when one pass isn't enough. It comes in two versions: fixed-rounds and adaptive. The adaptive version stops once the model has gathered sufficient information. Most production systems opt for a multi-stage approach, chaining different mechanisms together. A common pattern is using similarity retrieval to find entry points, structural traversal to expand context, and reranking to select the final evidence set. In practice, mechanisms usually combine in specific ways. Combining vector retrieval with graph retrieval is often described as HybridRAG. Letting a large language model plan the stages at query time is referred to as agentic retrieval, which is a composition pattern rather than a new mechanism. Matching Questions to Mechanisms A production system doesn't pick a mechanism per question at runtime. During the planning phase, you assess the questions, implement two or three mechanisms that cover them, and route between those in production. This table supports that assessment. The question is about…Reach forUsual granularityWhere the cost isThings semantically like X, or finding entry pointsSimilarityNode/tripletCheap, at query timeA specific entity or the routes between twoStructural traversalNode/path/subgraphCheap, at query timeMulti-hop relevance with an unknown targetPageRankRanked nodesCompute-heavy, query timeA broad theme across the whole graphCommunity detectionSubgraph + summaryExpensive, at index timeExplicit constraints: filters, counts, joinsDeclarative queryWhatever it projectsCheap, at query timeA pattern that must be inferred from the questionGenerativePath/subgraphModerate, one LLM callAccuracy-critical hard multi-hop QALearned (GNN)SubgraphExpensive, training Start with the basics. A similarity pass for entry points and structural traversal to expand. Add mechanisms as needed. PageRank or generative planning for harder questions, declarative queries for exact constraints, community summaries for thematic breadth, and learned retrieval when accuracy justifies it. Conclusion Graph retrieval involves making three key decisions. First, you need to choose the right granularity; this could be a node, triplet, path, or subgraph, depending on the answer you're looking for. The mechanism is also crucial: it's about selecting the right approach, such as similarity, traversal, graph algorithms, declarative queries, generative planning, or learned retrieval. Then there's the model: whether to use a single-pass, iterative, or multi-stage approach. By making these decisions with your system's specific questions in mind, you can design a tailored retrieval architecture rather than relying on trial and error.

Software testing has always been binary at its core. A test passes, or it fails. The build is green, or it is red. The release goes out, or it gets blocked. This binary model has served software teams well for decades because the systems being tested were deterministic — the same input reliably produced the same output, every time. AI systems are not deterministic. And yet most teams are still testing them with a binary framework that was never designed to handle probabilistic behavior. This is one of the most significant gaps in enterprise AI quality engineering right now — and it is quietly producing false confidence across organizations deploying AI at scale. The Problem With Binary Testing for AI Systems When you test a traditional function, a pass means the function behaved correctly for that input. When you test an AI model with a binary pass/fail framework, a pass means the model produced an acceptable output for that particular input at that particular moment. It tells you almost nothing about how the model will behave across the full distribution of real-world inputs it will encounter in production. Consider a practical example. You build a test suite of 500 cases for an AI-powered fraud detection system. The model passes 487 of them — a 97.4% pass rate. Your pipeline shows green. Confidence is high. What your test suite does not tell you: How confident was the model on each of those 487 passes? Was it 99% confident or 51% confident?How does the model perform on inputs that fall outside your 500 test cases?Are the 13 failures clustered in a specific transaction type that happens to represent 40% of your production volume?Is the model's confidence degrading over time as data distribution shifts? Binary pass/fail answers none of these questions. Confidence scores do. What Confidence Scores Actually Tell You A confidence score is the model's self-reported probability that its output is correct. A model that classifies a transaction as fraudulent with 98% confidence is telling you something very different from a model that makes the same classification with 54% confidence — even if both outputs look identical from a binary perspective. For enterprise teams, confidence scores unlock four dimensions of AI quality that binary testing simply cannot surface. 1. Uncertainty Mapping When you aggregate confidence scores across your test suite, you can map where your model is uncertain. Consistently low confidence scores on a particular input pattern signal a coverage gap — the model is operating outside its reliable domain. This is actionable information. Binary results just tell you the model passed. 2. Threshold Calibration Confidence scores allow you to define actionable thresholds. A model that is less than 70% confident should route to human review. A model that is less than 40% confident should reject the action entirely. You cannot build these guardrails without confidence data — you are just guessing at where the risk lies. 3. Distribution Shift Detection As your production data changes over time, confidence scores will drift before accuracy degrades. This makes confidence monitoring an early warning system for distribution shift. By the time your binary tests start failing, the model has already been making low-confidence decisions in production for weeks or months. 4. Risk Stratification Not all AI decisions carry the same consequence. A low-confidence recommendation in a product suggestion engine is recoverable. A low-confidence decision in a payment routing or medical triage system is not. Confidence scores let you stratify AI decisions by risk and apply proportional oversight — something binary results make impossible. Implementing Confidence-Aware Testing in Practice Shifting to confidence-aware testing does not require replacing your existing test infrastructure. It requires extending it. Add Confidence Capture to Your Test Assertions Instead of just asserting that the model output matches an expected value, capture the confidence score alongside every assertion. Your test output should include the confidence distribution across your test suite, not just the pass/fail count. Python def test_fraud_classification(model, test_input, expected_label): result = model.predict(test_input) confidence = result.confidence_score assert result.label == expected_label, f"Label mismatch: {result.label}" assert confidence >= MINIMUM_CONFIDENCE_THRESHOLD, \ f"Low confidence prediction: {confidence:.2%} on input type {test_input.category}" # Log for distribution analysis log_test_result( input_category=test_input.category, expected=expected_label, predicted=result.label, confidence=confidence, passed=(result.label == expected_label) ) Define Confidence Thresholds By Risk Tier Work with your domain experts to define what confidence level is acceptable for each category of AI decision. These thresholds should be part of your test specifications, not afterthoughts. YAML confidence_thresholds: high_risk_decisions: minimum: 0.85 human_review_below: 0.90 standard_decisions: minimum: 0.70 human_review_below: 0.75 low_risk_decisions: minimum: 0.60 Test the Distribution, Not Just Individual Cases A model can pass every test case individually while still having a problematic confidence distribution. Add aggregate assertions to your test suite that validate the shape of confidence across your full test set. Python def test_confidence_distribution(model, test_suite): results = [model.predict(case) for case in test_suite] confidence_scores = [r.confidence_score for r in results] mean_confidence = sum(confidence_scores) / len(confidence_scores) low_confidence_count = sum(1 for c in confidence_scores if c < 0.70) low_confidence_rate = low_confidence_count / len(confidence_scores) assert mean_confidence >= 0.80, \ f"Mean confidence too low: {mean_confidence:.2%}" assert low_confidence_rate <= 0.05, \ f"Too many low-confidence predictions: {low_confidence_rate:.1%} of test cases" Monitor Confidence in Production, Not Just in Testing Confidence-aware testing must extend beyond your test suite into production monitoring. Set up dashboards that track confidence score distributions on live traffic, alert on confidence degradation, and trigger retraining or review workflows when confidence drops below defined thresholds. What This Looks Like in Practice A retail enterprise I worked with deployed an AI model for inventory replenishment decisions. Their initial test suite had a 96% pass rate. The team was comfortable with the release. After introducing confidence-aware testing, the picture looked different. The model was consistently making replenishment decisions with confidence scores between 55-65% for seasonal products — a category that represented a significant portion of their inventory value. Binary testing had masked this entirely because the model's outputs happened to align with expected values in the test data, even though the model was operating with low certainty. After setting a confidence threshold of 80% for high-value inventory decisions and routing lower-confidence predictions to a human reviewer, the team caught a systematic miscalibration in the seasonal product segment before it reached production. The binary tests had given them a false green. The confidence scores gave them the truth. The Governance Case for Confidence Scores Beyond the technical benefits, there is a governance argument for confidence-aware testing that is becoming increasingly difficult to ignore. Regulatory frameworks and enterprise AI governance standards are beginning to require explainability and documented uncertainty bounds for AI systems making consequential decisions. A binary pass/fail test result does not satisfy an auditor asking how certain your AI system was when it made a particular decision. A confidence score does. If your organization is operating AI systems in regulated domains — finance, healthcare, retail payment processing — building confidence measurement into your testing and monitoring infrastructure is not just good engineering practice. It is the foundation of a defensible governance posture. Conclusion Binary pass/fail testing was built for deterministic systems. AI systems are probabilistic by nature, and testing them as if they are deterministic produces false confidence at exactly the moments when you need accurate confidence most. Confidence scores do not replace binary testing. They complete it. They answer the questions that pass/fail cannot: how certain was the model, where is it uncertain, and is that uncertainty clustered in ways that create production risk? The teams that get AI quality engineering right in the next few years will not be the ones with the greenest dashboards. They will be the ones who understood that green does not mean confident — and built their testing infrastructure accordingly.

Last year, I was working on deploying an agentic AI system to help manage cloud infrastructure at scale. The idea was straightforward: give the agent access to AWS APIs, let it observe infrastructure state, and allow it to take remediation actions autonomously. Scale a deployment here, restart a service there, update a configuration when metrics cross a threshold. What I did not fully appreciate at the time was how differently an AI agent fails compared to a traditional automation script. When a shell script goes wrong, it fails in a bounded, diagnosable way. You get an error code. You trace it. You fix it. When an agentic AI system fails, it can fail in ways you never anticipated, hallucinating resource states, misinterpreting instruction scope, or acting on adversarial inputs buried in a monitoring alert. These failures do not produce clean stack traces. They produce production damage. That realization sent me down a path of building a guardrail system. What I eventually learned, and this took four calibration cycles to prove empirically, is that no single guardrail layer can solve this problem. You need multiple complementary layers, and you need to design them to compensate for each other's blind spots. Here is what I built, what broke along the way, and what I would do differently from the start. Why the Obvious Solutions Did Not Work My first instinct was to use AWS Bedrock Guardrails. Configure a topic denial policy for destructive operations, set the content filters to HIGH, block PII like access keys. Simple, managed, done. I ran it against 100 representative agent prompts, a mix of read operations, staging changes, risky production changes, destructive operations, and adversarial jailbreak variants. The result stopped me cold. Tuned for zero false negatives, meaning I wanted to catch every genuinely dangerous action, the guardrail produced a 40% false positive rate. It was blocking list operations. It was blocking staging scale-outs. It was blocking service configuration updates that had nothing to do with deletion or destruction. That is not a deployable guardrail. That is a system that would make the AI agent useless within a day. The second problem was structural, not tuning-related. A Bedrock guardrail intercepts the model's text output. But an agent does not only produce text; it invokes tool calls. An agent can generate a perfectly compliant response like "I will scale the deployment safely" and then immediately invoke a delete API as a tool call. The guardrail never sees the tool call. It evaluated the wrong boundary. The third issue came when I looked at policy-as-code frameworks. OPA with Gatekeeper is excellent at Kubernetes admission time, evaluating manifests before they are deployed. But a DevOps agent is not deploying manifests. It is generating action proposals at runtime against live infrastructure that changes by the hour. A static policy that denies writes to "production resources" is useless unless it knows, at this exact moment, which resources are tagged as production. That information is not in a manifest. It is in live EC2 tags pulled from the AWS API. These were not flaws in the tools. There were boundary mismatches. Each tool was designed for a different problem. None of them was designed for the problem of governing an autonomous agent at the tool-call execution boundary. The Architecture I Landed On After a lot of iteration, I settled on a five-layer pipeline that intercepts at the tool-call boundary the moment the agent transitions from thinking to acting. Any layer can terminate the pipeline. The default is blocked. Blast-Radius Scoring Before any layer fires, every proposed action gets a blast-radius score between 0 and 1. Read-only operations (list, get, describe, monitor) score 0.1. Reversible mutations (restart, scale, update, patch) score 0.4. Potentially irreversible operations (delete, destroy, terminate, purge, wipe) score 0.9. This score drives routing. Low-risk actions auto-approve without touching the full pipeline. High-risk actions require human approval. Everything in between goes through policy evaluation. This is what keeps the system from adding 8 seconds of latency to every "list all EC2 instances" call. Layer 1: Bedrock Guardrail With a Bypass I kept Bedrock Guardrails as the first layer but added something critical: a selective bypass for low-risk and staging operations. Before the guardrail fires, the pipeline checks the blast-radius score and environment context. If the action is read-only or explicitly targeting a staging environment, the guardrail is skipped entirely. This one change took the false positive rate from 40% down to 18%. That occurred not by tuning the guardrail, but by changing the architecture around it. Layer 2: OPA Against Live State The second layer runs Open Policy Agent, but not against a static manifest. It pulls live AWS context via boto3 immediately before each evaluation: EC2 inventory with environment tags, S3 buckets, IAM roles. That live context becomes part of the input document that OPA evaluates. Now the policy can answer the question that actually matters: "Is this specific resource, right now, a production resource?" A rule that reads is_production(resource)` checks the live tag, not a manifest field. This is what catches the actions that Layer 1 misses, like "purge all messages from the SQS queue," that use vocabulary outside the guardrail's topic examples but clearly target production infrastructure. In my evaluation, Layer 2 was the sole blocking layer for 43% of correctly blocked prompts. Without it, those would have been false negatives. Layer 3: Confidence Delta Scorer This layer does something a little unconventional. It invokes the model twice, once with the guardrail attached and once without, and measures how much the guardrail changed the model's response. It looks at hedging vocabulary frequency: words like "should," "might," "could," "consider," "careful." A big divergence between the guardrail and unguarded responses means the guardrail substantially changed the model's behavior. That is a signal; the action is near the policy boundary and worth extra scrutiny. It is a probabilistic safety margin for borderline cases. Layer 4: HITL Gate With DynamoDB TTL For high-risk actions, a Lambda function writes a DynamoDB record with a 5-minute TTL and sends an SNS email with approve and reject URLs. The design principle that matters here: **silence equals rejection**. If the TTL expires without a response, DynamoDB automatically deletes the record, and the pipeline denies the action. An unavailable approver never produces an implicit approval. I built this using DynamoDB's native TTL feature, which means there is no polling loop, no Lambda checking every 30 seconds whether the token expired. It just works. Layer 5: Audit and Rollback Every pipeline execution writes a structured JSON record to CloudWatch: the prompt, risk classification, blast-radius score, per-layer decisions, latency, cost, and outcome. A CloudWatch alarm fires if the block rate drops below 70% in any 5-minute window, which would indicate something has degraded in the guardrail configuration. Before every approved action, boto3 captures a snapshot of the relevant resource state to S3. If health metrics degrade within 5 minutes after execution, automated rollback restores the pre-action state from that snapshot. What the Numbers Actually Showed I ran the complete pipeline against 100 prompts on live AWS infrastructure in us-east-1. Here is what came back: CategoryAccuracyFP RateFN RateAvg LatencyRead operations (20)95%5%0%910msSafe staging changes (20)100%0%0%7,920msRisky production changes (20)95%0%5%13,949msDestructive operations (20)100%0%0%9,590msAdversarial jailbreaks (20)90%0%10%8,125msTotal (100)96%1%3%8,099ms Three things stood out. Zero false negatives on destructive operations: Not one delete, destroy, terminate, or purge operation made it through. That was the most important number for me.22x latency reduction for blocked actions: Intercepted actions averaged 347ms. Allowed actions averaged 8,099ms. Because Layer 1 fires before the full LLM invocation, blocking a harmful action is actually faster than allowing a safe one. The safety mechanism is also a performance optimization for the adversarial case.Total cost: $0.0017 USD for 100 prompts: At 10,000 agent actions per day, that is roughly $0.17 daily. The cost of not having this is measured in incidents. The Calibration Study Nobody Talks About The finding I keep coming back to is the calibration progression: VersionAccuracyFP RateWhat Changedv1: Single Bedrock guardrail60%40%Baselinev2: Added low-risk bypass79%18%Architectural changev3: Added staging context in OPA89%8%Live state integrationv4: Expanded service config keywords96%1%Allow-list expansion What strikes me is that each improvement required a fundamentally different mechanism. The bypass addressed a structural mismatch. The staging context detection required live infrastructure data that no static guardrail can access. The keyword expansion fixed a vocabulary coverage gap. None of these is achievable by turning a dial on a single layer. This is the empirical case for layered defense-in-depth. Not as a philosophical preference. As a measurable engineering necessity. Practical Takeaways If you are building agentic DevOps tooling, here is what I would tell myself from a year ago: Intercept at the execution boundary: Your safety mechanism must fire when the agent calls a tool, not when it generates text.Pull live state before every policy evaluation: A policy that cannot see which resources are actually in production right now is not protecting production.Make your HITL gate fail closed: Design it so an unresponsive approver produces a denial, not a permit. DynamoDB TTL handles this elegantly without polling.Run your calibration study before going live: Measure FP and FN rates separately. They trade off against each other in ways that are not obvious until you measure them.Snapshot before every approved action: Automated rollback is not glamorous, but it is the safety net you will want when something approved turns out to be harmful. The Code Everything described here is open source: https://github.com/ManvithaP-hub/agentic-devops-guardrails That includes the Lambda functions, OPA Rego policies, boto3 state fetching, DynamoDB approval gate, CloudWatch audit, and a Terraform deployment module. You can run the full evaluation on your own AWS account for under a dollar.

The Failure You Have Probably Already Seen An enterprise AI agent is deployed against production data. It answers the first ten questions confidently and correctly. Then, on the eleventh question, it produces an answer that looks reasonable but is completely wrong. The team investigates. The model is fine. The prompt is fine. The tool integrations are fine. The problem is buried in the data itself. A field the agent relied on has drifted. A join it assumed existed no longer holds. A quality signal that used to be reliable has silently degraded. This is not a rare edge case. It is becoming one of the most common failure patterns in enterprise AI systems moving from prototype to production. And it points to a simple, uncomfortable truth: most enterprise data infrastructure was built for a consumer we no longer have. I have spent the past couple of years designing agentic AI systems against production data at Fortune 500 scale. What follows is the runtime governance pattern I now design around, and the failure modes it protects against. Who this article is for: This article is for data engineers, platform architects, AI engineers, and governance teams building enterprise agents that depend on production data. It focuses less on prompt design and more on the runtime data controls required to make agent answers reliable. Twenty Years of Data Built for Humans Every large enterprise data platform in production today was designed for human consumption. Analysts, business users, data scientists, and BI teams. Those consumers share a common trait: they exercise judgment. A human analyst looking at a broken dashboard notices it. A data scientist opening a table with unusual distributions asks a colleague. A finance user reviewing a report questions the number when it does not match their gut. Enterprise data governance evolved to support this consumer. Documentation lives in wikis. Quality is enforced by expected-value alerts that a human triages. Lineage is captured at the ETL job level, not the field level. Access is granted through role-based permissions and refined by manual data stewardship. All of this works when a human is at the end of the pipeline. An AI agent is not that consumer. An agent has no judgment. It processes what it is given and returns an answer. If the data is stale, the agent produces a stale answer with high confidence. If the lineage is broken, the agent cannot trace why. If a quality signal exists only as a wiki page, the agent cannot use it. The Four Gaps Most Enterprises Have Across the AI-in-production work I have seen, the same four gaps show up almost every time. Gap 1: Machine-Readable Data Contracts Most contracts exist as documentation, not as programmatic constraints. An agent cannot ask a Confluence page whether it is safe to trust a field. Data contracts need to be enforced at the platform layer, with schema, type, freshness, and quality guarantees expressed as executable rules. Gap 2: Use-Case-Aware Quality Fitness A dataset that is 95 percent complete may be fine for a marketing dashboard and completely wrong for a clinical AI model. Traditional data quality checks are use-case-agnostic. Agentic AI requires quality signals that answer a different question: is this data fit for this specific decision, right now? Gap 3: Field-Level Lineage That Updates in Real Time When a pipeline changes, human consumers get an email. Agents get a wrong answer. Lineage systems need to update as pipelines evolve and expose change signals in a form agents can consume, not just visualize. Gap 4: A Discovery Layer Agents Can Query Most catalog systems are designed for humans to browse. Agents need a machine interface to ask questions like which tables contain the concept I care about, and which of them is authoritative for this domain. Design Principles for Agentic Data Governance Closing these gaps does not require rebuilding the entire data platform. It requires making governance executable in the same path where the agent retrieves data, evaluates context, and produces an answer. Three design principles matter most. Start with the decision, not the data. For each production AI use case, define what a wrong answer looks like and work backward to the data requirements that would prevent it. This surfaces the specific quality signals, lineage nodes, and freshness constraints that matter. Make governance runnable, not readable. Every governance artifact your agents depend on should be programmatically executable at inference time. If a rule cannot be checked in code, an agent cannot use it. Documentation is useful for humans, but for agents it is invisible. Instrument for continuous evaluation. A governance framework that only fires at deployment is not enough. Models drift, data drifts, and use cases evolve. The governance layer needs to continuously evaluate agent outputs against real-world outcomes and flag drift before it becomes damage. Reference Architecture: Runtime Data Governance for AI Agents A practical implementation usually introduces a lightweight runtime governance layer between the agent and the underlying data platform. The goal is not to slow the agent down. The goal is to give the agent a reliable way to ask whether the data behind an answer is safe to use. At a minimum, this pattern includes five components: a data catalog that exposes authoritative sources, a contract registry that stores schema and business rules as executable checks, a lineage service that tracks upstream dependencies at the field and metric level, a quality service that publishes freshness and fitness signals, and an agent guardrail service that evaluates these signals before the agent responds. Runtime flow: User question → Agent → Semantic/data resolver → Governance service → Catalog, contract registry, lineage service, and quality service → Pass/Warn/Block decision → Agent response. Layer Responsibility Example Signal Catalog Identify authoritative datasets and business definitions. Certified source for booked deal value. Contract registry Validate schema, data types, null thresholds, and business rules. Discount variance must use the approved baseline method. Lineage service Track upstream source, transformation, and metric dependencies. Metric changed because a new source was added. Quality service Publish freshness, completeness, anomaly, and fitness scores. Dataset refreshed within SLA and passed threshold checks. Agent guardrail Block, warn, or allow the answer based on governance signals. Answer allowed only if lineage and contract checks pass. The agent should not directly trust a dataset simply because it can access it. Before answering, it should evaluate the data path, the contract status, the freshness window, the lineage change history, and the use-case-specific fitness score. If any critical check fails, the agent should either decline to answer or return the answer with an explicit data reliability warning. How the Runtime Governance Check Works In practice, the check is a short pre-answer step. The agent does not need to understand every governance rule directly. It needs a stable contract with a governance service that can evaluate the data path and return a decision. The user asks a business question.The agent resolves the requested metric, entity, dataset, or semantic concept.The agent calls the governance service with the resolved data assets and intended use case.The governance service checks catalog certification, contract status, lineage changes, freshness, completeness, and use-case fitness.The service returns a pass, warn, or block decision with machine-readable reasons.The agent answers, adds a caveat, escalates, or declines based on that decision. What a Machine-Readable Data Contract Actually Looks Like The abstract idea of a data contract only becomes real when you can point to one that an agent can actually consume. Here is a compact YAML example for a deal variance metric, expressing schema constraints, business rules, freshness expectations, and quality thresholds in a single artifact: YAML contract: dataset: deal.discount_variance schema: - field: discount_variance_pct type: decimal(18,2) required: true calculation: approved_discount_baseline_v2 - field: source_system type: string allowed_values: [crm_v3, revenue_hub] freshness: sla_hours: 24 breach_action: warn quality: completeness_threshold: 0.95 anomaly_score_max: 3.0 lineage: change_window_days: 30 on_upstream_change: require_review With this in place, an agent can call a single governance endpoint before responding, receive a machine-readable pass, warn, or block decision, and either answer confidently, answer with a caveat, or decline. The rule is not buried in a wiki page. It is live at inference time. Example Runtime API Pattern The runtime call does not need to be complicated. A minimal request can identify the metric, dataset, use case, and decision context. The response should be small enough for the agent to use directly in its control flow. JSON POST /governance/evaluate Request: { "metric": "deals.discount_variance_pct", "dataset": "deals.discount_variance", "use_case": "deal_desk_agent_review", "decision_context": "discount_variance_explanation" } Response: { "decision": "warn", "reasons": ["upstream_lineage_changed", "freshness_within_sla"], "agent_action": "answer_with_caveat" } In the agent workflow, this response becomes a control decision. A pass allows the agent to answer normally. A warn allows the answer but requires a reliability caveat. A block prevents the answer and routes the request to review, remediation, or a safer fallback path. Pseudocode: Turning Governance Into Agent Control Flow Python decision = governance.evaluate(metric, dataset, use_case) if decision.status == "block": return decline_with_reason(decision.reasons) if decision.status == "warn": return answer_with_caveat(query, decision.reasons) return answer(query) This is the core shift: governance is no longer a document the team reads during design review. It becomes a runtime dependency that the agent uses to decide whether to answer, qualify the answer, or stop. Runtime Checks an AI Agent Should Perform Before Answering Is this dataset or metric certified for the requested business domain?Has the schema changed since the agent workflow was last validated?Did all required fields meet completeness and validity thresholds?Is the data fresh enough for the decision being requested?Has any upstream lineage changed within a defined risk window?Does the requested answer depend on a metric with multiple calculation methods?Should the agent answer, warn, escalate, or decline based on the governance outcome? This does not require a heavyweight approval workflow for every query. In many cases, the runtime check can be a fast metadata call that returns a simple decision: pass, warn, or block. The important design principle is that governance must be available in the same execution path as the agent response, not in a separate documentation process that only humans can interpret. Failure Modes and Runtime Controls Failure mode What causes it Runtime control Stale answer Dataset missed its refresh SLA. Freshness check with warn or block behavior. Wrong metric Multiple calculation methods exist for the same business concept. Contract and semantic registry validation. Silent lineage change An upstream source or transformation changed after validation. Field-level lineage check within a defined risk window. Misused dataset The dataset is accessible but not certified for the requested domain. Catalog certification and use-case fitness check. Incomplete evidence Required fields fail completeness or validity thresholds. Quality service decision with explicit failure reasons. A Concrete Example From the Field On one enterprise AI project in a regulated environment, we deployed an agentic assistant to help analysts explore a large deal registration and booking dataset. Early testing looked solid. Several weeks into production, the agent began returning confidently wrong answers about a specific discount variance metric. The model had not changed. The prompt had not changed. What changed was an upstream ingestion job that added a new source that computed discount against a different price baseline. A human analyst would likely have questioned the number because it felt off. The agent did not. It saw a valid number in a valid field and reported it as authoritative. The fix was not in the model. We added a machine-readable contract for the approved discount baseline, a lineage signal for recent upstream changes, and a runtime check the agent could call before answering. After that, the same failure could not recur silently. The agent either answered correctly or flagged that the underlying data had changed and required review. The lesson was not that agents are unreliable. It was that agent reliability is a property of the data layer, not the model layer. Once we treated the governance layer as an active runtime dependency instead of static documentation, the entire class of silent-failure risk collapsed. Implementation Considerations Cache low-risk governance decisions to reduce latency, but recheck high-risk metrics at runtime.Separate warn rules from block rules so agents can still answer safely when risk is explainable.Version data contracts alongside pipelines, semantic models, and metric definitions.Log every agent answer with the governance decision, reasons, dataset version, and lineage snapshot used.Start with high-risk metrics and regulated workflows before expanding the pattern across the broader data estate. Why This Belongs in the Architecture, Not the Prompt Prompt engineering can reduce some surface-level errors, but it cannot solve a missing contract, stale dataset, broken lineage path, or ambiguous metric definition. Those failures sit below the model. They need to be handled in the platform architecture, where data access, metadata, quality, lineage, and policy decisions are available at runtime. For teams building enterprise AI agents, the practical takeaway is straightforward: treat runtime governance as part of the agent stack. If an agent can call a retrieval service, vector index, SQL endpoint, or workflow tool, it should also be able to call a governance service before committing to an answer. The next generation of enterprise AI reliability will not come only from better models. It will come from data platforms that can tell agents, in real time, whether an answer is safe to give. About the Author. Avinash Maddineni is a Lead Data Engineer with 15 years of enterprise data infrastructure experience across healthcare, financial services, energy, and travel. He builds agentic AI and data governance systems at Fortune 500 scale and is founder of PureStrokeAI (USPTO provisional patent filed May 2026).

If you have spent any time inside a mid-to-large organization that has embraced AI-assisted development, you've probably seen the pattern already. Teams move fast. New apps get spun up in days. Business units that used to wait months for IT now have working tools in a week. On the surface, it looks like a win. But look a little deeper, and a different picture starts to emerge. I've seen this happen firsthand: within twelve months of an organization adopting AI-assisted development, the internal app count can double, sometimes triple. And with every new app comes a fresh copy of the customer table, a slightly different definition of what a "transaction" means, and another team that has no idea what the team next door already built. The result is two compounding problems, and most organizations are treating them as if they're separate issues when they share the same root cause. The Two Problems Nobody Is Connecting There are two main problems that are impacting companies developing and deploying AI apps. They are: App Sprawl: Dozens of small applications accumulate. Each needs maintenance, security patches, dependency updates, and an owner. Most were built fast and designed by no one; they were generated. I have watched engineering teams burn entire sprints just cataloging what exists, let alone maintaining it. The long tail of unmaintained micro-apps quietly becomes an engineering liability. Data Scattering: The same business entities, customers, products, orders, and employees are defined slightly differently in every application. No canonical version exists anywhere. The same customer record lives in six places with six slightly different schemas. Reporting turns out to be like being an archaeologist! Integrations become fragile. Resuming any reconstruction means untangling a whole lot of divergent assumptions over the course of months. Most organizations look at them as individual issues: App governance is one, and data warehouse is the other. They come late and cure both the symptoms and not the cause. The actual root cause? No shared platform layer makes it structurally easy to build new applications without duplicating data and easy to share capabilities without reinventing them. Every new app starts from scratch. It creates its own database, its own auth, its own version of "what a customer is." The AI assistant helping build it has no way to know what already exists. So it builds freshness every time. The problem isn't that developers are building too much. The problem is that nothing they build connects to a common foundation. Introducing the Tectonic AI Platform The Tectonic AI Platform has been the architecture I've been working on that's actually a response to this. The governing idea is borrowed from geology: just as tectonic plates form the stable foundation beneath the dynamic surface of the earth, a Tectonic Platform provides a stable, canonical data and service layer beneath the fast-moving applications built on top of it. Applications are surface features fast to build, easy to replace, and expendable. The plate beneath them is the source of truth. It doesn't care what sits on top. It endures. This is not a product you install. It is an architectural posture, a set of structural decisions that organizations adopt before the sprawl begins or use to bring order after it already has. One important distinction worth making upfront: this is not a data warehouse. A warehouse is downstream and read-only. It doesn't stop three apps from each maintaining their own operational definition of a customer; it just lets you query all three versions in one place. The Tectonic plate is operational and live. It sits in the application layer, not below it. Apps read and write through it. It is the authoritative version, not a copy of one. The Four Pillars The framework is organized in this way. Each pillar addresses a specific failure mode that I've seen emerge when organizations skip the foundation. Pillar 1: Canonical Data Plates Shared, versioned data domains are owned by the platform, not by any single application. They include customers, products, transactions, and employees. These live on the plate. Applications interact with them through defined contracts (APIs), never by owning the underlying data store. Any app can read from the plate. Writing to it requires going through the contract. That's the word "owned by the platform" that is to be taken into account. I've seen people go to such trouble as trying to choose one app as the system of record to solve this problem. But that is no good — it would move ownership depending on how many people are on the roster. The plate does not belong to anyone; it is only legal to host the platform. Pillar 2: App Scaffolding Layer A generator framework that provisions new applications pre-wired to the plate layer from day one is also needed. When a developer or an AI assistant spins up a new app, it inherits auth, logging, observability, and data contracts automatically. The app starts connected, not isolated. Vibe coding stays fast. The structure comes for free. This is the foundation upon which the entire framework is designed to be interoperable with AI-assisted development. You aren't stopping anybody; you are just ensuring that the thing that they build into something also plugs in. Pillar 3: Capability Registry Organizations then need a discoverable catalog of everything that already exists, including APIs, workflows, AI models, reports, and integrations. Before building anything, developers (and AI coding assistants) query the registry first. Duplication becomes visible before it happens. "Does a customer lookup API already exist?" becomes a question with an answer. This is actually one of the most powerful pillars that are easy to acquire in practice. The overduplication is a mistake because people did not realize that it already existed. This is where the Register comes in. It also provides AI assistants with a surface to query before generating new code, changing the default from "build fresh" to "reuse first." Pillar 4: Governance at the Seam Rules and reviews live at the boundary between apps and plates. They are not inside individual apps. A new app can be built freely and quickly. What is allowed to be written on the plate is governed. This separates the fast surface (application layer) from the stable core (plate layer). Speed doesn't get sacrificed. Data integrity doesn't either. I want to make it clear what this pillar is NOT: it's not a committee, it's not a "ticket queue," and it's not a "review board." Governance at the seam should be automated wherever possible, including contract validation, schema versioning checks, and write permission enforcement. It's all about guardrails, not gatekeeping. What This Prevents Five Years From Now Without a Tectonic layer, here's what the organization typically looks like five years into an AI-assisted development culture: A long tail of unmaintained micro-apps, each with its own auth, its own schema, its own error handlingEngineers are spending more time stitching data together than building new capabilitiesAn AI-assisted development culture that has paradoxically made the codebase harder to understand because the surface area has exploded without any unifying structureRebuilding the same core capabilities repeatedly across teams that never knew the others existed A Tectonic layer is now in place, and every new application, no matter how quickly it is created, takes on its structure. The transformative era of vibe coding keeps on rolling. Technical "debt" is not compounded. Speed Without Structure Is Just Faster Entropy The Tectonic AI Platform is not anti-AI and not anti-speed. It is the infrastructure argument for why AI-assisted development can scale inside an organization without eventually collapsing under its own weight. The organizations that define their plates early, their canonical data domains, their shared capability contracts, and their scaffolding standards will find in a few years that they have a large and growing estate of AI-generated applications that actually work together. Those who don't will have a different, large, and growing estate. And a much harder problem to fix. The plate layer is what makes the speed sustainable. Define it early, or spend years paying for not having done so.

I spent years building data pipelines, mostly in Snowflake, in a regulated banking environment. For most of that time, lineage was straightforward: data moves through a transformation, and you can trace exactly where every number came from. That changed the moment LLM functions started showing up inside those same pipelines. The pattern showed up the same way every time. A Cortex function would generate a narrative, a summary, a piece of text meant for a report someone downstream would rely on. The data going into the report was fully traceable. The text coming out of the LLM was not. I could tell you which table fed a number. I could not tell you which prompt, which model version, or which configuration produced a specific sentence. That gap kept showing up, and it bothered me enough that I eventually went and checked whether the tools I was using were ever going to close it on their own. The realization did not come from a single dramatic moment. It came from working backward. After a Cortex function ran and produced output that ended up in a report, I tried to reconstruct what had actually happened: which prompt had been active, what parameters had governed the run. The query history showed the function had executed. It showed the timestamp, the user, the warehouse. What it could not show me was what had been sent to the model or what version of the prompt template had produced the result. I was looking at evidence that something had happened, with no record of what that something actually was. They are not going to close it. I looked across the major data governance and lineage tools commonly used in this space: dbt, MLflow, Apache Atlas, Snowflake's own native tooling, Informatica. Every one of them is genuinely good at tracking structured data through deterministic transformations. Table versions, transformation logic, pipeline runs, all well covered. None of them, as far as I could find, natively captures what happens the moment an LLM enters the picture: which prompt template was used, what version of it, what parameters the model ran with, or how the output maps back to a specific section of a specific report. That is not a criticism of those tools. They were built before this problem existed in its current form. But it means that right now, if someone asks you to reconstruct exactly how an AI-generated paragraph in a regulated report came to exist, in most environments, you cannot. You have the output. You do not have the chain that produced it. The Question That Kept Coming Up The question that kept surfacing in compliance conversations was some version of: which version of this process produced this output? Not just which data, not just which model, but which version of the entire process, prompt included, was active at the time a specific report section was generated. That question is unanswerable with standard data governance tooling, because prompts are not treated as versioned process components the way SQL transformations are. A dbt model gets a version, a run ID, a test result. A prompt template gets saved somewhere, maybe, by someone, whenever they remember to. The governance gap is not subtle. It is the difference between a process that is version-controlled end to end and one that treats its most consequential step as an untracked artifact. Regulatory frameworks are beginning to reflect this expectation even if they do not yet spell out the technical solution. The EU AI Act, in Article 12, requires that high-risk AI systems allow for the automatic recording of events over the lifetime of the system. That language is more specific than most summaries suggest: it rules out manual log exports or after-the-fact human notes as a substitute. It requires automatic, system-level capture. That is exactly the kind of infrastructure that does not exist in most LLM reporting pipelines today. The Fix: Build It Into the Pipeline The fix, for me, was not to wait for a vendor to solve this. It was to treat prompt lineage as something that belongs inside the pipeline from day one, not something to bolt on after the fact. Concretely, that meant logging the prompt template and its version, the model and its configuration, and a hash of the output, automatically, every time the function ran, as part of the same process that writes the report, not as a separate step someone has to remember to do later. The architecture has six layers, each capturing a specific category of governed artifact. Source data provenance tracks which tables and rows fed the model. Transformation logic captures which pipeline version prepared the data. Prompt construction records exactly what was sent to the model, including template ID, version, variables, and rendered prompt hash. Model parameters log the specific model version, temperature, and inference settings. Output integrity creates a tamper-evident hash of the generated text. Report context maps the output to a specific filing section, including approval records. If You Can Only Start With One Thing If I could only implement one layer first, I would start with output hashing. The reason is practical: everything else in the lineage chain can potentially be reconstructed or approximated after the fact. You can check version control for the prompt template. You can look at model documentation for parameters. But once a generated output has been filed in a regulatory document and time has passed, there is no way to prove retroactively that what was filed matches what the model produced, unless you captured a hash at the moment of generation. Output hashing is the layer that makes the rest of the chain defensible. Without it, even a complete lineage record can be questioned, because you cannot prove the output it describes is the output that was actually filed. What to Do Starting Now A few things I would tell another data or IT leader looking at this same gap: Inventory every place an LLM touches something that ends up in a regulated or customer-facing document. You cannot fix what you have not mapped.Do not assume your existing data governance stack already covers this. Check specifically whether it captures prompt versions and model configuration, not just source data.Build the logging into the pipeline itself, not as a side process. If it is optional or manual, people will skip it under deadline pressure, and you will be back where you started.Start with output hashing if you have to prioritize. That single layer gives you tamper-evident proof of what was generated, which is the foundation everything else depends on.Treat this the same way you treat any other production logging you cannot afford to lose. Once a report goes out, the question is not whether someone will eventually ask how it was produced. It is when.

Six months ago, building a RAG pipeline meant a full week of plumbing: an embedding job here, a vector store there, a retriever glued on with duct tape, and an orchestration layer that broke every time you touched it. I've built enough of these the hard way — hand-rolled vector search, custom chunking scripts, the works — to know exactly how much pain that "week" usually hides. Last week, I rebuilt the same thing on Azure AI Foundry. It took an afternoon. Not because the underlying problem got easier — grounding an LLM in your own data is still genuinely hard — but because Microsoft finally killed most of the integration tax that used to eat the first sprint of every RAG project. Here's what actually happened, warts included. The Old Way Was a Trap If you've built RAG before, you know the pattern: you don't fail at RAG, you fail at the seams between the pieces. Your chunking strategy doesn't match your embedding model's context window. Your retriever returns great results in a notebook and garbage in production because nobody wired up hybrid search. Your "agent" is really just a for-loop that stuffs retrieved text into a prompt and hopes. Foundry's whole pitch is that it owns those seams instead of leaving them to you. I was skeptical. I'm less skeptical now. What I Actually Did Step one: spin up a Foundry project. Not a hub-based one — those are legacy at this point, and if a tutorial has you creating one, skip it. The newer Foundry project type is the one to use. Step two: deploy two models. A chat model and an embedding model. Click, click, done. Both show up with their own endpoints. This part genuinely takes five minutes, and it's the first sign you're not building infrastructure anymore — you're configuring it. Step three: point Foundry at my documents. Blob storage in, Azure AI Search out. Foundry handles the chunking and embedding generation itself. I turned on hybrid search (keyword plus vector) because pure vector search on enterprise docs tends to miss exact terms people actually search for — product names, error codes, that sort of thing. If your content has a lot of that, don't skip this. Step four — and this is the part that's different from every tutorial I read two years ago. I didn't write a retrieval pipeline. I registered the search index as a tool on the agent and let the agent decide when to call it. Here's the whole thing: Python from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential project = AIProjectClient.from_connection_string( credential=DefaultAzureCredential(), conn_str=os.environ["AIPROJECT_CONNECTION_STRING"], ) agent = project.agents.create_agent( model="gpt-4o-mini", name="docs-assistant", instructions=( "Answer only using retrieved context. " "Cite the source document for every claim. " "If the answer isn't in the retrieved content, say so." ), tools=[{ "type": "azure_ai_search", "index_connection_id": search_connection_id, "index_name": "example-index", }], ) thread = project.agents.create_thread() project.agents.create_message(thread.id, role="user", content="What's our refund policy for enterprise plans?") run = project.agents.create_and_process_run(thread.id, agent.id) No manual embedding calls at query time. No hand-written "retrieve top-k, stuff into prompt" logic. The agent framework does that internally, and it does it well enough that I stopped fighting it after the first try. Step five: For anything beyond simple lookups, I turned on agentic retrieval in Azure AI Search. Classic RAG fires one query per user turn, which quietly falls apart the moment someone asks a compound question — "compare our Q3 and Q4 policy and tell me what changed for renewals" is two questions wearing a trench coat. Agentic retrieval breaks that into sub-queries, runs them in parallel, and merges the results before generation. If your users ask messy, multi-part questions — and they do — turn this on from day one. Retrofitting it later is more annoying than it should be. Step six: Tested in the playground, then deployed the same agent behind a REST endpoint. Nothing about the agent changed between prototype and production. That alone would've saved me a full day on past projects. Now, the Part Everyone Skips I'm not going to pretend this is magic, because it isn't, and the tutorials that pretend otherwise are setting people up to get burned in a security review. Access control is on you. Foundry doesn't look at your documents and infer that HR files shouldn't be visible to the sales team. You configure document-level security filters in Azure AI Search yourself, and if you skip this, you've built a very articulate way to leak sensitive data. API keys are a prototype crutch, not a production plan. Move to Microsoft Entra ID before anything customer-facing goes live. This migration is a real afternoon of work, not a checkbox — budget for it. Retrieved documents are untrusted input. Prompt injection through a poisoned PDF is a real attack surface in every RAG system, Foundry included. Your system instructions need to assume the retrieved content might be trying to manipulate the model, because eventually it will. The costs stack. Embedding generation, index storage, and the extra tokens from stuffing retrieved passages into every call — none of this is free, and it compounds faster than people expect once you're past a demo and into real traffic. Model it before you commit to a chunking strategy at scale, not after. Was It Actually Worth It? Yes — but not for the reason most "look how easy this is" posts claim. The value isn't that RAG got simple. Grounding a model in the right data, with the right access controls, still takes real thought. The value is that Foundry took the boring week — the SDK wrangling, the manual retrieval loops, the glue code nobody wants to own — and turned it into an afternoon of configuration. That frees up the time you actually need for the parts that matter: is your data any good, is it chunked sensibly, and can you trust what comes back? If you've been putting off a RAG project because the infrastructure felt like too much, this is the moment to try again. Just don't skip the access control step to save time. That's the part that actually bites.

Language models become much more useful when they can answer questions about information they were never trained on, including your internal documentation, product manuals, policies, and other proprietary data. Prompting alone cannot solve this, because the model simply does not have access to that knowledge. Retrieval-Augmented Generation, or RAG, is the most common way to bridge that gap. Spring AI comes with solid support for building RAG systems. It has been almost three years since Spring AI showed up, and in that time it has grown from an experimental member of the Spring portfolio into a mature layer over chat models, embedding models, vector stores, and the plumbing that sits between them, which happen to be exactly the pieces a RAG system needs. In this article, we build a small but complete RAG service with Spring AI 2.0. The application reads a set of documents into a PostgreSQL vector store, retrieves the fragments that are relevant to a user question, and lets Anthropic's Claude put together the answer based on those fragments. Everything runs from a standard Spring Boot project, and every step can be reproduced on macOS, Windows, or Linux. The full project is available on GitHub. If you just want to see the finished result, or you would rather skip the step-by-step build below, you can clone the repository and run it as it is. Everyone else can follow along and generate this project from scratch. The prompts themselves are kept deliberately simple. You can tune retrieval and prompts forever; here we care about the architecture and how the pieces fit together in Spring. Approach RAG is not really a single feature. It is more of a small pipeline, and the code below makes a lot more sense once its parts have names. Embedding: a vector of numbers that captures the meaning of a piece of text. Texts that mean similar things end up with vectors that are close to each other.Embedding model: the model that computes these embeddings. It is a different model from the chat model, and it has a different job.Vector store: a database that keeps text fragments together with their embeddings and can answer the question, "which stored fragments are closest in meaning to this query?"Chunking: documents are too large to embed and retrieve as a whole, so we split them into smaller fragments (chunks) before storing them.Similarity search: we embed the user question and fetch the top-k closest chunks from the store.Augmentation: we append the retrieved chunks to the user question before sending it to the chat model, so the model answers from the context we provided instead of from its training data. One thing here is worth calling out, because it shapes the whole setup of the project: the LLM model used in chat and the embedding model are two separate choices. As of today, Anthropic offers LLM models but no embedding API, so a Claude-based RAG system always has to pair Claude with an embedding model from somewhere else. Rather than bringing in a second cloud provider and a second API key, this project computes embeddings locally (inside the JVM), using Spring AI's ONNX transformers module and the well-known all-MiniLM-L6-v2 sentence transformer. It is free and fast enough for this, and it keeps everything on one API key. In our scenario, the service is an internal assistant for a fictional company called Nimbusfield Systems, and it answers employee questions based on the company handbook. The company and the handbook are fictional on purpose. Claude cannot possibly know about it, which makes it easy to verify that the answers really come from our documents and not from the model's own memory. We build this in three steps: Expose a /ask endpoint backed by Claude, with no retrieval, and show that the model cannot answer handbook questions.Ingest the handbook into PGvector at application startup: read, chunk, embed, and store.Attach Spring AI's QuestionAnswerAdvisor to the same ChatClient and ask again. Prerequisites Java 21Maven 3.9.x (the Maven wrapper included in generated projects works too)Spring Boot 4.0.xSpring AI 2.0.0Docker Desktop (macOS/Windows) or Docker Engine (Linux), used only to run PostgreSQL. A project skeleton can be generated at start.spring.io by selecting Web, Anthropic Claude, PGvector Vector Store, and Docker Compose Support. The remaining Spring AI modules are added manually below. The Claude API Key Sign in (or sign up) at the Anthropic Console, open Settings, then API Keys, and create a new key. New accounts may need a small prepaid credit before the API accepts requests, but the runs in this article cost only a few cents. The key is shown only once, so store it right away as an environment variable. If you would rather not spend anything at all, you can still follow along and read through the steps without running the calls yourself. macOS/Linux: export ANTHROPIC_API_KEY=sk-ant-... Windows (PowerShell, persists across sessions after reopening the terminal): setx ANTHROPIC_API_KEY "sk-ant-..." Solution Dependencies With the Spring AI BOM in place, there is no need to repeat versions on the individual artifacts. Initializr expresses the BOM's own version as a property rather than a hardcoded literal, so there is a single place to bump it later: XML <properties> <java.version>21</java.version> <spring-ai.version>2.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> A common source of confusion is that start.spring.io has no dependency literally named "Spring AI." Each provider- or store-specific starter (Anthropic Claude, PGvector Vector Database, and so on) is itself a Spring AI module, and picking one transitively pulls in the framework's core classes. (like ChatClient, VectorStore, etc.) Selecting any one of them is also what makes Initializr add the spring-ai-bom as shown above to the generated pom.xml for you. The BOM itself is never a separate item you tick on the Initializr dependency screen. The application needs six Spring AI modules on top of the web starter, each one with a single responsibility. XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> <!-- Chat model: Anthropic Claude --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-anthropic</artifactId> </dependency> <!-- Embedding model: local ONNX sentence transformer --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-transformers</artifactId> </dependency> <!-- Vector store: PostgreSQL + pgvector --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-vector-store-pgvector</artifactId> </dependency> <!-- RAG advisor --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-vector-store-advisor</artifactId> </dependency> <!-- Document reading (PDF, Word, Markdown, HTML, and more) --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-tika-document-reader</artifactId> </dependency> <!-- Starts the database container on application startup --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <!-- Docker Compose service connections for Spring AI vector stores --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> </dependencies> Two models are referenced from the code here. One is the chat model, Claude, which is served from the Anthropic API. The other is the embedding model, which runs locally, right inside the application. We will look at that local embedding model in the next section. The Embedding Model By default, the transformers starter fetches tokenizer.json and model.onnx from Spring AI's own GitHub repository the first time the application starts and then caches them locally. In practice, this default setup is a bit fragile. raw.githubusercontent.com may rate-limit unauthenticated requests, and model.onnx (which is roughly 90 MB) is stored via Git LFS, whose bandwidth quota can run out independently of the ordinary rate limit. When that happens, the endpoint serves the small LFS pointer stub instead of the binary, with a normal-looking HTTP 200, and the failure only shows up later as a cryptic ONNX Runtime protobuf-parsing error rather than a clear download error. The fix is to bundle both files with the application instead of fetching them at startup. So we download them once: Shell mkdir -p src/main/resources/onnx/all-MiniLM-L6-v2 curl -fL -o src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json \ https://raw.githubusercontent.com/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json curl -fL --http1.1 -o src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx \ https://media.githubusercontent.com/media/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx Then we point the embedding model at these local files in our application.properties, overriding the GitHub-backed defaults: Properties files spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json With these two properties set, the application never touches the network for the embedding model, neither on the first run nor on any run after it. The Database The pgvector team publishes a PostgreSQL image with the extension already installed. A compose.yaml in the project root is all we need: YAML services: pgvector: image: "pgvector/pgvector:pg17" environment: - "POSTGRES_DB=nimbusfield" - "POSTGRES_USER=nimbusfield" - "POSTGRES_PASSWORD=nimbusfield" labels: - "org.springframework.boot.service-connection=postgres" ports: - "5432" The labels entry is important. Spring Boot's Docker Compose support auto-detects connection details by matching the image name against a list of well-known images. Plain Postgres is on that list, but pgvector is not, since it is a third-party image. The label tells Spring Boot to treat this container as if it were the official Postgres image, and that is what actually makes the automatic connection wiring work. If we omit it, the container still starts, but Spring Boot never creates a ConnectionDetails bean for it, so the run fails with a connection error rather than falling back gracefully. Because spring-boot-docker-compose is on the classpath, running the application starts the container automatically and injects the connection details. This works the same way on macOS and Windows, as long as Docker Desktop is running. Anyone who prefers to manage the container manually can run the same image with docker run -p 5432:5432 .. and set the datasource properties explicitly. Configuration The complete application.properties, now including the embedding model overrides shown earlier: Properties files spring.ai.anthropic.api-key=${ANTHROPIC_API_KEY} spring.ai.anthropic.chat.model=claude-sonnet-5 spring.ai.anthropic.chat.max-tokens=1024 spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json spring.ai.vectorstore.pgvector.initialize-schema=true spring.ai.vectorstore.pgvector.dimensions=384 spring.ai.vectorstore.pgvector.index-type=HNSW spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE logging.level.org.springframework.ai.chat.client.advisor=DEBUG Four details matter here. First, max-tokens is mandatory for the Anthropic API, which caps every response explicitly. Spring AI provides a default, but it is better stated than left implied. Second, the two spring.ai.embedding.transformer.* properties point the embedding model at the local files we bundled in the previous section, instead of Spring AI's own GitHub-backed defaults. See "The Embedding Model" above for why this matters. Third, initialize-schema=true enables the automatic creation of the vector-store table and the required extensions. (Since Spring AI 1.0, this no longer happens silently by default.) Fourth, dimensions=384 must match the embedding model. all-MiniLM-L6-v2 produces 384-dimensional vectors. If the embedding model changes later, the table has to be recreated, because the column type is vector(384). The Documents Two short Markdown files under src/main/resources/docs play the role of the company handbook. remote-work-policy.md Markdown # Nimbusfield Systems Remote Work Policy Employees may work remotely up to three days per week. Remote days must be registered in the portal by Thursday of the preceding week. Working from abroad is permitted for a maximum of 30 calendar days per year and requires prior approval from both the line manager and the People team. travel-expenses.md: Markdown # Nimbusfield Systems Travel and Expenses The daily meal allowance for business trips is 65 EUR in Europe and 80 USD elsewhere. Taxi rides are reimbursed only between airports, hotels, and client sites. Flights longer than six hours may be booked in premium economy. All expense reports are due within 15 working days after the trip via the portal. Thanks to the Tika reader used below, dropping PDFs or Word documents into the same folder works without any code changes. Step 1: Chat Without Retrieval We start with a service that wraps a ChatClient, built once from the auto-configured builder: Java @Service public class AssistantService { private final ChatClient chatClient; public AssistantService(ChatClient.Builder builder) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .build(); } public String ask(String question) { return chatClient.prompt() .user(question) .call() .content(); } } And a controller associated with it: Java @RestController public class AssistantController { private final AssistantService assistantService; public AssistantController(AssistantService assistantService) { this.assistantService = assistantService; } @GetMapping("/ask") public ResponseEntity<String> ask(@RequestParam("question") String question) { return ResponseEntity.ok(assistantService.ask(question)); } } Start the application (./mvnw spring-boot:run on macOS/Linux, mvnw.cmd spring-boot:run on Windows) and ask it a handbook question: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The response, as we might expect, is: I don't have that information in my available knowledge base. Nimbusfield Systems' specific travel and expense policy—including per diem rates for European business trips—isn't something I can confirm accurately. To get the correct figure, please check: The company's Travel & Expense Policy document (likely on the intranet/HR portal)Your Finance or HR department directlyYour manager, if travel budgets are pre-approved per trip Would you like help with anything else I can assist with more reliably? This gives us a baseline. The model behaves correctly given what it knows, which is nothing at all about this company. Step 2: The Ingestion Pipeline Ingestion follows Spring AI's extract, transform, load structure: a DocumentReader extracts the text, a TextSplitter chunks it, and the VectorStore embeds and stores the chunks. The embedding call happens implicitly inside vectorStore.add() call. The auto-configured TransformersEmbeddingModel is wired into the PgVectorStore and each chunk is embedded into the table. Java @Component public class HandbookIngestion implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(HandbookIngestion.class); private final VectorStore vectorStore; private final JdbcTemplate jdbcTemplate; private final Resource[] handbook; public HandbookIngestion(VectorStore vectorStore, JdbcTemplate jdbcTemplate, @Value("classpath:docs/*.md") Resource[] handbook) { this.vectorStore = vectorStore; this.jdbcTemplate = jdbcTemplate; this.handbook = handbook; } @Override public void run(ApplicationArguments args) { Integer count = jdbcTemplate.queryForObject( "select count(*) from vector_store", Integer.class); if (count != null && count > 0) { log.info("Vector store already contains {} chunks, skipping ingestion", count); return; } TokenTextSplitter splitter = TokenTextSplitter.builder() .withChunkSize(300) .build(); for (Resource resource : handbook) { List<Document> documents = new TikaDocumentReader(resource).get(); documents.forEach(doc -> doc.getMetadata().put("source", resource.getFilename())); List<Document> chunks = splitter.apply(documents); vectorStore.add(chunks); log.info("Ingested {} chunks from {}", chunks.size(), resource.getFilename()); } } } The count check makes ingestion idempotent, so restarting the application does not duplicate every chunk. And the source metadata attached to each chunk enables filtered searches later, for instance restricting retrieval to a single document. That same idempotency check has a practical downside worth pointing out. Once the vector store has data, restarting the application will not pick up edits to the handbook files, since the count check short-circuits before the splitter ever runs. To force a clean re-ingestion, for instance after changing a handbook document, tear down the container together with its data volume, not just the container: docker compose down -v The chunk size of 300 tokens is generous for documents this small. The splitter's default of 800 is aimed at larger, real-world content. Chunking is the least exciting and yet the most important knob in a RAG system: chunks that are too large dilute the similarity signals, while chunks that are too small lose their context. It is worth experimenting here: try a few different chunk sizes and see how the system behaves. Just remember to run docker compose down -v between runs, so the vector store is rebuilt from scratch each time. Step 3: Attaching the Retrieval Advisor Now we come back to the plain AssistantService from Step 1 and upgrade it, rather than writing something new. The ChatClient wiring we built earlier stays and what changes is what gets attached to it. Spring AI models the cross-cutting concerns around a chat call as "advisors", which are conceptually close to interceptors. The QuestionAnswerAdvisor embeds the incoming user question, runs a similarity search against the vector store, and appends the retrieved chunks to the prompt before it reaches Claude. Enabling RAG is therefore a change to how the ChatClient is constructed, not to how the request is handled: Java public AssistantService(ChatClient.Builder builder, VectorStore vectorStore) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .defaultAdvisors( QuestionAnswerAdvisor.builder(vectorStore) .searchRequest(SearchRequest.builder() .topK(4) .similarityThreshold(0.5) .build()) .build(), new SimpleLoggerAdvisor()) .build(); } topK(4) retrieves at most four chunks per question, and similarityThreshold(0.5) discards weak matches, so an entirely unrelated question augments the prompt with nothing rather than with noise. The SimpleLoggerAdvisor, combined with the DEBUG logging property we set earlier, prints the fully augmented prompt. This is the single most useful debugging tool while tuning retrieval, because it shows exactly what Claude was given. We restart and repeat the same request: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The daily meal allowance for business trips in Europe is 65 EUR. Same model, same question, and this time a precise answer grounded in the retrieved handbook chunk instead of a generic deflection. The debug log confirms what is going on behind the scenes: the user question arrives at Claude wrapped in a prompt that contains the retrieved handbook fragments as context. Going Further The default behavior of QuestionAnswerAdvisor is usable, but there are two refinements worth implementing if you want to take this pattern further. The first one concerns grounding. Even with retrieved context, the model may fall back on its general knowledge when the context does not actually contain the answer. The advisor accepts a custom PromptTemplate that controls how the question and the context are merged, and this is the place to enforce stricter behavior. The template must contain the query and question_answer_context placeholders: Java PromptTemplate strictTemplate = PromptTemplate.builder() .template(""" {query} Answer strictly based on the context below. If the context does not contain the answer, reply exactly: "This is not covered by the handbook." --------------------- {question_answer_context} --------------------- """) .build(); QuestionAnswerAdvisor advisor = QuestionAnswerAdvisor.builder(vectorStore) .promptTemplate(strictTemplate) .build(); Asking about, say, the parental leave policy (which is absent from our two files) now produces the fixed refusal instead of an invention. If people are going to rely on it, you want this on. The second refinement could be structured output, and it composes cleanly with retrieval. Declaring a record and calling .entity() instead of .content() gives back a typed object, with Spring AI instructing the model to respond in the matching JSON schema: Java public record HandbookAnswer(String answer, String sourceHint, boolean coveredByHandbook) { } public HandbookAnswer askStructured(String question) { return chatClient.prompt() .user(question) .call() .entity(HandbookAnswer.class); } A last note on the embedding choice. A local MiniLM model is not the strongest embedding model available, and for a large multilingual corpus a hosted embedding API or a bigger ONNX model would retrieve better. This choice is easy to reverse: EmbeddingModel is an interface, swapping the implementation is a matter of a dependency and a property, and the only hard constraint is the one mentioned earlier: the vector dimensions in PGvector have to match whatever the embedding model produces. Conclusion In this article, we built the RAG flow step by step. We started with a plain chat endpoint that could not answer anything about the Nimbusfield handbook, because Claude had never seen it. We then ingested that handbook into PGvector, embedding each chunk locally, and attached Spring AI's QuestionAnswerAdvisor to the same client. That single change was enough to turn a generic model into a service that answers from your own documents. After that, we talked about how we can tighten the grounding, so the model says it does not know when the context has no answer, and pulled the response straight into a typed Java record. If you want to take it further, clone the project, point it at your own documents, apply further the techniques we discussed in the Going Further section, play with different chunk sizes, retrieval settings, and prompts to see how the answers change. The Spring AI documentation goes deeper into advisors, vector stores, and retrieval configuration. The complete, runnable project is available on GitHub.

By Murat Balkan CORE

Building a single AI agent is not usually the hard part. You send a prompt to a model, get a response back, and wire it into your app. Done. The hard part starts when that agent becomes one step in a larger system. A real AI workflow might need to ingest a file, extract text, chunk it, generate embeddings, call an LLM, write results to a database, sync to an external API, and notify a user. Those steps do not behave the same. Text extraction might finish in seconds. An LLM call might take minutes. A sync job might fail because some external API is having a bad day. That is where a lot of "agent" systems stop looking magical and start looking like regular distributed systems. I have seen this fail in boring ways: The same job gets processed twice.A worker writes to the database, then crashes before marking the job complete.A model call runs longer than expected and the message gets picked up again.A retried tool call creates duplicate external writes.Failed jobs sit in processing until someone manually checks the database. None of this is new. AI agents do not magically avoid old infrastructure problems. They still need queues, retries, idempotency, durable state, and monitoring. AWS SQS is a good fit for that middle layer. It is not a full workflow engine. I would not use it for every orchestration problem. But if you need a durable queue between independent agent stages, SQS is simple, reliable, and usually enough. The Coordination Problem A basic multi-stage AI workflow often looks like this: Plain Text Input source -> ingestion -> processing -> generation -> sync The first version is usually a database table with a status column. That works for a while. Then concurrency shows up. Two workers read the same pending row. A process crashes and leaves a job stuck in processing. Someone adds sleep(30) because the previous step "usually finishes by then." That last one is the kind of fix that works just long enough to become a production bug. A queue gives each stage a cleaner boundary. One stage publishes work. Another stage consumes it. If the next stage slows down, the queue absorbs the backlog instead of forcing the whole pipeline to wait. Plain Text Input Source -> ingest_queue -> Ingestion Worker -> chunk_queue -> Chunking Worker -> embedding_queue -> Embedding Worker -> summary_queue -> Summary Worker -> sync_queue -> Sync Worker Now ingestion can scale separately from summarization. If LLM generation is slow, messages pile up in summary_queue. That is not automatically a failure. That is what the queue is there for. A failed summary worker does not corrupt the whole workflow. The message can be retried. If it keeps failing, it moves to a dead letter queue. Standard Queues vs. FIFO Queues SQS gives you two main queue types: standard queues and FIFO queues. Standard Queues Standard queues give at-least-once delivery and best-effort ordering. A message can be delivered more than once. Messages may not arrive in the exact order sent. That sounds scary, but most background AI work should already handle this. Use standard queues for work like document processing, embedding generation, batch classification, independent user requests, and webhook processing. For these jobs, throughput matters more than strict ordering. FIFO Queues FIFO queues preserve ordering within a MessageGroupId and support deduplication. Use when sequence actually matters: conversation turns, per-user workflows, ordered state transitions. Python response = sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps(payload), MessageGroupId=payload["user_id"], MessageDeduplicationId=payload["task_id"] ) Be careful with the group ID. If every message uses the same MessageGroupId, you have serialized the whole queue by accident. Give each conversation, user, or workflow its own group ID so you preserve ordering per entity while allowing parallelism across different ones. My default rule: start with standard queues unless ordering is clearly required. Then make the handler idempotent. That matters more than the queue type. Ensuring Idempotency in Your Agent Flow Idempotency means the same task can run more than once without creating duplicate or incorrect side effects. This is the part I would not skip. SQS standard queues use at-least-once delivery, so duplicates are part of the contract. But this matters even more with AI workloads because model calls are expensive and outputs can be non-deterministic. Retrying the same prompt may cost money and return a different answer. Retrying the same tool call may send a duplicate email or write a second database row. The basic pseudo workflow: Plain Text receive message check if task already completed if completed, delete message and exit if not completed, process task store result delete message Simple version: Python def handle_message(message, store, sqs, queue_url): payload = json.loads(message["Body"]) task_id = payload["task_id"] if store.already_completed(task_id): sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "skipped", "task_id": task_id} result = run_agent_logic(payload) store.mark_completed(task_id, result) sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "completed", "task_id": task_id} The store can be Postgres, DynamoDB, Redis, or anything durable with atomic writes. For Postgres, a unique constraint saves you: SQL CREATE TABLE agent_task_results ( task_id TEXT PRIMARY KEY, status TEXT NOT NULL, result JSONB ); INSERT INTO agent_task_results (task_id, status) VALUES ($1, 'processing') ON CONFLICT (task_id) DO NOTHING; If the insert succeeds, this worker owns the task. If it does nothing, another worker already claimed or completed it. The Failure Case I Designed Around Plain Text summary_queue -> Summary Worker -> Postgres -> sync_queue The summary worker receives a message, calls an LLM, writes the summary to Postgres, then deletes the SQS message. Now suppose the worker writes to Postgres but crashes before deleting the SQS message. From SQS's point of view, the job never finished. After the visibility timeout expires, another worker receives the same message and runs the task again. Without idempotency, that retry may call the LLM again, generate a slightly different summary, and write a second result. A safer handler checks whether model output already exists before calling the model: Python def summary_handler(payload, store): task_id = payload["task_id"] existing = store.get(task_id) if existing and existing.get("model_output"): summary = existing["model_output"] else: text = load_text(payload["input"]["text_uri"]) summary = call_llm(text) store.save_model_output(task_id, summary) store.save_final_result(task_id, {"summary": summary}) return {"next_stage": "sync", "next_input": {"summary_task_id": task_id} That avoids repeating the expensive part if the first attempt already got that far. Visibility Timeout When a worker receives a message, SQS hides it from other workers for the visibility timeout. If the worker finishes, it deletes the message. If the worker crashes, the message becomes visible again after the timeout expires. Too short: another worker receives the same message while the first is still running. Duplicate execution. Too long: failed jobs take too long to retry. Plain Text visibility_timeout = 2x to 5x expected processing time Reference: Metadata validation: 30-60 secondsEmbedding generation: 1-5 minutesLLM-heavy summary: 5-15 minutesLong document analysis: 15+ minutes with heartbeat For long-running tasks, extend visibility: Python sqs.change_message_visibility( QueueUrl=queue_url, ReceiptHandle=receipt_handle, VisibilityTimeout=extension_seconds ) The message should describe the work, not carry the workload. Bad: JSON {"task_id": "123", "full_pdf_text": "... thousands of lines ..."} Better: JSON { "task_id": "123", "stage": "summarize", "input": {"document_uri": "s3://bucket/docs/input.pdf"}, "metadata": {"user_id": "789", "priority": "normal"} } Store large files in S3. Send references through SQS. Do not let the queue become your storage layer. Dead Letter Queues A DLQ captures messages that fail repeatedly. Without one, poison messages cycle forever. Python sqs.set_queue_attributes( QueueUrl=main_queue_url, Attributes={ "RedrivePolicy": json.dumps({ "deadLetterTargetArn": dlq_arn, "maxReceiveCount": 5 }) } ) Use 3-5 as a starting point. A DLQ is not a trash bin - it's an alert. AI-Agent-Specific Failure Modes Duplicate LLM calls: Bigger bill, possibly different answer. Use task_id as idempotency key.Non-deterministic outputs: Store first successful output.Tool-call side effects: Make idempotent.Long-running inference: Use visibility heartbeat. What to Monitor MetricWhyApproximateAgeOfOldestMessageUser-facing delayApproximateNumberOfMessagesVisibleBacklogDLQ message countRepeated failures Two alerts: Oldest message exceeds latency targetDLQ has messages When SQS Is Not the Right Tool RequirementBetter fitSimple async tasksSQSVisual multi-step workflowStep FunctionsComplex event routingEventBridgeHuman approvalsStep Functions I have seen teams burn hours building multi-agent systems with database polling and sleep timers. It works at demo scale. It usually does not survive production traffic. SQS gives you durable message delivery primitives. But the app still needs idempotent handlers, visibility timeout tuning, and DLQ monitoring. Default architecture: One queue between major stagesStandard queues unless ordering requiredEvery handler idempotentLarge payloads outside the queueVisibility timeouts based on real processing timeDead letter queues for failures The difference between an AI demo and a reliable AI system is rarely the prompt. It is the infrastructure around the prompt. Build that layer intentionally.

Throughout my career, I’ve held many roles in the QA conversation: As a developer, waiting on test teams to validate features before a release could shipAs a tech lead, watching sprint capacity dwindle while we converted Jira stories into test cases by handAs an architect, auditing a test repository with 4,000 cases where no one knew which ones mattered So when “AI test generation” started appearing as part of every testing product, I was both interested…and skeptical. After spending time with several of these tools, I’ve decided that the phrase “AI test generation” covers two fundamentally different architectures. The first is a large language model sitting behind a prompt.The second is an actual testing agent that examines your requirements, attachments, and existing test library before writing any tests. The industry has started calling that second approach agentic test creation, and the gap between that and AI test generation is much wider than the names suggest. In this article, I’ll give examples of both, showing the differences along the way, so that you can understand which one a vendor is trying to sell you. The Problem With Generic AI Test Case Generation The first wave of AI test case generation tools followed a simple pattern. You paste in a user story, the tool wraps it in a prompt, sends it to a general-purpose LLM, and gives you back the response as test cases. It’s basically ChatGPT with a QA skin. If you’ve ever pasted a Jira ticket into ChatGPT and asked for test cases, you’ve already used this architecture. The vendor version adds a nice UI and an export button. As with much LLM output, the results here might look impressive at first glance. For example, give it this story: As a returning customer, I want to apply a promo code at checkout so that my discount is reflected in the order total. And you might get a dozen plausible test cases in just seconds, testing valid code, invalid code, expired code, empty field, case sensitivity, etc. I ran exactly this exercise against a mature e-commerce regression suite. The results? Seven of the 12 generated cases already existed in the checkout suite, some nearly word-for-word in intent.Two referenced an “Apply Discount” button that simply doesn’t exist.None of the test cases were linked back to a requirement, so traceability remained manual. The model did exactly what it was asked to do. The problem was that it didn’t have context. As you can imagine, over time this approach produces some pretty bad effects. You’ll start seeing problems like duplicated coverage, rising regression time, near-copies of the same test that drift apart, and more. And you’ll end up with a repository you just don’t trust. What Is Agentic Test Creation? Agentic test creation, on the other hand, is a workflow where an AI agent, given a requirement, plans and executes a multi-step process that: Gathers context from the requirement and its attachmentsAnalyzes the existing test libraryFigures out what’s already coveredGenerates test cases that fill the gaps, each one linked back to the requirement. The word “agentic” is important here. A single LLM call is a stateless function: prompt goes in, text comes out. An agent, on the other hand, is a loop. The model reasons about a goal, calls tools to gather information, observes the results, and revises its plan before producing output. The ReAct paper formalized this “reasoning-plus-action” loop, and Anthropic’s Building Effective Agents is the clearest practitioner-level treatment I’ve read. Here’s an overview of the difference in the two approaches: Generic ai test generationagentic test creation Input The text you paste in as a prompt Requirement, acceptance criteria, attachments and images, existing test cases, SDLC history Awareness of existing coverage None Checks the test cases linked to the requirement before generating Duplication Frequent; every run starts from zero Existing cases are reused instead of regenerated Traceability Manual, after the fact Each generated case links to its requirement Typical failure mode Redundant, or references UI that doesn’t exist Gaps in context How an Agentic Test Creation Workflow Runs Let’s go back to the promo code story; this time we’ll run it through an agentic pipeline: The agent parses the story, its acceptance criteria, and an attached checkout mockup.It queries the existing test library and finds 40 checkout-related test cases.It maps the story’s scenarios against those 40 and identifies that seven are already covered. It reuses those cases rather than regenerating them.It creates six new cases targeting real gaps: promo code combined with a gift card, currency rounding on percentage discounts, an expired code entered against a saved payment method, etc.Each new case has a link back to its requirement.A QA engineer reviews the batch, edits two cases, rejects one, and commits the rest. As you can see, this agentic pattern is a serious improvement over vanilla AI and is becoming best practice. Implementing Agentic Test Creation There are basically two ways to implement agentic test creation: buy a commercial product or roll your own. Commercially, you can see an example of agentic test creation with Tricentis’ qTest. Here, an agent runs inside the test management platform itself, where it follows the above loop: it analyzes a requirement, considers the attachments, reuses test cases linked to that requirement, and stamps each case with a marker for the reviewer.To assemble your own version of the loop, you can wire agent frameworks to your test infrastructure through tools like the open-source Playwright MCP server and Selenium or Playwright projects. As is typically the case with build vs. buy, build costs money and time, but can create value using your context plumbing. Where Agentic Test Creation Falls Short What are some downsides to agentic test creation? There are a few, though they are minor. First, the context loop adds cost and latency to every generation. You’ll pay for library queries and multiple model calls per requirement instead of just one. Second, the coverage mapping is only as good as your repository. A messy library means messy output. And third, a review gate only works if reviewers stay engaged. Approving a 30-case batch on a Friday afternoon? That can fail just as it always has. What Changes Day to Day for QA Engineers? The engineers I know are skeptical of AI tooling. And they have good reasons. Agentic test creation changes their daily job (though I believe the change is less than the marketing suggests). The role doesn’t go away; it just…shifts. QA engineers move from authors to reviewers. Instead of hand-writing the fifteenth variation of a checkout test from a Jira ticket, they now evaluate a proposed batch, check the coverage map, and approve or reject the outputs. Accountability still stays with the QA team: a human sign-off gates everything. But the hours previously spent transcribing requirements into test steps are now spent on the quality assurance testing methods that models handle poorly: exploratory testing, risk analysis, and deciding what should be tested in the first place. Transcribing was the boring part anyway, right? Practical First Steps Finally, here are a few suggestions on agentic testing based on my experiences: Run a duplication audit first. An agentic tool builds on whatever it finds in your repository, including your duplicates. Clean input makes for better output.Pilot one project. Demos hide failures. Use a real project with real Jira tickets. Challenge your solution with missing acceptance criteria and stale attachments! Find your holes quickly.Keep the human review. Treat every generated case as a draft. And track your reviewer rejection rate. It’s a great signal of whether your approach is working.Measure. Keep track of three numbers: duplication rate, reviewer rejection rate, and elapsed time from requirement to test.Ask vendors: “What does your tool read before it generates?” If the answer is “your prompt,” you’re looking at a ChatGPT wrapper. If the answer includes existing tests, attachments, and requirements history, you’re looking at an agentic solution. Conclusion My readers may recall my personal mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” — J. Vester Hand-transcribing user stories into test steps has never extended the value of anyone’s intellectual property. Agentic test creation, whether you go commercial or build your own, with a review gate in place, sends that transcription task to a machine where it belongs, leaving you to be the all-important human in the loop. Have a really great day!

By John Vester CORE