DZone Spotlight

Wednesday, September 9 View All Articles »

Memory-First Indexes in SQL Server 2025: Redefining Performance for Hybrid Workloads

By arvind toorpu

CORE

Modern database environments rarely run a single type of workload. Most production systems handle both transactional operations and analytical queries simultaneously. These mixed workloads, often referred to as hybrid workloads, place significant pressure on traditional database indexing and storage strategies. In such environments, disk-based indexes can become a performance bottleneck. When transactional and analytical queries compete for disk I/O, it often results in increased latency, reduced throughput, and inconsistent query performance. To address these challenges, SQL Server leverages memory-optimized tables and indexes as part of its In-Memory OLTP capabilities. These features reduce reliance on disk I/O by enabling data and index access directly from memory, while still maintaining durability through logging and checkpoint mechanisms. This article explores how memory-optimized indexing works and demonstrates how it can significantly improve performance in real-world hybrid workload scenarios. Core Characteristics Mandatory inclusion: Every memory-optimized table must have at least one index, as they serve as the "entry points" for row access.Purely in-memory: Indexes are rebuilt entirely from scratch during database recovery based on their definitions and the data loaded into memory.Non-persistent: Unlike traditional indexes, changes to these indexes are not written to the transaction log, reducing I/O overhead.Fragmentation-free: These structures do not suffer from traditional page fragmentation, eliminating the need for regular REORGANIZE or REBUILD operations. Index TypeBest Use CaseBehaviorHash IndexEquality SearchesUses an array of buckets; highly efficient for point lookups (e.g., WHERE ID = 5).Nonclustered IndexRange QueriesUses a lock-free B-tree structure (Bw-tree); ideal for range scans and sorted results (e.g., WHERE Price > 100). The Challenge With Traditional Indexing Traditionally, database indexes are stored on disk to ensure durability. While this design protects data, it introduces a major limitation: disk I/O latency. In environments with heavy workloads, disk access becomes a bottleneck. This is particularly noticeable when: Large analytical queries scan index rangesTransactional queries require fast point lookupsMany concurrent users access the system When both workloads run together, index operations often compete for disk resources, resulting in slower queries and higher latency. Introducing Memory-First Indexes Memory-First Indexes in SQL Server 2025 take a different approach. Instead of relying primarily on disk-based indexes, the system prioritizes in-memory index access for frequently used data while maintaining a synchronized copy on disk for durability. The key idea is simple: Hot data (frequently accessed index ranges) is kept in memory.Cold data remains on disk.Changes made in memory are synchronized with disk replicas in the background. This approach allows SQL Server to serve many queries directly from memory while still maintaining persistence. The feature also includes monitoring mechanisms that track query patterns. When the system detects frequently accessed index partitions, it moves them into memory automatically. Less frequently accessed portions are pushed back to disk to conserve memory resources. The result is faster query execution without requiring manual tuning from database administrators. Real-World Example: Retail E-Commerce Database To understand the benefits, consider a retail company running an e-commerce platform. The company stores millions of products in a table with the following structure: ProductID – unique identifierProductCategory – category of the productPrice – product priceStockQuantity – available inventory The application runs two types of queries. Transactional Query This query checks stock availability for a specific product. SQL SELECT StockQuantity FROM Products WHERE ProductID = 102345; Analytical Query This query calculates aggregated metrics by product category. SQL SELECT ProductCategory, AVG(Price) AS AvgPrice, SUM(StockQuantity) AS TotalStock FROM Products WHERE Price > 500 GROUP BY ProductCategory; In a traditional setup, both queries rely on disk-based indexes. When concurrency increases, disk access becomes saturated, and query performance suffers. With Memory-First Indexes, the most frequently used index ranges, such as ProductID and ProductCategory, are loaded into memory, allowing much faster lookups. Testing the Feature To evaluate the impact of Memory-First Indexes, we can simulate a large dataset and compare query performance before and after enabling the feature. Step 1: Create the Table SQL CREATE TABLE Products ( ProductID INT PRIMARY KEY, ProductCategory NVARCHAR(50), Price DECIMAL(10,2), StockQuantity INT ); Step 2: Populate Test Data The following script generates a large dataset for testing. SQL INSERT INTO Products (ProductID, ProductCategory, Price, StockQuantity) SELECT TOP 50000000 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS ProductID, CASE WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 5 = 1 THEN 'Electronics' WHEN ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 5 = 2 THEN 'Clothing' ELSE 'Home Appliances' END AS ProductCategory, ABS(CHECKSUM(NEWID()) % 1000) + 1.00 AS Price, ABS(CHECKSUM(NEWID()) % 5000) + 1 AS StockQuantity FROM sys.all_objects a CROSS JOIN sys.all_objects b; Step 3: Create Traditional Indexes SQL CREATE INDEX IX_Products_ProductID ON Products (ProductID); CREATE INDEX IX_Products_Category ON Products (ProductCategory); At this stage, run the transactional and analytical queries and capture baseline metrics using Query Store or dynamic management views. Step 4: Enable Memory-First Indexes Next, recreate the indexes with Memory-First enabled. SQL DROP INDEX IX_Products_ProductID ON Products; CREATE INDEX IX_Products_ProductID ON Products (ProductID) WITH (MEMORY_FIRST = ON); DROP INDEX IX_Products_Category ON Products; CREATE INDEX IX_Products_Category ON Products (ProductCategory) WITH (MEMORY_FIRST = ON); Step 5: Execute Test Queries SQL SELECT StockQuantity FROM Products WHERE ProductID = 102345; MS SQL SELECT ProductCategory, AVG(Price) AS AvgPrice, SUM(StockQuantity) AS TotalStock FROM Products WHERE Price > 500 GROUP BY ProductCategory; Record execution time, CPU usage, and disk activity again. Observed Performance Improvements The results typically show noticeable performance gains. For example: Transactional queries Before: ~50 msAfter: ~15 ms Analytical queries Execution time reduced by about 50% System metrics also reveal additional improvements: Disk I/O reduced by more than 70%Memory usage increased only moderatelyCPU utilization became more stable during peak workloads These improvements occur because queries are able to retrieve indexed data directly from memory rather than waiting for disk operations. Why This Matters for Modern Workloads Hybrid workloads are becoming the norm across many industries, including retail, finance, and IoT platforms. Systems must support both real-time transactions and large analytical queries without sacrificing performance. Memory-First Indexes help address this challenge by: Reducing disk I/O bottlenecksImproving response time for critical queriesAutomatically adapting to changing workload patternsMaintaining durability with synchronized disk replicas Final Thoughts Memory-First Indexes represent an important improvement in SQL Server 2025’s indexing architecture. By prioritizing in-memory access for frequently used data, SQL Server can deliver significantly faster query performance while still preserving data durability. For organizations running mixed transactional and analytical workloads, this feature can reduce latency, improve system stability, and make better use of available hardware resources. As hybrid workloads continue to grow, features like Memory-First Indexing will play a key role in helping database platforms keep up with modern application demands. More

Cutting Telemetry Volume Is Not the Same as Cutting Noise

By Severin Neumann

Almost every conversation about observability budgets I have been in ultimately arrives at the same conclusion: “we need to reduce our telemetry volume.” That sentence is usually followed by a number. Thirty percent. Half. Whatever the finance spreadsheet needs it to be. Then someone says the thing that makes everyone in the room relax. "Good news: most of it’s noise anyway. We can cut the volume and improve the signal at the same time." It is a comforting idea, because it turns an unpleasant budget cut into an engineering improvement. But it only gets you so far. It is true that some of your telemetry is noise, but it’s much less of it than "most." But it doesn’t follow that you can then simply cut volume and automatically improve signal. There is real noise in your telemetry, and I will get to where it lives. But "reduce volume by thirty percent" is not an instruction to remove noise. It is an instruction to remove bytes, and your noise and your signal are made of the same bytes. The target doesn’t differentiate, so what you end up removing is dictated by whatever is easiest to find. What is easy to find is a category. All INFO logs. All user agent strings. Everything below WARN. Categories are easy because your pipeline already knows them, and that is the whole of their appeal. Whether a category happens to be useful or not is a coincidence. So your telemetry is full of junk, but the problem isn't that there is too much of it. It is that by adopting a volume reduction target, you are not looking at whether the telemetry data you cut has any value. Once you hit the byte target, the exercise is seen as a success. Two Axes, Loosely Coupled When you change your telemetry pipeline, two things move. The first is easy: bytes through the pipeline, or active series if it is metrics, or whichever unit your contract happens to price. One number, on a chart, updated hourly. This is what we call volume. The second is what those bytes enable you to find out. Whether, six weeks from now, you can still answer the question in front of you. This is what we commonly call signal, and everything else is noise. It is measurable, but it is not measured in bytes, and it is probably not on any chart you are currently looking at. The two are related, obviously. Delete everything, and both go to zero. But across the range you actually operate in, they are only loosely coupled, because the bytes in your telemetry are not distributed anything like the value. The smallest fields often do the most work. A tenant identifier is a few dozen bytes, and it tells you whether something is impacting everyone or just one customer. A trace ID is thirty-two hex characters, but without it you are correlating your signals by hand, across three browser tabs. If the resource attributes naming the deployment are missing, good luck telling a bad release from a bad node. On the flipside, fields that take the most space frequently do the least. Meanwhile, the ten-thousandth identical stack trace in an hour is several kilobytes and tells you the same thing the first one did. So a lever that operates on bytes will spend most of its effect in the wrong place, and no exchange rate exists that would let you convert one axis into the other. Drawing them as two axes is a crude picture for that reason. But it is still worth doing, because it separates four moves that a byte count reports as only two. Let me walk through each one. Q1: The Free Lunch, Real But Limited This is the noise I promised at the top, and finding it feels great. Every tutorial on making your observability pipeline better has these prominent examples: Kubernetes liveness and readiness probes logging every few seconds, per pod, forever. A debug logger somebody enabled during an incident last quarter, and nobody turned off. The same records shipped twice because a node agent and an application-level exporter both picked them up. Most of this can go. But be careful even here, because a health check is not the same thing as a worthless record. Probe failures and probe latency are how you find a sick node before your users do. What you want to drop is the successful ones, the ninety-nine percent that only ever confirm that nothing is happening. The filter processor will do it: YAML processors: filter/healthchecks: log_conditions: - 'IsMatch(log.attributes["http.route"], "^/(healthz|readyz)$") and log.attributes["http.response.status_code"] == 200' This assumes http.route has been promoted onto the log record; it is a span attribute by default, so on the trace side the equivalent lives under trace_conditions, with a span. prefix instead of log.. That status code check is the difference between Q1 and Q2. Without it, you have removed probe observability rather than probe noise, and you will find that out the next time readiness starts flapping and nothing in the logs can tell you when it began. With it, volume goes down, and signal is untouched, or arguably goes up, because you are no longer scrolling past successful probe traffic to find a real request. Sounds like a good deal, right? This is the quadrant everybody is imagining when they say "most of it is noise anyway." The same trade is available on the retry storm that repeats one stack trace ten thousand times in an hour. The logdedup processor collapses each ten-second window into one record carrying the count, so the storm stops drowning the query you are running, and you can still see how big it was. Finding the rest of this kind of waste means clustering records by shape and looking at what dominates, which is a different class of tool than a filter, and it is the part most volume-reduction programs skip. The challenge is that this quadrant is finite. In my experience, it is somewhere in the range of 10-20%, depending on how neglected the pipeline has been. If your mandate was 30%, you exhaust Q1 in the first week, and then you keep going, because the mandate does not stop when the free lunch does. Q2: Paying With Data Instead of Money So the free lunch got you 15%, the middle of that range, and the mandate was 30%, so the next 15% has to come out of data that somebody might actually need. Which is a good moment to read the mandate again, because almost nobody means it literally. "We need to reduce our telemetry volume by thirty percent" is very rarely a statement about telemetry. It is a statement about an invoice. Does anybody in that meeting actually want fewer log lines? They want a smaller number at the bottom of a bill. Volume is simply the variable their contract happens to be calculated on. The distinction matters because volume reduction and reducing your bill have different solution spaces. Reducing volume by 30% has one family of answers, and every one of them involves deleting something. Reducing observability spend by 30%, has a different set of options, several, and deleting your data is the one with the worst terms. A logging config goes from INFO to WARN and ships with the next release. Retention drops from thirty days to seven. Traces get sampled at 5%: YAML processors: probabilistic_sampler: sampling_percentage: 5 None of these options is free. Each one of them is defensible in isolation, and what makes them defensible is that they have a big impact. INFO is most of your log volume, seven days covers most incidents, and 5% is a perfectly good sample if all you want is a latency distribution. You end up paying the bill twice, but only one of the payments shows up on the invoice. You are also settling the bill in a second currency: answers you will not have, because you didn’t store the data needed for them. Nobody counts that. Nothing fails and nothing alerts, because a trace that was never recorded does not raise anything. When a customer sends an order ID on Thursday, and the trace behind it was one of the ninety-five per cent, the investigation stalls; somebody says we do not have that, and nobody goes back to look at the config change from earlier in the year that caused it to be dropped. My position is that most of this work should not exist. The engineering is fine! The sampler is correct, the retention change is correct, and both do exactly what they say on the tin. It is just that the whole exercise is effort spent making a bad unit price easier to swallow. It's like an old fridge: defrost it, keep the door shut, put less in it, and yes, your bill really does go down every month. Somebody should still go and look at what a new fridge costs. Q3: The Enrichment Nobody Gets To There is a second way to improve signal-to-noise: instead of removing noise, you add signal. You make the data you are already paying for be more useful. Attaching Kubernetes and cloud metadata with the k8sattributes processor, so a log line knows which namespace, deployment, node, and pod produced it. Parsing an unstructured message body into named, queryable fields with OTTL. Making sure trace context actually propagates across the boundary where it currently drops, so your logs and traces can be correlated instead of merely coexisting. Carrying code.file.path and code.line.number on the records that warrant it, so a log line points at the statement that emitted it instead of leaving you to grep the repository for the format string. YAML processors: k8sattributes: extract: metadata: - k8s.namespace.name - k8s.deployment.name - k8s.pod.name - k8s.node.name transform/parse_access_log: log_statements: - context: log statements: - merge_maps(attributes, ExtractPatterns(body, "^(?P<method>\\w+) (?P<path>\\S+) (?P<status>\\d{3}) (?P<duration_ms>\\d+)$"), "insert") These changes make your telemetry substantially more valuable, but they also increase volume. But most of that is cheaper than you would guess. The Kubernetes metadata are resource attributes, written once per batch in OTLP and shared by every record from the same pod, so at the collector's egress they cost a fraction of a byte per record. The parsing is the real exception: you keep the original body alongside the extracted fields, so the record roughly doubles, and no amount of batching recovers that. A bytes-per-day chart shows you none of that. The enrichment that costs almost nothing and the one that doubles every record show up the same way: the budget line went up. So the work never really gets argued about. Nobody is blocking k8sattributes – it ships enabled in half the Helm charts you might install – and most teams already intend to do all of the above. They just do not do it now, because a volume program has a number in it, and programs with numbers in them end when the number is hit. Q1 gets you fifteen percent, Q2 grinds out the rest, somebody screenshots the graph for the quarterly review, and the work is closed. There is no step after "we reduced it by 30%," because reducing it by 30% was the entire brief. Whether your observability spend is value for money is unanswerable while the telemetry is unusable. You can't defend a bill for data nobody can query, and you can't really attack it either, so the argument settles on price – the only number anybody in the room actually has. Enriched telemetry gets used, and usage is evidence. Most of what produces it is unglamorous work: consistent structure, correlation IDs that survive a hop, log levels that mean the same thing across services. But a team that can name the investigations that resolved faster this quarter, and the correlation that did it, walks into the budget meeting with something to say. Q4: The Change You Were Sure About The framework logs the request. Then the middleware logs it, because the framework's version does not carry the tenant. Then the application logs it a third time with slightly different wording, because by that point nobody trusts the other two. Three records, one event, and no reliable way to say which is authoritative. Every one of those lines was added by somebody trying to improve matters, and each has a different team behind it. That is what Q4 actually is, and why I think of it as the backfire. It is not really the stuff that piles up while nobody is looking; that was the double-shipping back in Q1, where either copy is safe to delete because they are identical. These three records differ from one another, and none of them goes without a conversation. Logging whole request and response bodies for completeness is the same story: you add a great deal of data, and the four fields anybody queries end up inside a blob that nothing has parsed. The same thing happens with a processor from the previous section. Take the k8sattributes block from Q3, change nothing about it, and point it at a different pipeline: YAML service: pipelines: metrics: processors: [k8sattributes] On logs, that was enrichment. On metrics, as soon as the backend treats resource identity as series identity, it is a separate series for every pod – and a fresh set of them after every deploy, because pod names churn. That is how a well-meaning label addition takes out a Prometheus. The config did not change, and neither did the intention behind it. Underneath all three is an assumption that more data is the same thing as more signal, and that if the answer is not in there yet then adding should get you closer. It is the same mistake the volume mandate makes, pointed the other way, and I have watched one team make both inside about two years. The awkward thing is that Q3 and Q4 are not separable at the time, and not only on the chart. From the inside, they are the same act: somebody adds something to a pipeline because they are fairly confident it will help. The engineer putting a pod name on a metric is doing what the engineer putting it on a log did. One of them is right. Review will not catch it either, because the reviewer is working from the same information and the same instinct. You need something that checks whether a question actually got easier to answer. What to Govern Instead Put the four quadrants back together, and the problem shows up in one line. Q1 and Q2 both report as a reduction in volume, so dropping probe traffic and dropping the log lines that explain a failure show up in the quarterly review as the same green arrow. Q3 and Q4 both report as volume up, so the enrichment that made an incident tractable and the label that took out your metrics backend are reported as the same red arrow. A bytes-per-day number cannot separate any of that, but it is the number the entire program is steered by. None of which is an argument against governing telemetry. It grows without limit if nobody is watching, somebody has to own the bill, and a team that has never questioned its telemetry costs is not being principled, is just not looking. The argument is about which variable should be on the dashboard. The goal is to try and measure signal, and it is less work than it sounds. Take the ten questions your team actually asks during an incident. Can I segment this failure by tenant? Can I get from this alert to the trace that caused it? Can I tell which deployment introduced it? Write each one as a literal query, in a file, checked into the repository that holds your collector config, and run them in CI against a replay of real telemetry, once with the proposed change and once without. If any answer moves, the build fails. Not just if it comes back empty: sampling does not empty a result; it quietly changes it. That is the difference between Q3 and Q4 made mechanical. The engineer adding pod name to a metric finds out in the pull request instead of during the next incident. It works in reverse too, which is the part that matters for Q3: adding a question and watching it fail is how you justify an enrichment to somebody whose only other number is bytes per day. And if you would rather start with something off the shelf, the Instrumentation Score is an open specification for grading OTLP against semantic conventions and instrumentation best practice, which is a different cut at the same question. Either way: your observability pipeline is probably the only production system you own with no tests on it, and there is no particular reason for that. Changing the Constraints I want to end somewhere slightly uncomfortable, because I do not think this is really a discipline problem or an education problem. Which quadrants you can operate in is dictated by your observability platform's cost model, not by your engineers. If ingest cost scales linearly with bytes, and retention is tiered so that older data becomes slow or expensive or both, then the economics have already made your architectural decisions. Q3 is priced out of existence. Q2 becomes not just permitted but mandatory, because it is the only lever that moves the number anybody is measured on. Your telemetry strategy is a downstream consequence of a pricing page. Teams under that constraint are not making bad choices. They are making the only choices available, and then rationalizing them as noise reduction, because "we improved our signal-to-noise ratio" is a much better sentence than "we deleted data we may need." The interesting question is what changes when volume stops being the binding constraint. When enriching a log record does not require a budget conversation, the matrix opens up. You can attack Q4 aggressively and invest in Q3, which is the combination that actually improves the ratio. Until then, at minimum, name the quadrant. When somebody proposes a pipeline change, ask which of the four it is. It is a five-second question, and I have not yet seen it fail to change the conversation. More

Designing Safe Agent Permissions: Why Least Privilege Must Exist Outside the Model

By Igboanugo David Ugochukwu

CORE

Refcard #267

Getting Started With DevSecOps

By Akanksha Pathak

CORE

Refcard #291

Code Review Core Practices

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

Document SDK vs Basic PDF Library: What Growing Teams Should Know

When you’re standing up a web app, developers are trying to build required functionality quickly and at a low cost: budgets are still tight, teams are small, and resources are thin. These teams often turn to open-source tools to add PDF viewing functionality, and these libraries work: they give you the basics with simple integration and zero cost. However, whether you’re a scrappy startup or an established organization building new functionality in a platform with a large existing user base, open-source libraries can become a bit of a monkey’s paw: they’ve granted your wish, but the pain comes later. In the case of document processing functionality, this pain comes in the form of integration hell, as you need to add more document capabilities one after another, and the capabilities and dependencies of all the open-source libraries you’ve integrated start to show some cracks. Maintenance grows, user experience suffers, and developers are gently resting their foreheads on the desk. Basic PDF Library vs. Document SDK: What to Choose? The best PDF library for your app depends on how far your document requirements are going to grow, not on how they start. While a basic library renders a document and handles one or two operations well, a document SDK covers the full range a growing application eventually needs: viewing, annotation, editing, redaction, security, and accessibility, from a single license. The table below breaks down the three tiers teams typically move through as document requirements expand. The Three Tiers of PDF and Document Tooling Document tooling generally falls into three tiers, and knowing which one you are in is the first step toward the right decision. Tier Option Best for Limitation1Basic PDF library (open-source, for example, PDF.js, PDFBox, MuPDF)Simple, single-purpose PDF manipulation: view, merge, splitLimited scale, support, and feature breadth. Your team owns maintenance and vulnerability patching.2Point API / cloud document API (for example, Adobe PDF Services, AWS Textract, Azure Document Intelligence, Google Document AI)One specific task like conversion or OCR, fast to prototypeDocuments leave your environment. Per-page costs compound at scale. Adding a second task means fragmented workflows across vendors.3Full document SDK (for example, Apryse)Embedded, scalable document workflows across web, server, and mobileRequires more upfront integration planning than dropping in a single-purpose library. Basic PDF Library (Open-Source) PDF.js, PDFBox, and MuPDF are free, source-available, and fine for a basic viewer. PDF.js is the default free web viewer, built into Firefox, and wins the zero-cost use case outright. MuPDF is a proven rendering engine with decades of use behind it. The limitation shows up once the requirement grows. PDF.js loses fidelity on complex documents, redaction, signatures, and compliance formats like PDF/A and PDF/UA. MuPDF ships as a C-level API with no viewer UI, annotation layer, or forms support, which raises integration cost for anything beyond rendering. All three are single-purpose by design, so a non-trivial workflow means stitching several libraries together and maintaining the glue code between them, with no vendor accountable when something breaks. For a closer look at the tradeoffs between the two models, check out the article open-source vs. proprietary PDF SDKs. Cloud Document API Adobe PDF Services, AWS Textract, Azure Document Intelligence, and Google Document AI get you to a working prototype fast. You call an endpoint, get a converted file or extracted text back, and the vendor manages the scaling behind it. For low or unpredictable volume, pay-as-you-go pricing can make sense. The tradeoff is what happens once you need more than one capability. Each task — conversion, OCR, extraction — tends to live behind a different vendor endpoint, and every one of those endpoints is a place your documents leave your environment before the workflow finishes. Per-page or per-call pricing compounds at production volume, and none of these four hyperscalers offer an air-gapped or offline option if your compliance posture requires it. Full Document SDK A full-document SDK puts extraction, redaction, conversion, and signing behind one engine, instead of several vendors glued together with different conditional code paths. The Apryse PDF SDK runs inside your own environment, whether that is your VPC, on-premises, or fully air-gapped. Document content does not route through a third party to get processed. While Apryse offers a full suite of document-processing capabilities, different tools are licensed as separate add-ons, so you’re not paying for capabilities such as digital signatures or secure redaction unless you actually need them. For example, Docaposte moved its document conversion pipeline to Apryse and saw conversions run 16 times faster than its prior setup. Apryse also runs production document workflows for Dropbox, with more than 700 million users, and Egnyte, across 17,000 businesses. How to Tell When You've Outgrown a Basic PDF Library For developers, it may be time to recognize that your basic PDF library is no longer enough when one or more of these shows up in your backlog: Rendering breaks or slows down on complex or large files your library was not built to handle.Your team is maintaining two or more separate libraries stitched together for one workflow.The roadmap now asks for annotations, redaction, or e-signatures your current library does not support.A compliance requirement shows up, such as SOC 2, ISO 27001, or a data residency rule your current stack cannot meet.Your product needs to render and edit documents consistently across web and mobile, not just one platform.Engineers are spending sprint time patching an open-source dependency instead of building product features. Any one of these on its own might be manageable, but dealing with more usually means the maintenance cost of the current setup has started to exceed the cost of moving to a document SDK. Best PDF Library for Enterprise Apps: What to Evaluate Enterprise-grade performance isn’t just for large organizations. When it’s time to migrate from free libraries to a document SDK, evaluate these criteria to get an enterprise-grade solution: Performance at scale: How does the solution handle concurrency and large, complex files?Feature breadth across the document lifecycle: Does the solution provide viewing, annotation, editing, redaction, and signing from one vendor instead of a different license for each?Security and compliance posture: Look for true content redaction, which permanently removes underlying text and image content rather than masking it visually, plus other document security features such as encryption. On the vendor side, look for independent certifications like SOC 2 and ISO 27001.Support and SLAs: Does the vendor offer a dedicated point of contact for open issues?Deployment control: Can the SDK run on-premises, in your VPC, or fully air-gapped, or does it require routing documents through a vendor's cloud?Licensing model: Does the vendor license cover the full feature set, instead of a separate product and a separate contract for each platform or capability? PDF SDK vs. API: Avoiding Fragmented Workflows An API service solves one task well, but problems can start when the second task arrives. Conversion from Adobe, OCR from AWS Textract, and extraction from Azure Document Intelligence means your application accumulates a different conditional code path for every provider, plus potentially a whole new data residency questionnaire to answer during procurement processes. Check out the article, A Developer’s Guide to Reducing Dependencies, to learn more about vendor consolidation. Apryse consolidates that surface area into a single solution. Office-to-PDF conversion, full-text search across a searchable PDF, redaction, and signing all come from the same engine and the same license, so adding a capability is a configuration change rather than a new vendor integration. That consolidation also keeps document content within your own infrastructure, rather than routing it through several third parties to complete a workflow. Migrating From a Library To an SDK: What It Actually Costs The concern teams raise most often is the cost of moving later, after the app has grown around the library's limitations. That cost is real, but so is the cost of staying on a basic library past the point it fits: slower rendering, an inconsistent user experience, and engineering time spent on patching instead of product work. Let’s look at a real example: Blue Voice built its first version on an open-source React PDF viewer. As the product scaled across police departments, maintaining that PDF functionality started consuming engineering time that the team wanted to spend on its core product instead. After moving to Apryse, according to CTO and co-founder Amit Patankar, "the product felt more polished, our users immediately noticed the difference, and our team could focus on building Blue Voice instead of maintaining a PDF viewer." For a closer look at what the maintenance side of that decision costs over time, read the article The Hidden Costs of Choosing the Wrong PDF Library. If you are ready to compare specific SDKs against your requirements, the Document SDK Buying Guide walks through how to evaluate and buy one. What’s Next for Your Team? Whether you use an open-source document processing library today, or are still planning your project, you can try all Apryse capabilities in a test environment instantly (without needing to talk to sales) by starting your trial. When it’s time to use Apryse in production, contact sales to get licensing that fits your needs. FAQ What is the best PDF library for an enterprise app? The best PDF library for an enterprise app is usually not a basic library at all. Enterprise apps typically need viewing, editing, redaction, and security together, which points toward a full document SDK, like Apryse, rather than a single-purpose library. When do I need a document SDK instead of a basic library? You need a document SDK once your app requires more than one document capability, needs those capabilities to share state, or needs document content to stay inside your own environment for compliance reasons. Apryse offers viewing, editing, redaction, and security together, along with premise-based deployment options. What is the difference between a PDF SDK and a PDF API? A PDF SDK is embedded directly in your application and runs in your own environment. A PDF API is typically a cloud endpoint you call for a single task, which means documents leave your environment, and multiple tasks mean multiple vendor integrations. Is an open-source PDF library good enough for production? An open-source PDF library works well for simple, single-purpose tasks like viewing or merging. It becomes harder to justify once you need broader features, vendor accountability for security patches, or support beyond a community forum. How much does it cost to migrate from a library to an SDK later? The migration cost depends on how much the application has grown around the library's limitations. Teams that wait until rendering issues, maintenance load, or compliance gaps are already affecting users typically face a larger migration than teams that move earlier.

By Isaac Maw

RavenDB Launches Quill to Bring Production AI Agents to Enterprise SQL Systems, No Migration Required

Hadera, Israel, September 8th, 2026, TechnologyWire This article was provided by TechnologyWire and does not represent the editorial content of DZone. RavenDB, a NoSQL document database used by more than 12,000 customers, announced today the launch of its new product, Quill, a context layer for SQL databases that makes them ready for production AI agents without migrating the system of record or architecting a custom AI stack. With AI becoming a board-level mandate, CTOs and VPs of engineering are under pressure to ship AI capabilities fast. But for organizations whose mission-critical data sits in legacy SQL systems, built years before embeddings or agents existed, AI can’t access their data. Modernizing or replacing the systems is expensive, risky, and time-consuming. By the time the system is updated, nobody remembers what the project was supposed to achieve or how it measured ROI. Recently, a Gartner survey of infrastructure and operations leaders found that one in five AI initiatives fail, and only 28% report a positive ROI, which is linked to how well the technology is integrated, governed, and aligned with operational needs, not to the sophistication of the model. As AI becomes the industry standard, organizations have been left without a clear path to deliver until now. "Anyone can stand up an AI demo in an afternoon, but getting that demo into production with data pipelines, semantic search, security, governance, all the plumbing a small proof of concept doesn't need until it has to run at scale, is the hard part," said Oren Eini, founder and CEO of RavenDB. "Quill exists because we'd rather hand teams that plumbing already assembled than watch them rebuild the same project after project. You get access to the live data you need, decide the scope on day one, and change it as you go, instead of building everything from scratch." Quill connects directly to an organization's existing SQL database and adds a context layer on top, making it possible to launch production-ready agents in weeks rather than the 18 to 24 months of a typical in-house build. The source system stays exactly where it is and remains authoritative, and the full AI stack- search, retrieval, and agents that can answer questions- is included. Agents built on Quill support web chat, WhatsApp, Telegram, Slack, and Discord out of the box. "With Quill, the plumbing was already there, so we spent our time building the actual feature," said Hagay Albo, CEO at Albos Technologies and Holdings, an early adopter of Quill. By default, Quill is governed, sitting between the AI and the source system, and it is built on the assumption that the model itself cannot be trusted with unrestricted access, so organizations decide exactly what an agent can and cannot see, independent of the source database's own permissions. In a healthcare setting, for example, an agent can answer a patient's question about an upcoming appointment, while prescription data is never part of the dataset it can query. What is usually a custom security project becomes a configuration choice. Quill is also model-agnostic, so teams can use any AI model, switch providers, or run entirely on their own hardware. Quill is now available for organizations running PostgreSQL, SQL Server, or MySQL, with more databases to be supported in the future, and can be deployed in the cloud or on-premises to meet data-residency or regulatory requirements. To start using Quill today, visit: https://ravendb.net/quill About RavenDB: RavenDB is a hybrid NoSQL document database built for modern application development. Used by more than 12,000 customers across 50 industries, RavenDB helps teams move faster with seamless data management across cloud, on-prem, and edge environments. With full-text search, automatic indexes, and an easy-to-use studio for monitoring and administration, RavenDB is the database developers love and enterprises trust. Learn more at www.ravendb.net

By Technology Wire

Apple-OpenAI Fight Escalates With New MacBook Evidence

Apple is turning a former engineer’s old MacBook into the latest flashpoint in its increasingly bitter fight with OpenAI. Apple told a federal court Monday that evidence recovered from former engineer Chang Liu’s Apple-issued MacBook strengthens its claims that confidential company technology was accessed and used after he joined OpenAI. In a filing supporting its request for expedited discovery, Apple said the defendants produced the laptop only after weeks of delay. An initial forensic review allegedly found that Liu and others at OpenAI “were well aware” of his continued unauthorized access to Apple’s third-party cloud storage providers, according to Bloomberg. Apple also alleges that Liu downloaded a confidential circuit schematic and later used it in his work at OpenAI. The company further claims he used a tool at OpenAI with the same name as an internal Apple engineering application. More seriously, Apple alleges that Liu instructed OpenAI colleague Yu-Ting Peng to destroy evidence after learning he was under investigation and that Peng agreed to comply. Apple’s lawyers wrote that the laptop “shows Apple is not conducting ‘fishing expeditions’ but that its trade secrets are being used and evidence is being destroyed,” according to Bloomberg. The newly cited material is largely sealed or redacted, so the public record does not yet provide a complete picture of what Apple found. OpenAI blames Apple’s security practices OpenAI disputes Apple’s interpretation of Liu’s actions. The company says Liu accessed Apple files to help former colleagues with Apple-related work and argues that the documents he accessed were irrelevant to his responsibilities at OpenAI. OpenAI has also criticized Apple’s handling of departing employees, arguing that poor access controls allowed Liu to retain access after leaving the company. It has published excerpts of Liu’s messages as part of its defense. OpenAI has called the dispute “a mess of Apple’s own making” and warned that a broad injunction could “chill employee mobility,” according to Bloomberg. The competing accounts leave several questions for the court, including whether Liu accessed protected material without authorization, whether it qualified as a trade secret and whether OpenAI acquired or benefited from it. The bigger fight is about hardware The stakes extend beyond one engineer. Apple alleges that more than 400 former Apple employees now work at OpenAI. That employee movement has become more sensitive as OpenAI expands into consumer hardware, including products being developed with former Apple design chief Jony Ive. For Apple, proving that confidential engineering material crossed into OpenAI’s hardware work could strengthen its case for court restrictions. For OpenAI, a broad order could complicate hiring and product development while the company builds its hardware business. For IT and security leaders, the dispute also highlights the importance of immediately revoking departing employees’ access, recovering company devices and preserving records when staff move to competitors. What happens next Apple is seeking damages, expedited discovery, and a preliminary injunction to prevent OpenAI from using the disputed technology while the lawsuit proceeds. It also wants proprietary material allegedly obtained by OpenAI destroyed. A federal judge is scheduled to hear arguments on Apple’s request for a preliminary injunction on Oct. 1. The hearing could determine whether OpenAI faces immediate restrictions while the lawsuit proceeds, although the court has not ruled that any trade-secret misappropriation or evidence destruction occurred. Editor’s note: This article originally appeared on our sister publication, TechRepublic.

By Aminu Abdullahi

Optimize an AI Agent to Sound Human, Judged by an AI Detector

You can tell when an LLM wrote an email. The “I hope this email finds you well” opener, the three polite paragraphs answering a one-line question. I wanted a reply-drafting agent that didn’t do that, and “don’t sound like an AI” turned out to be hard to put in a prompt. Banning a few phrases is easy. The rest is judgment, and a single prompt that holds across a friendly dinner invite and a recruiter cold email took more iterations than I’d guessed. This is not only an email problem. Some platforms down-rank content that reads as AI-generated, so teams publishing at scale have a real stake in prose that clears a detector, even when a human wrote it. The workflow here applies to any of that. So I stopped hand-tuning and let LaunchDarkly agent optimization search for the prompt. You give it a judge that scores “better,” and it generates prompt variations and keeps the ones that beat the bar. For the reasoning behind the feature, read the agent optimization announcement. This tutorial is the how. If you don’t have an account yet, sign up for LaunchDarkly to follow along. Two pieces do the work here. Claude (claude-haiku-4-5-20251001) runs both roles: it drafts the replies, and it writes each new candidate prompt when the loop asks for one. Scoring comes from GPTZero, which isn’t a language model at all but a closed AI detector. I wired it in inverted, so the score is the probability a reply reads as AI, and the optimizer drives it down. I went with a detector instead of an LLM-as-a-judge for a reason: grading one model’s prose by asking another model whether it sounds human is exactly the call language models are unreliable at, and a tool trained for that one question gives a number you can defend. A run is cheap. Each iteration costs around $0.002 and a few seconds, so a full run lands near a penny or two, and the loop tries variations I’d never sit down and type by hand. This Tutorial Runs From a Saved Config You bootstrap the agent, the judge, and the optimization, then work in the UI. Every iteration streams to the optimization Results tab, the winner lands on the agent Variations tab, and you tune thresholds and inputs on the optimization itself. The code does two things: it scores AI-likeness with GPTZero, and it runs the optimization. What You Will Build A reply-drafting agent, email-agent, seeded with one deliberately thin instructionAn inverted AI-likeness judge, ai-likeness, scored by GPTZero in code rather than by a promptA saved optimization, email-agent-opt, that runs as a candidate generator and streams to the Results tabA path from the candidates it surfaces into a fuller offline eval, where the real measurement happens The GPTZero integration is the reusable part. The same shape works for any external scorer you might bring, whether a moderation API, a classifier you host, or a scoring endpoint of your own, so what you learn here isn’t limited to email or to AI detection. The companion repo is agent-optimization-sample. Clone it to follow along. Prerequisites Sign up for LaunchDarkly and request access to agent optimizationPython 3.11+ and uvAn Anthropic API key, which Claude uses to both draft replies and write each new promptA GPTZero API key for the AI-likeness detectorA LaunchDarkly SDK key and a REST API key, in a local .env Install the project and its dependencies: Shell uv sync # .env holds LD_SDK_KEY, LD_API_KEY, LD_PROJECT_KEY, # ANTHROPIC_API_KEY, AI_LIKENESS_API_KEY The repo aliases the short LD_* names to the LAUNCHDARKLY_* names the SDK expects, so the short names in .env are enough. How Agent Optimization Works Agent optimization runs an iterative loop against an AgentControl config. It measures your current variation as a baseline, generates candidate variations, and scores each against your acceptance criteria. The loop has a simple shape: Plain Text ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ Define │────▶│ Explore │────▶│ Commit │ │ "Better" │ │ candidates │ │ a winner │ └──────▲──────┘ └──────────────┘ └──────┬──────┘ │ │ └────────────────────────────────────────┘ Define better. You set acceptance criteria with a judge and a threshold. A judge scores a response on one dimension. You reference a judge saved as an AgentControl config by its key. Explore candidates. Each iteration drafts against your inputs, scores the result, and writes the next candidate from what the scores tell it. The threshold here is a gate that keeps the loop generating. When a candidate clears it, the optimizer re-runs that same prompt against a few more of your input samples and keeps it only if it passes those too, so a prompt that got lucky on one message doesn’t win. Even then, clearing the gate doesn’t certify that a candidate is good enough to ship. A fuller eval decides that, later. Commit the winner. The recommended variation shows up in LaunchDarkly, and with autoCommit it publishes back to the agent’s Variations tab so you can read what the optimizer wrote. You also pick an evaluation mode. Exploratory mode infers quality from the judge alone, which suits open-ended inputs that have no single correct output. Expected Output mode scores against known-correct answers. Replies have no single right answer, so this tutorial stays in Exploratory mode. The Technique Behind It Agent optimization is an instance of OPRO (Optimization by PROmpting), introduced in Google DeepMind’s Large Language Models as Optimizers. A model reads the history of prompts and their scores, then writes the next candidate to try. It searches over prompts, not model weights, so each candidate is cheap to run, and nothing is ever trained. How the Sample Is Organized The companion repo keeps the moving parts in small files, so each piece is easy to find and swap: bootstrap.py: seeds the three LaunchDarkly objects, the email-agent config, the ai-likeness judge, and the email-agent-opt optimization. It’s safe to re-run, and it prints links to the configs and the Results tab.optimize_from_config.py: the one run command. It reads the saved optimization, runs it, streams each iteration to the Results tab, and prints the tab’s link at the end.optimize.py: the two callbacks the run needs. handle_agent_call drafts replies on Claude, and writes the next candidate prompt on Claude too when the SDK asks for one. handle_judge_call scores AI-likeness with GPTZero and hands the optimizer the per-reply detector output.detector.py: the GPTZero client. score_with_response returns the number the judge gates on and the full GPTZero JSON.messages.py: the synthetic input messages.gptzero_test.py: a standalone probe for scoring a draft by hand.clients.py and env.py: the LaunchDarkly and Anthropic clients, built once each, and the .env loader. The saved optimization holds what you’re optimizing for: the judge, the threshold, the inputs, and the model choices. The code holds how the work happens, drafting on Claude and scoring with GPTZero. You edit the what in the UI and the how in code, and the run command brings the two together. Step 1: Bootstrap the Agent, Judge, and Optimization One command seeds everything this tutorial needs. bootstrap.py creates three objects in LaunchDarkly, and it’s idempotent, so anything that already exists is left alone: email-agent: the agent config whose instructions the optimizer tunes. The baseline gives the model the task and the output contract, a JSON {"replies": [...]} envelope, and the {{messages} variable, and nothing about tone. That’s on purpose. It leaves the humanization, the part you want optimized, to the optimizer.ai-likeness: the inverted judge config. GPTZero scores it from code, which leaves the judge prompt as a placeholder.email-agent-opt: the saved optimization the Results tab runs. Thresholds, inputs, and model choices all live here. The baseline is thin on tone but carries the output contract the parser needs, on the Claude model the agent drafts with: Python { "key": "baseline", "name": "Baseline", "model": {"modelName": "claude-haiku-4-5-20251001", "parameters": {}, "instructions": ( "You are an email assistant. Write a reply to each message below. " 'Return ONLY a JSON object {"replies": ["<reply to Message 1>", ...]} ' "with one reply per message, in order.\n\n{{messages}" ), } Run the bootstrap: Shell uv run python bootstrap.py It prints links to the two configs and the one command that runs the optimization. One Judge by Design The New optimization form in the UI attaches a single judge, so this tutorial uses one: AI-likeness. The bootstrap creates the same single-judge optimization in code, so the run matches what the UI supports. To build it by hand instead, open Agent optimization, then New optimization, target email-agent, add the ai-likeness judge, set the threshold and the input messages, and save. Step 2: Define the Input Messages The agent drafts against a fixed set of messages, injected into the prompt through the {{messages} variable. Keep them diverse on purpose, so a winning prompt generalizes instead of overfitting to one kind of message. The agent replies to all of them in one call and the judge averages the AI-likeness scores, and Step 3 explains why that averaging matters. messages.py holds the message list: Python SYNTHETIC_MESSAGES = [ "Hey! Are you free Saturday for dinner? A few of us are getting together and it'd be great to see you.", "Hi, I came across your background and I'm hiring for a senior role that looks like a strong fit. Open to a quick call this week?", "It's been way too long! I'll be in town Thursday and Friday. Any chance you're around to grab coffee?", "Just a reminder that your dentist appointment is Tuesday at 2pm. Reply to confirm or reschedule.", "Hey neighbor, a package addressed to you was left at my door by mistake. Want to swing by this weekend?", # 5 more, spanning tone, intent, and length ] Step 3: Score AI-Likeness With GPTZero AI-likeness is the target, and it’s the one judge that has to live in code, because it isn’t an LLM. You score the reply with GPTZero, an AI detector. The score is 1 - P(human): near 0 when GPTZero reads the reply as human, which is your goal, and near 1 for AI or mixed text. Lower is better, so the bootstrap marks the judge inverted and the optimizer drives the score down. GPTZero is also the more useful integration to learn, because it generalizes to any external scorer. The SDK gives a judge config no hook to reach outside code, so the config exists only so the optimization can attach a judge; its prompt stays a placeholder, and the real number comes from a small client you call yourself: Python def score_with_response(text: str): data = _api_call(text) # POST the reply to GPTZero doc = (data.get("documents") or [{}])[0] return 1.0 - doc["class_probabilities"]["human"], data # 1 - P(human), + full JSON score_with_response returns both the number the judge gates on and the full GPTZero JSON, so the callback can forward the detail to the optimizer. Set AI_LIKENESS_API_KEY to your GPTZero key. Two things here cost me time. GPTZero sits behind Cloudflare, and a plain urllib request comes back as a 403 with error code: 1010 before it reaches the API. That reads like a bad key, but it isn’t. Sending any real User-Agent header clears it. The score also comes from 1 - P(human) rather than the average_generated_prob field, which is the fraction of sentences flagged AI and reports 1.0 even on text the detector still classifies as human. Gating on that field punishes replies that already passed. GPTZero is the only judge, so handle_judge_call doesn’t branch on the judge key. Every call pulls the batch of replies, scores each one with GPTZero, and averages: Python async def handle_judge_call(key, config, context, is_evaluation=True): text = _extract_candidate(context.user_input or "") replies = [r for r in (json.loads(text) or {}).get("replies", []) if isinstance(r, str) and r.strip()] if not replies: # empty draft must FAIL the inverted gate return OptimizationResponse(output=json.dumps( {"score": 1.0, "rationale": "empty or degenerate candidate"})) results = [] for reply in replies: # one GPTZero call per reply in the batch s, raw = detector.score_with_response(reply) # 1 - P(human); + full JSON results.append({"reply": reply, "ai_likeness": s, "gptzero": raw}) avg = round(sum(r["ai_likeness"] for r in results) / len(results), 4) # The gate is the AVERAGE; the full per-reply GPTZero JSON becomes the rationale. return OptimizationResponse(output=json.dumps( {"score": avg, "rationale": f"Average AI-likeness = {avg}. {json.dumps(results)}"})) Two choices in there are what make optimizing against a detector actually work. Score a batch and average. A single short reply’s GPTZero score is noisy. The very same prompt can produce a reply it calls 99% human on one message and 99% AI on the next. Each turn drafts replies to all the messages in one batch, and the judge averages those scores into something steady enough to optimize against. Forward the whole detector response. The rationale you return goes straight to the model that writes the next prompt. Instead of a bare number, return the full GPTZero JSON, with its per-sentence probabilities and predicted class. The optimizer reads that directly and revises around whatever scored as AI, with no parsing on your side. The empty-draft case is the one I got wrong first. An empty reply scores 0.0, which the inverted gate reads as perfectly human, so an early version let the optimizer win by drafting nothing. Now an empty or malformed batch, or a detector error, returns 1.0 and fails the gate. To build intuition before you run the loop, probe GPTZero on a draft by hand: Shell uv run python gptzero_test.py "your draft reply" A Detector Is a Black Box, So Gate Accordingly A detection service gives you a defensible score immediately, with no model to train. It also has real limits. You can’t tune it; it leans toward calling short LLM text AI, and it bills per call across every iteration. That confidence is the reason to average over a batch instead of gating on a single reply. Step 4: Run the Optimization The saved optimization holds the judge, the threshold, the inputs, and the model choices, so the run command takes none of them. The agent drafts on Claude, the optimizer writes each new prompt on Claude too, and the threshold keeps the loop generating rather than picking a winner. Here is the saved optimization: Python { "key": "email-agent-opt", "aiConfigKey": "email-agent", "maxAttempts": 10, "judgeModel": "claude-haiku-4-5-20251001", # required by the API; the GPTZero judge never calls it "modelChoices": ["claude-haiku-4-5-20251001"], # the Claude model the agent drafts with "judges": [{"key": "ai-likeness", "threshold": 0.5}], # generator gate, not a winner test "variableChoices": [ # interpolated into the instructions; the optimizer must use every one {"sender_type": "friend", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK}, {"sender_type": "professional contact", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK}, ], "userInputOptions": ["Draft the replies now."], # trigger turn; messages come from {{messages} "autoCommit": True, } MESSAGES_BLOCK is the message list from Step 2, formatted and fed in through {{messages}. respondent_name and sender_type are the other two variables, so replies come out signed and pitched to the right register. The optimizer has to use every variable you declare, which is what keeps {{messages} and the JSON envelope intact through every rewrite. The threshold is 0.5, and that number came from watching GPTZero, not from theory. A confident detector scores even clearly human-sounding short replies well above 0, so a gate near 0 never trips and the loop never finds anything to keep. At 0.5, this run passed at iteration 6 with 0.43, while the earlier iterations landed between 0.54 and 1.00. That gave the loop room to explore without rubber-stamping every candidate. Run it from the saved config: Shell OPTIMIZATION_KEY=email-agent-opt uv run python optimize_from_config.py The command prints a link to the Results tab. One callback drafts replies on Claude, and the SDK reuses that same callback to write the next prompt, also on Claude. Each iteration posts its prompt and score to the Results tab as a candidate. Settings Live on the Optimization Thresholds, inputs, models, and maxAttempts are baked into email-agent-opt at bootstrap time. Change them by editing the optimization in the UI, or by deleting it and re-running bootstrap.py with the matching environment variables set. Committing the winner back as a variation needs the REST API key. Step 5: Read the Winner Open the Results tab from the printed link. Each iteration posts as it runs, with its candidate prompt and AI-likeness average alongside the variation the run currently recommends. The Results tab after a passing run. Iteration 6 cleared the 0.50 gate at 0.43 and committed optimistic-coyote, with per-iteration charts for AI-likeness, latency, tokens, and cost. Click any iteration to drill into its candidate prompt, the input it ran against, and the replies it produced. Iteration 1 is the baseline template itself, scoring 0.64. Its replies already read decently (“Thanks for the invite! I’d love to come to dinner Saturday.”), but the thin prompt left enough AI signal for the detector to flag. Iteration 1, the baseline template: the JSON-and-{{messages} instruction, the 'Draft the replies now.' trigger input, and the replies it produced. At 0.64 it didn’t clear the gate. The optimization sets autoCommit, so on success the winner publishes back to email-agent as a new variation. Open the agent Variations tab to read what the optimizer wrote. What the Optimizer Changed The run committed a new variation, optimistic-coyote. The Variations tab shows it next to the baseline, so you can read the change directly. The optimizer kept the JSON envelope and the {{sender_type} and {{messages} variables, and built a full humanization spec around them: The Variations tab: the thin baseline above the committed winner optimistic-coyote, both on claude-haiku-4-5-20251001. The optimizer preserved the {{sender_type} and {{messages} variables and the JSON envelope while adding the humanization guidance. Every line of it is a humanization lever, and they map onto how GPTZero separates the two classes: How people actually write. Contractions, first person, sharply varied sentence length, the occasional fragment, and opening with the point instead of a preamble.A stop-list of AI tells. “I hope this email finds you well,” “Thank you for reaching out,” and “Best regards,” plus connectors like “furthermore,” “moreover,” and “consequently.”Specific and signed. Match the sender’s register, reference concrete details from the message, and sign with {{respondent_name} instead of a placeholder. The optimizer took a prompt that said nothing about tone and built out a detailed spec, and GPTZero scored the resulting replies as more human. That’s one of the candidates the loop surfaced. To decide whether it’s worth shipping, take it into a fuller offline eval for real data, which the sections below cover. The UI, the Saved Config, and the SDK You can run all of this from the UI or from code. The New optimization form builds the same optimization by hand and streams to the same Results tab. This tutorial used optimize_from_config, which runs a saved optimization from code while its judge, threshold, inputs, and models stay editable in the UI. To define everything in code instead, optimize_from_options takes the settings directly and accepts more than one judge, and optimize_from_ground_truth_options handles Expected Output mode when you have correct answers to match. This demo leaves several controls unused: token_optimization and latency_optimization for a cost-and-latency pass, token_limit for a spend cap, variation_key, output_key, context_choices, and the on_turn, on_passing_result, on_failing_result, and on_status_update callbacks. Optimization Is Not Evaluation Offline evals and optimization sit next to each other in AgentControl and do opposite jobs. An offline eval measures a configuration you already have: you run your agent over a dataset, score it with judges, and answer “how good is this, and did anything regress?” Optimization runs the other direction, generating new variations until one clears that same judge. Running a surfaced candidate back through an eval is that measuring job again, telling you how it actually performs on a fuller set. They’re strongest together. Here’s a path through AgentControl that gets the most out of both: Define judges for what “better” means, the way you defined human-sounding replies here.Baseline with offline evals over a dataset, so you know where the current agent stands and have a regression check to compare against. For a worked example, read Offline evaluation of RAG-grounded answers.Optimize against that judge to discover a stronger variation, which is what this tutorial walks through.Watch production with online evals and AI Insights, then feed what you learn back into the dataset and the next optimization. When to add online evals covers the tradeoffs. Run it long enough, and production signals become the next round’s eval data, so each optimization starts from what you actually saw in production rather than a guess. Wrap Up You started with a prompt that said nothing about tone and let agent optimization rewrite it against GPTZero, then read the winning humanization spec off the Variations tab. The loop’s job was to explore cheaply, and the trustworthy verdict comes from a proper offline eval over a comprehensive dataset. Agent optimization is one step in a larger workflow. You define judges for what “better” means, optimize against them to surface candidates, then check those candidates with offline and online evals. What you learn in production feeds the next round. If you’re getting started, Build a LangGraph multi-agent system is a good place to begin.

By Scarlett Attensil

From ETL, ELT, and EtLT to Agent: What Is Changing in Enterprise Data Engineering?

For the past two decades, most enterprise data engineering systems have been built on one default assumption: People understand the system. The system executes the pipeline. Engineers understand the business context, break a requirement into steps, write SQL, Spark jobs, shell scripts, synchronization tasks, and scheduling workflows, and then let the system run them. The scheduler does not need to understand the business. The sync engine does not need to understand the metric. It only needs to execute the predefined flow reliably. That model supported the era of data warehouses, data lakes, BI reporting, and batch scheduling very well. But now that assumption is starting to break down. Enterprise data systems are becoming more complex in every direction: More data sources.Longer pipelines.Stronger real-time requirements.Faster business changes.More conflicting metric definitions.More AI application data, model feedback data, vector indexes, and unstructured content. In this environment, enterprises do not just need more pipelines, and they do not just need a better Copilot that can write SQL faster. They increasingly need a Data Engineering Agent that can understand the system, plan tasks, call tools, validate outcomes, and accumulate experience over time. In that shift, Apache SeaTunnel becomes especially important. Because in the agent era, it is not enough for a system to "think." It also has to connect to real data sources, capture changes, execute synchronization, process incremental updates, preserve consistency, and move data to target systems in a reliable and cost-effective way. In other words: The agent understands the goal and plans the action. SeaTunnel turns that action into real, reliable, and recoverable data movement. That is why SeaTunnel is well positioned to become a core execution foundation in the evolution from ETL, ELT, and EtLT to agent-driven data engineering. ETL to ELT: The First Major Shift Traditional ETL is straightforward: Extract data from the source.Transform it in an intermediate layer.Load the processed result into the target system. This model fit the early data warehouse era well. At that time, data sources were relatively limited, the pipeline was easier to understand, and compute resources were more centralized. Enterprises wanted to clean the data, standardize the structure, and define the core logic before loading data into the warehouse. At its core, ETL is a deterministic pipeline model. Its key assumption is: People define the process in advance. The system executes the process. Later, with the rise of cloud warehouses, data lakes, lakehouse architectures, and elastic compute, ELT became more popular. ELT changed the order: ExtractLoadTransform inside the target platform. Instead of transforming everything before loading, enterprises started moving raw or near-raw data into a unified storage layer first, then using the target platform's compute power for downstream modeling and analytics. ELT solved several ETL limitations: It reduced upfront processing complexity.It preserved more original data.It gave analysts and modeling teams more flexibility later. But ELT also created a new problem. If all transformation is delayed until after loading, then dirty source data, schema drift, type mismatches, CDC events, privacy fields, and format inconsistencies all arrive directly in the target system. That might be acceptable in simple batch scenarios. It becomes much more expensive in real-time synchronization, CDC, multi-table sync, lakehouse ingestion, SaaS API ingestion, and AI-oriented data engineering. That is where a third pattern becomes more useful: EtLT. Why EtLT Matters EtLT is not just a compromise between ETL and ELT. A more useful way to understand it is: Extract -> lightweight transform -> Load -> semantic Transform That means: Extract the data.Apply the minimum engineering transformations required to make the data usable.Load it into a unified data foundation.Apply business-level and semantic transformation later. The key idea is the distinction between lowercase t and uppercase T. Lowercase t is not heavy business modeling. It is the engineering work that must happen before data enters the platform safely and consistently, such as: Field projectionType mappingFormat normalizationPrimary key or partition field handlingSensitive field maskingCDC event conversionMulti-table routingSchema evolution handlingPre-ingestion quality validationOne-read, multi-write patternsRate limiting and parallelism control. These transformations should not always be postponed to the target system. Otherwise, the lakehouse or warehouse becomes full of inconsistent, weakly governed, and semantically unclear raw data. At the same time, lowercase t should not try to absorb all business logic. Complex business definitions, KPI semantics, subject-area modeling, and cross-domain aggregation still belong to uppercase T, which should happen in the warehouse, lakehouse, semantic layer, or metric layer. That is the value of EtLT: Standardize the data engineering layer before loading, then apply business semantics after loading. This is exactly the place where SeaTunnel fits naturally. Its Source, Transform, and Sink architecture is well suited for the lowercase t in EtLT. It can connect heterogeneous systems, apply lightweight transformation during movement, handle CDC, adapt schemas, route multiple tables, and write the result into the target platform. In an EtLT architecture, SeaTunnel is not just a data mover. It becomes the data integration runtime that prepares data before it enters the unified data foundation. Why Traditional ETL Starts to Struggle Traditional ETL is built for relatively stable pipelines. You write the rules, draw the DAG, schedule the tasks, and fix failures when they happen. But modern enterprise data environments are no longer that simple. Today a single enterprise may operate across: OLTP databasesKafka streamsCDC pipelinesSaaS APIsObject storageLogs and eventsLakehouse platformsReal-time OLAP systemsVector databasesAI interaction logsModel output datasets. The problem is not only that there is more data. The data is also more fragmented, more heterogeneous, and more real-time. Pipeline length is another issue. A single business metric may depend on dozens of tables, multiple layers of wide tables, several business domains, and a long chain of definition changes. At that point, many enterprises no longer struggle with "Can we build the workflow?" They struggle with "Can anyone still explain the whole pipeline end to end?" This is where traditional ETL shows a structural limitation. One renamed field can break hundreds of tasks.One changed enum can silently shift multiple core metrics.One incorrect incremental logic branch can pollute an entire downstream analysis chain. The scheduler can tell you that a task failed. It usually cannot tell you why that task matters. The sync tool can move the data. It usually cannot tell you which business metric is now at risk. The engineer can fix the script. But only if that engineer can first rebuild the missing context. So the real weakness of traditional ETL is not just performance or reliability. It is that: It can execute the process, but it does not understand the system. Why Copilot is Not Enough Many teams first bring AI into data engineering through Copilot-style workflows: Generate SQLComplete Spark codeDraft YAMLProduce test samples. These capabilities are useful. They improve local productivity. But they do not solve the deepest problem in enterprise data engineering. Because the hardest part of data engineering is rarely just code generation. It is system understanding. Copilot can help generate a SQL statement, but it does not know the real business meaning of the field. It can help draft a synchronization task, but it does not know which downstream metrics will be affected by a schema change. It can help generate a scheduler config, but it does not know whether the change breaks historical consistency or recovery semantics. What enterprises actually struggle with includes: Lineage reasoningDependency analysisSemantic understandingMetric governanceRisk estimationImpact analysisIncremental recovery. These are not just autocomplete problems. So enterprises do not only need an AI IDE. They increasingly need an agentic data engineering system that can understand the target, decompose tasks, call engineering tools, and verify the result. The Real Shift: From Pipeline to Agent If we keep only one conclusion, it is this: Traditional ETL is "people define the process, systems execute the process." Agentic data engineering is "people define the goal, systems generate the process." That is not a slogan. It is a change in how work is organized. In the traditional model, engineers design the task chain first, configure Source, Transform, and Sink, and then let the scheduler execute the pipeline. The system faces a fixed process. In the agent model, the input may only be a business goal. For example: Add a new gross margin metric for orders and keep it aligned with the finance definition. Traditionally, the engineer must: Identify relevant data sources.Read table schemas.Inspect lineage.Design transformation logic.Configure sync and scheduling jobs.Add quality checks.Run regression validation. In an agent-oriented workflow, the system should be able to generate a sequence of actions around the goal: Identify the affected business entities.Discover candidate data sources.Analyze upstream lineage.Decide whether the job belongs to ETL, ELT, or EtLT.Generate or update the SeaTunnel synchronization task.Configure full-load or CDC mode.Apply lightweight transformation.Write the result into the warehouse or lakehouse.Trigger data quality validation.Evaluate downstream impact.Present the result for human confirmation. That is the real difference. The breakthrough is not "AI wrote a SQL statement for me." The breakthrough is: The system starts generating engineering actions from a business goal. But this immediately raises a critical question: When the agent plans a data action, who executes it reliably? That is exactly where SeaTunnel becomes essential. Apache SeaTunnel in the Agent Era: The Data Integration Execution Layer An agent cannot stop at reasoning and recommendations. If a Data Engineering Agent decides that a table should be synchronized, a CDC job should be adjusted, a broken pipeline segment should be replayed, or a data slice should be reloaded into the target system, it needs a stable and observable execution layer to carry out that decision. That execution layer needs several core capabilities. 1. It Must Connect to Many Kinds of Data Sources Enterprise data systems are inherently heterogeneous. An agent cannot live in a world with only one database or one file system. It needs to connect to MySQL, Oracle, PostgreSQL, SQL Server, Kafka, Hive, Iceberg, Doris, ClickHouse, StarRocks, Elasticsearch, S3, HDFS, MongoDB, and many other systems. SeaTunnel's connector architecture is designed for exactly this kind of environment. It abstracts Source, Transform, and Sink through a consistent plugin model so heterogeneous systems can be integrated in a unified way. 2. It Must Support Batch, Streaming, CDC, and Large-Scale Synchronization The agent era does not run on a single data movement pattern. It needs: One-time full migrationContinuous CDCOffline batch movementReal-time synchronizationSingle-table syncMulti-table or database-level sync. SeaTunnel is valuable here because it is not just a script wrapper for ETL. It is a real data integration runtime that can support full load, incremental sync, real-time processing, CDC, and multi-table movement in the same ecosystem. 3. It Must Handle the Lowercase t in EtLT Agentic systems do not need every business transformation to happen inside the sync layer. But they do need the sync layer to complete the minimum engineering transformation required to make the data trustworthy and usable before it lands in the platform. SeaTunnel's Transform layer is a strong fit for: Field mappingType conversionFilteringColumn projectionData maskingRoutingSimple reshaping. That is exactly the role of the lowercase t in EtLT: Do not overload the movement layer with heavy business modeling, but make the data governable and ready for the next stage. 4. It Must Provide Consistency, Fault Tolerance, and Recovery An agent can decide that a broken link should be replayed. But replay only matters if the underlying system can recover correctly. The execution layer still needs checkpointing, failure recovery, state handling, restart behavior, and strong delivery guarantees where needed. A reasoning layer without a reliable execution layer becomes a planner without hands. That is why execution quality still matters as much as intelligence. What the Future Stack Starts to Look Like If we look one step ahead, enterprise data engineering increasingly resembles a layered operating system rather than a collection of disconnected pipelines. In that stack: The semantic layer defines the business model.Metadata provides structure and context.Memory accumulates operational experience.The planning layer turns goals into actions.The execution layer performs synchronization, CDC, movement, replay, and recovery. SeaTunnel belongs to this execution layer. That placement is important. The future is not "put a large language model on top of ETL." The future is a coordinated system where reasoning and execution are separated clearly: The agent decides what should happen.SeaTunnel ensures that it actually happens in a reliable way. The Evolution in One Sentence ETL built data pipelines. ELT moved raw data into a unified platform first. EtLT rebalanced pre-load engineering standardization and post-load semantic modeling. The agent era pushes data engineering one step further: From fixed pipelines to goal-driven systems. In that world, SeaTunnel is not just a synchronization tool. It becomes a practical execution foundation for agentic data engineering. Agents make data systems understand goals. EtLT makes ingestion more controllable. SeaTunnel turns those goals into reliable data engineering actions. That is the deeper change now happening across enterprise data engineering.

By David Zollo

Why AI Hallucinations Are a Quality Engineering Problem

The first time most teams encounter a hallucination in production, they treat it like a bug. They investigate the specific output, trace it back to a prompt or a context window, adjust something, and move on. What they do not do is ask the more important question: how many other outputs like this have already reached users without anyone noticing? That question is uncomfortable because the honest answer, for most enterprise AI deployments, is that nobody knows. Traditional quality engineering was built to answer a different question. Did the system do what we expected it to do? For deterministic software, that question has a clean answer. The same input always produces the same output. You write a test case, define the expected output, run the test, and the result is either pass or fail. AI systems do not work that way. A language model predicts the most probable next output given its training data and context. That prediction can be fluent, confident, and completely fabricated. Unfortunately, no traditional test case will catch it, because no traditional test case was written to verify whether generated content is actually true. This is the gap that quality engineering needs to close. Not by waiting for AI vendors to solve the hallucination problem (they are working on it, but they will not eliminate it), but by building the testing infrastructure to detect, classify, and govern hallucinations as the production defect category they already are. The Defect That Doesn't Look Like One When a traditional system fails, something breaks. An exception gets thrown. A test turns red. A pipeline fails. The failure is visible. When an AI system hallucinates, nothing breaks. The system runs normally. The response comes back with full confidence. The test passes. And somewhere in that output is something factually wrong, fabricated, or internally inconsistent, delivered as if it were completely reliable. From a quality engineering perspective, that's the worst kind of defect. It looks exactly like a pass. The word "hallucination" doesn't help either. It makes the problem sound rare and strange, like something that only happens in edge cases or poorly built models. It isn't. It's a predictable failure mode of probabilistic systems, and it happens regularly in production deployments that look completely healthy by every traditional quality metric. Why Your Existing Test Suite Won't Catch It Traditional QA was built for deterministic systems. Same input, same output, every time. You write a test case. You define the expected output. You run the test. It passes, or it fails. AI systems don't work that way. A language model doesn't retrieve answers from a verified database. It predicts what comes next based on patterns in training data. When that prediction produces something plausible-sounding but factually wrong, no existing test catches it, because no existing test was written to check whether generated content is actually true. Think about what that means in practice. A customer-facing AI assistant that confidently cites a return policy that doesn't exist. A code generation tool that produces syntactically valid but logically broken functions. A contract summarization system that omits a material clause because the model weighted other content as more statistically relevant. Each of these is a hallucination. None of them would be caught by a standard test suite. All of them have already happened in production environments. Treating Hallucinations Like the Defects They Are The first thing quality engineering needs to do is stop treating hallucinations as an AI problem and start treating them as a defect category, with a taxonomy, a detection strategy, and a severity classification. Not all hallucinations carry the same risk. That distinction matters because it determines how you test for them and what threshold you'll accept. Factual hallucinations are the most straightforward. The model generates content that contradicts verifiable facts. These are the most detectable because you can validate them against a known correct answer. Contextual hallucinations are trickier. The output is plausible, but it's inconsistent with the specific context provided. A document summarization that introduces information not present in the source. A question-answering system that answers a slightly different question than the one asked. The output sounds right. It just isn't right for this situation. Confident hallucinations are the most dangerous. The model assigns high confidence to a wrong answer. Your confidence score monitoring won't surface these because the confidence signal itself is broken. Three Things to Add to Your Testing Strategy Now You don't need to replace your existing test infrastructure. You need to extend it. 1. Ground Truth Validation For any AI system working with factual content, such as policy documents, product catalogs, regulatory filings, and technical specs, build test cases with a verifiable correct answer. Don't just check that the model produced a response. Check whether the response matches what's actually true. Python def test_policy_response(model, query, ground_truth): response = model.generate(query) similarity = semantic_similarity(response, ground_truth) assert similarity >= FACTUAL_ACCURACY_THRESHOLD, \ f"Response diverges from ground truth: {similarity:.2%} similarity\n" \ f"Query: {query}\n" \ f"Response: {response}\n" \ f"Expected: {ground_truth}" log_hallucination_check( query=query, response=response, ground_truth=ground_truth, similarity=similarity, passed=(similarity >= FACTUAL_ACCURACY_THRESHOLD) ) 2. Consistency Testing Hallucinating models are often inconsistent. Ask the same question in different ways and compare the answers. A model that gives meaningfully different factual answers to semantically equivalent questions is telling you something important about how reliable its knowledge is in that domain. Python def test_response_consistency(model, query_variants): responses = [model.generate(q) for q in query_variants] similarity_scores = [] for i in range(len(responses)): for j in range(i + 1, len(responses)): score = semantic_similarity(responses[i], responses[j]) similarity_scores.append(score) mean_consistency = sum(similarity_scores) / len(similarity_scores) assert mean_consistency >= CONSISTENCY_THRESHOLD, \ f"Inconsistent responses detected: {mean_consistency:.2%} mean similarity" 3. Adversarial Probing Design test cases specifically intended to elicit hallucinations. Ask about events that didn't happen. Request citations that don't exist. Query edge cases well outside the model's reliable knowledge domain. A model that says "I don't know" or expresses appropriate uncertainty is behaving correctly. A model that fabricates a confident answer has failed. That failure is worth knowing about before your customers find it. The Real-World Example That Makes This Concrete A retail enterprise I worked with deployed an AI model for inventory replenishment decisions. Their test suite showed a 96% pass rate. The team was confident in the release. After adding confidence-aware testing and ground truth validation, the picture changed. The model was consistently producing replenishment recommendations for seasonal products with confidence scores between 55% and 65%, a category that represented a significant portion of inventory value. Binary testing had masked this entirely because the model's outputs happened to align with expected values in the test data, even though the model was operating with low certainty. The binary tests said green. The confidence scores said something was wrong. The confidence scores were right. This Is a Governance Problem, Too Quality engineering teams that haven't built explicit hallucination detection into their validation frameworks are creating governance gaps, whether they know it or not. As AI systems take on more consequential work, such as drafting legal documents, summarizing financial filings, generating clinical notes, advising on compliance, the organizational liability for hallucinated outputs grows. Regulators are starting to ask questions about AI explainability and documented uncertainty bounds. A binary pass/fail result doesn't answer those questions. A hallucination risk assessment does. Every enterprise AI deployment should be able to answer three questions: What categories of hallucination are possible in this system? What's the acceptable threshold for each category? What happens when one is detected in production? If your team can't answer those questions for the AI systems currently live, that's where to start. Conclusion Hallucinations aren't going away. They're a predictable consequence of how probabilistic AI systems work, and no amount of prompt engineering or model improvement will eliminate them entirely. What can change is whether your quality engineering team is equipped to find them before customers do. That means a defect taxonomy. A detection strategy. Ground truth validation. Consistency testing. Adversarial probing. Production monitoring that flags suspicious outputs for human review. The teams that get enterprise AI quality right won't be the ones waiting for hallucinations to surface in incident reports. They'll be the ones who decided hallucinations were a QE problem, and built the testing infrastructure to treat them like one.

By Rajeshkumar Rajaseakaran Nair

What Actually Makes AI Infrastructure Agents More Reliable (It's Not More Agents)

I keep seeing the same pattern. Someone builds an "AI agent" for infrastructure monitoring — it answers questions about Prometheus metrics, pulls logs from ELK, suggests restarts. Impressive in a demo. Then you push on it: what happens when its logs query times out mid-investigation? What happens when the context window fills up while correlating signals across four systems? What happens when a tool call hallucinates a metric name that doesn't quite exist? Usually it doesn't fail catastrophically. It fails quietly, in ways that are hard to debug. And quiet failures during incident response are the worst kind. I've spent the last year prototyping multi-agent architectures for infrastructure observability — coming at it from a decade of SRE and network reliability work. My working hypothesis going in was straightforward: split the investigation across specialized agents, and the reliability problems that plague a single agent — a blown context window, a hallucinated tool call — should ease. Below is the architecture pattern I built to test that, the failure modes I watched it run into, and — since I've since put the hypothesis through a more rigorous test than a demo — what actually held up. Why a Single Agent Hits a Ceiling A good on-call engineer doesn't open one dashboard and stare. They move between tools, check Prometheus, scan logs, look at deploy history, consult runbooks. Each step informs the next. Investigations have structure. A single LLM agent trying to replicate that workflow runs into two real constraints. First, the context window. Every tool call, every metric result, every log snippet goes into that window. Short investigation: fine. Anything complex — multiple services, ambiguous signals, a failure mode the model hasn't seen — and the window fills. Early observations get pushed out. The model loses the thread. Second, the tool problem. The more tools you give a single agent, the more likely it is to hallucinate one — invoking a function that doesn't exist, or constructing a query with the right name but the wrong parameter. I've reproduced this in my own prototyping: the agent confidently calls a metric query with a typo, gets an empty result, and concludes the metric doesn't exist rather than that the query was wrong. Split the work across specialized agents, and both pressures ease. Each agent owns a smaller tool set it actually knows. Each agent has a manageable context. And when one agent fails — they will, eventually — it fails in a bounded, debuggable way. The Architecture: Four Roles, One Investigation The pattern I keep returning to has four agent roles, communicating through shared state rather than direct message-passing. The state is a typed Python object accumulating findings as the investigation progresses. No agent starts from scratch; each picks up where the last one left off. Python class InvestigationState(TypedDict): incident_id: str trigger: AlertTrigger telemetry_findings: list[Finding] causal_hypothesis: Optional[Hypothesis] recommended_actions: list[Action] confidence_score: float audit_trail: list[AgentStep] Every agent reads from and writes back to this state. That single design choice — shared structured state instead of free-form message-passing — is what makes the system auditable. The four roles: Supervisor Agent receives the raw alert. It doesn't investigate — it routes. It classifies the incident, identifies the services involved, and decides which specialist agents to invoke. Telemetry Investigation Agent is the data-gathering specialist. Given an investigation context, it runs queries against the observability stack — Prometheus, Grafana, ELK, AppDynamics — finds anomalies, and returns structured findings. It doesn't explain what it finds. It just finds things. Python def route_to_specialists(state: InvestigationState) -> list[str]: trigger_type = state["trigger"].classification if trigger_type == "network": return ["telemetry", "reasoning"] elif trigger_type == "application": return ["telemetry", "logs", "reasoning"] else: return ["telemetry", "logs", "infra", "reasoning"] Reasoning Agent takes the Telemetry Agent's findings and tries to answer: what is actually going on? Given a RAG index over historical post-mortems, it can reason like "this pattern resembles a connection-pool exhaustion failure mode I've seen documented before." When it works, the experience is impressive. When it's wrong, it's confidently and eloquently wrong — a class of failure I'll come back to. Action Agent turns a hypothesis into something executable. For low-risk actions, it could, in principle, act autonomously once confidence crosses a threshold. For anything riskier, it drafts a recommendation with full context and routes to a human for approval. I'd treat that human-in-the-loop gate as non-negotiable for any first deployment. A Worked Example (Prototype, Not Production) Python # Investigation trace — synthetic test environment # Alert received: payments-service p99 latency anomaly # Supervisor → routing to Telemetry Agent # Telemetry: db-proxy connection pool utilization elevated # Telemetry: db-proxy deployed recently (within last 20 min) # Telemetry: no downstream dependency anomalies # Reasoning: hypothesis — connection-pool regression in recent deploy # Action: recommend rollback to prior db-proxy version # Action: draft escalation with evidence → human approval required The point of the multi-agent system isn't to replace the engineer. It's to do the legwork before the human even opens their laptop, so the human is reviewing evidence rather than gathering it. Where This Pattern Breaks Three failure modes worth naming: Confidence scores are not well-calibrated. An 87% confidence score sounds authoritative. Language models don't express uncertainty the way a careful engineer would. Any deployment needs a conservative threshold for autonomous action and a generous fallback to human review. Context grows faster than you expect. Five services, 15 tool calls of data in shared state, and the Reasoning Agent starts dropping things. State summarization helps, but it's lossy. Agent observability is its own problem. You're building a system that monitors infrastructure, and now you need to monitor the monitor. Without per-step tracing, debugging an agent failure is genuinely painful. What a Rigorous Test Actually Showed Everything above is prototype-stage reasoning — the kind you form watching a system work and fail in front of you. I didn't want to leave it there, so I ran the architecture against two real fault-injection benchmarks, AIOps Challenge 2020 and RCAEval, 75 incidents each, across six pre-registered configurations comparing the four-role design against a well-built single agent given the same tools and the same context budget. Decomposition alone didn't win. Across both benchmarks, the multi-agent version came out statistically indistinguishable from the single agent (McNemar's test, p > 0.05 in every configuration), and a plain rule-based baseline stayed competitive with both. That's not the result I expected going in, and it's worth sitting with rather than explaining away: splitting an investigation into roles does not, by itself, make it more accurate. What did move the needle was a narrower idea: a Falsifier agent that checks the Reasoning agent's hypothesis against evidence it wasn't shown, instead of taking the hypothesis at its word. That improved accuracy on single-service incidents meaningfully — 33.3% vs. 21.3%, p = 0.023 — and made multi-service incidents worse at first — 24.0% vs. 42.7%, p < 0.001 — because with no notion of which services depend on which, the falsifier mistook a downstream symptom for the root cause. Giving it real service-topology data closed that gap. Then the less comfortable check: I gave the same falsifier to a single agent instead of the four-role pipeline. It scored indistinguishably from the multi-agent version (p = 1.0). The gain wasn't coming from decomposition. It was coming from the verification step, and the verification step doesn't care how many agents are asking the question. This work is accepted at CNSM 2026 (IFIP); the full benchmark, raw results, and eval scripts are in the repo linked below, and the falsifier design specifically is written up in more depth in the preprint linked at the end of this article. Implementation Notes LangGraph fits this pattern well. The explicit graph model lets you define exactly what happens after each agent step. The graph is code — versionable and testable. For tool management, typed schemas validated before execution eliminate most hallucinated tool calls. The discipline is the same regardless of framework: every tool input is typed, every tool call is validated, nothing executes on an unstructured string. If I had to give one piece of advice: invest in your tools before you invest in your prompts. The ceiling on what an agent can do is set by the quality of the tool interface, not the eloquence of the system prompt. So, Why Multi-Agent? Because single agents fail in ways that are hard to predict and hard to debug, and because bounded roles with structured shared state make an investigation's failures easier to trace, whatever the accuracy numbers say. But I'd stop short of the clean version of this pitch. My own testing didn't support "multi-agent is more reliable" as a general claim — it supported something narrower: a verification step that checks a hypothesis against evidence it hasn't seen is what earns its complexity, and you can bolt that onto a single agent just as well as onto four. The four-role design is still a reasonable way to build one of these systems — the shared-state pattern, the tiered autonomy, the human gate on risky actions are all still doing real work. Just don't assume the agent count is what's buying you the reliability. Test that part before you ship it. The working prototype for this architecture is available at: github.com/Kinjal-Oza/multi-agent-observability-demo Originally published on Medium.

By Kinjal Vaishnav

Databricks Lakebase: Give Your Agent a Branch, Not Your Production Database

There's an awkward moment in every agentic data project. The agent works. It writes decent SQL, it reasons about schema, it proposes a migration that looks right. And then somebody asks the question nobody wants to answer: what happens when it's wrong against production? The usual answers are all bad. Point it at production with read-only credentials, and you've capped the damage but also capped the agent — it can diagnose and never fix. Point it at a seeded staging database, and you get an agent that's confidently correct about data that stopped resembling production four months ago. Spin up a container per run, and you're maintaining fixture pipelines forever while paying for compute that sits idle between runs. What you actually want is a full copy of production that appears in under a second, costs almost nothing while it exists, and vanishes without a trace when the agent is done. Which is to say: you want git checkout -b for your database. Databricks Lakebase does this, and the mechanism is interesting enough to be worth understanding rather than just enabling. How Lakebase Gets There Lakebase is Postgres — genuinely Postgres, not a wire-compatible reimplementation. What Databricks changed is the part underneath. Image credit: Databricks Standard Postgres is a monolith with compute and storage tightly coupled. Databricks decoupled the two and moved storage onto lake storage, which creates an immediate problem: object storage has high latency and no transactional consistency, and Postgres assumes neither. Two components patch the gap. Safekeepers, built on the Paxos consensus algorithm, handle low-latency durable writes. Page servers handle low-latency reads, materializing pages on demand from the lake. Once storage is decoupled and page-addressed, branching becomes almost free. It's copy-on-write: the data stays in one place on the lake, and Lakebase tracks only the deltas between branches. Writing to a branch records the change independently without touching the others. The numbers that matter for agent work: a branch takes roughly 500 milliseconds, a new instance comes up in under 500 milliseconds, rollback to an earlier snapshot is equally fast, and idle branches scale to zero so you pay only cheap lake storage while they sit around. Databricks reports 12 million database launches per day in production, which is less a benchmark than an indication that ephemeral instances are the intended usage pattern rather than an edge case. That last property is what makes this viable for agents specifically. As Ali Ghodsi put it on the summit keynote stage, agents "don't want to wait 10 minutes on a database to come up." An agent that has to wait ten minutes for an environment will simply be given production instead, because the alternative makes the loop unusable. Sub-second provisioning is what makes the safe option also the convenient one. Isolation Strategies, Compared ApproachProvisioning timeData realismIdle costBlast radius of a bad writeOngoing maintenanceRead-only prod credentialsNonePerfectNoneZero — but agent can't actNoneFull prod write accessNonePerfectNoneCatastrophicNoneSeeded staging databaseMinutes to hoursDegrades continuouslyFull instance costContained, but shared between runsHigh — fixture pipelines, refresh jobsContainer per run with dump restoreMinutes, scales with data sizeGood at restore timeNone between runsContained per runMedium — dump/restore toolingLakebase branch per run~500 msIdentical to parent at branch timeNear zero (storage deltas only)Contained per run, discardableLow The column that usually decides it is provisioning time, because it governs whether developers actually use the isolated path or route around it under deadline pressure. Building the Loop Here's the pattern end to end. The Databricks-specific surface is deliberately isolated into one thin layer; everything else is ordinary Postgres, because that's what it is. Step one — get an instance. Lakebase is now included in Databricks Free Edition alongside Genie Code, serverless GPUs, Agent Bricks, and LakeFlow Designer, so this costs nothing to follow along with. Python # lakebase_control.py — the only Databricks-specific layer in this design. # Endpoint shapes for branch operations are still settling; check current # docs before shipping. Everything downstream of get_dsn() is plain Postgres. import os import time import requests WORKSPACE = os.environ["DATABRICKS_HOST"].rstrip("/") TOKEN = os.environ["DATABRICKS_TOKEN"] HEADERS = {"Authorization": f"Bearer {TOKEN}"} def create_branch(parent: str, name: str) -> dict: """Branch an existing Lakebase instance. Copy-on-write; returns in ~500ms.""" started = time.perf_counter() resp = requests.post( f"{WORKSPACE}/api/2.0/database/instances/{parent}/branches", headers=HEADERS, json={"name": name}, timeout=30, ) resp.raise_for_status() elapsed_ms = (time.perf_counter() - started) * 1000 print(f"branch {name!r} ready in {elapsed_ms:.0f} ms") return resp.json() def delete_branch(parent: str, name: str) -> None: requests.delete( f"{WORKSPACE}/api/2.0/database/instances/{parent}/branches/{name}", headers=HEADERS, timeout=30, ).raise_for_status() def get_dsn(instance: str, branch: str) -> str: """Standard Postgres connection string for a branch.""" resp = requests.get( f"{WORKSPACE}/api/2.0/database/instances/{instance}/branches/{branch}", headers=HEADERS, timeout=30, ) resp.raise_for_status() host = resp.json()["read_write_dns"] Step two — wrap it so the agent physically cannot see production. This is the important part, and it's boring on purpose. The agent never receives a DSN; it receives a session bound to a branch. Python # agent_sandbox.py import contextlib import psycopg from lakebase_control import create_branch, delete_branch, get_dsn PROD_INSTANCE = "orders-prod" @contextlib.contextmanager def branched_session(run_id: str, keep_on_failure: bool = True): """ Yield a Postgres connection on a throwaway branch of production. On success the branch is discarded. On failure it is retained so you can attach a psql session and see exactly what the agent did — which is the single most useful debugging affordance in this whole design. """ branch = f"agent-{run_id}" create_branch(PROD_INSTANCE, branch) conn = None try: conn = psycopg.connect(get_dsn(PROD_INSTANCE, branch), autocommit=False) yield conn conn.commit() delete_branch(PROD_INSTANCE, branch) except Exception: if conn: conn.rollback() if not keep_on_failure: delete_branch(PROD_INSTANCE, branch) else: print(f"retained branch {branch!r} for inspection") raise finally: if conn: conn.close() Step three — verify before you believe anything. An agent reporting success is not evidence. Snapshot the invariants you care about before and after, and diff them. Python # verify.py from dataclasses import dataclass INVARIANTS = { "order_count": "SELECT count(*) FROM orders", "open_order_total": "SELECT coalesce(sum(total_cents), 0) FROM orders WHERE status = 'open'", "orphaned_items": "SELECT count(*) FROM order_items i " "LEFT JOIN orders o ON o.id = i.order_id WHERE o.id IS NULL", "negative_totals": "SELECT count(*) FROM orders WHERE total_cents < 0", "duplicate_skus": "SELECT count(*) FROM (" " SELECT sku FROM products GROUP BY sku HAVING count(*) > 1" ") d", } @dataclass class Snapshot: values: dict @classmethod def take(cls, conn) -> "Snapshot": out = {} with conn.cursor() as cur: for name, sql in INVARIANTS.items(): cur.execute(sql) out[name] = cur.fetchone()[0] return cls(out) def diff(self, other: "Snapshot") -> dict: return { k: (self.values[k], other.values[k]) for k in self.values if self.values[k] != other.values[k] } def assert_no_corruption(before: Snapshot, after: Snapshot) -> None: """Hard invariants must not move regardless of what the task was.""" for guard in ("orphaned_items", "negative_totals", "duplicate_skus"): if after.values[guard] != before.values[guard]: raise AssertionError( f"invariant {guard} violated: " f"{before.values[guard]} -> {after.values[guard]}" ) Step four — the run itself. Python # run_agent_task.py import uuid from agent_sandbox import branched_session from verify import Snapshot, assert_no_corruption def run(task: str, agent) -> dict: run_id = uuid.uuid4().hex[:8] with branched_session(run_id) as conn: before = Snapshot.take(conn) # The agent gets a live connection to real-shaped data and full write # access — to a branch that exists only for this run. agent.execute(task, conn) after = Snapshot.take(conn) assert_no_corruption(before, after) return { "run_id": run_id, "changed": after.diff(before), } The Full Cycle This isn't a pattern I invented for the article — it's the shape Databricks uses internally for Genie ZeroOps, its autonomous pipeline-remediation agent. ZeroOps investigates a failure, drafts a fix, then creates shallow clones of production data using the same branching mechanism as Lakebase, deploys the proposed fix to the clone, verifies row counts, and presents the result as a pull request. Nothing reaches production without explicit human approval. The verification-on-a-clone step is what makes autonomous remediation defensible rather than reckless, and it's available to you as a primitive. What Branching Does Not Solve Worth being direct about, because the isolation is narrower than it feels. Side effects escape the branch. Your database is isolated. The payment API your agent calls is not. Neither is the email it sends, the Slack message it posts, or the Kafka topic it publishes to. A branch protects state you own in Postgres and nothing else — every external integration needs its own sandboxing, and this is where teams get burned after assuming the branch covered them. Branch sprawl is real. Sub-second creation plus retain-on-failure produces a lot of branches. Enforce TTLs from day one, tag branches with the run that created them, and put a reaper on a schedule. Copy-on-write storage is cheap, not free, and a branch that diverges heavily from its parent stops being cheap. Fresh at branch time, stale immediately after. A long-running agent task is reasoning about a snapshot. If it takes twenty minutes and the underlying question was time-sensitive, the conclusion may be wrong by the time it lands — not because the isolation failed but because you asked a real-time question of a point-in-time copy. Invariant checks only catch what you thought to write down. The assert_no_corruption function above is a floor, not a ceiling. It catches structural damage. It won't catch an agent that correctly updated the wrong customer's record, because that's semantically valid and structurally clean. Human review of the diff remains load-bearing. The Broader Shift The reason this matters beyond Databricks is that it closes a gap that has quietly constrained every agentic data project: agents were either safe or useful, and picking both required infrastructure most teams weren't going to build. Application code got this right decades ago. Nobody tests a refactor by editing production source — you branch, you break things freely, you merge what works. Databases never got that affordance because copying one was expensive enough that the ergonomics never arrived. Decoupling compute from storage is what finally made the copy cheap, and cheap copies are what make experimentation safe. Give the agent a branch. Let it break things. Read the diff. Lakebase branch APIs are evolving; the control-plane calls above are structurally accurate, but verify endpoint and field names against current Databricks documentation before running in anger. Everything downstream of the DSN is standard Postgres and will behave exactly as you expect. References "Lakebase — Serverless Postgres on the Lakehouse." Databricks — https://www.databricks.com/product/lakebase"Introducing Genie ZeroOps." Databricks Blog — https://www.databricks.com/blog/introducing-genie-zeroopsMarattha, P. "Databricks Data + AI Summit 2026 recap: Genie One, LTAP, Lakehouse//RT and every major launch." Flexera Blog, June 30, 2026 — https://www.flexera.com/blog/perspectives/databricks-data-ai-summit-2026/"Databricks Launches LTAP, the First Lake Transactional/Analytical Processing Architecture." Databricks Newsroom — https://www.databricks.com/company/newsroom/press-releases/databricks-launches-ltap-first-lake-transactionalanalytical"Databricks Sandbox — Serverless Compute Documentation." Databricks Docs — https://docs.databricks.com/aws/en/compute/serverless/sandbox"Introducing Genie One, Genie Ontology, and Genie Agents." Databricks Blog — https://www.databricks.com/blog/introducing-genie-one-genie-ontology-and-genie-agents

By Jubin Soni, FBCS

CORE

Architecting Trust: Agentic Microservice Testing Strategies in the Era of Non-Deterministic AI

The enterprise software landscape is undergoing a foundational architectural shift that rivals the original transition from monolithic applications to distributed systems. For the past decade, the microservice architecture has successfully allowed engineering teams to manage application complexity through the strict decomposition of business domains into independently deployable, scalable units communicating over well-defined application programming interfaces (APIs). However, the aggressive integration of large language models (LLMs) into the application execution layer has catalyzed an entirely new paradigm: the agentic microservice architecture. In this advanced model, the core tenet of the single responsibility principle evolves from decomposing static business domains (such as an Order Service or a Payment Service) to decomposing dynamic cognitive loads (such as a Planner Agent, a Researcher Agent, and an Execution Agent). As organizations rush to deploy these intelligent systems, a profound engineering crisis is emerging. A production AI agent is not merely a generative feature or a "magic box"; it is, fundamentally, a non-deterministic microservice. This architectural reality introduces severe complexities in system verification, observability, and quality assurance. Traditional microservices manage state transitions through strict, deterministic code where an input consistently yields a predictable output. Agentic microservices, conversely, operate via probabilistic reasoning, where the system is given a goal and granted the autonomy to determine the execution plan. This shift from static orchestration to dynamic, goal-oriented autonomy requires a radical reimagining of how distributed systems are tested, monitored, and deployed in enterprise environments. With industry analysts recording a massive surge in multi-agent system deployments, including a staggering 1,445% increase in enterprise inquiries within a single year, the question is no longer whether organizations will adopt agentic AI, but whether they possess the engineering discipline to run these systems in production reliably. This comprehensive report provides an exhaustive analysis of agentic microservice testing strategies, contrasting them deeply with traditional automation approaches. It explores the semantic protocols enabling multi-agent communication, the necessary evolution of the continuous testing pyramid, trajectory evaluation frameworks, behavioral chaos engineering, and the integration of agentic evaluation loops into continuous integration and continuous delivery (CI/CD) pipelines. The Evolution from Microservices to Agentic AI Architecture Before addressing how to test agentic systems, one must first dissect their architectural composition. As the industry transitions toward agentic AI, a common misconception among software architects is that existing infrastructure knowledge must be discarded. In reality, agentic AI architecture is the natural evolution of distributed microservices, enhanced by an active cognitive routing layer. In a traditional distributed system, intermediaries such as API gateways and load balancers function primarily as infrastructure components. They route network traffic and enforce generic security policies without deep application awareness or workflow intelligence. In an agentic architecture, the intermediaries often function as orchestrators or brokers and encapsulate significant application logic. They actively direct the sequence of operations, make content-aware routing decisions based on semantic understanding, and negotiate tasks dynamically. To manage this complexity, enterprise architectures are adopting structured multi-agent frameworks that align with the unique characteristics of AI technologies. These frameworks manage complexity through decomposition, improve resilience through decoupling, and simplify agent accountability through rigid specialization. A robust agentic system design typically models its architecture around specific layers and components: User layers: These define the human actors interacting with the system, ranging from external customers to authenticated internal employees.Agent layers: These describe the required autonomous entities, the specific design patterns they exhibit, their relationships with one another, and the systemic instructions used to actualize specific behaviors.Context and actions: These represent the resources, capabilities, and execution actions that the agent manages or has permission to access during its lifecycle.Sources: These encompass the underlying deterministic systems, such as relational databases, legacy applications, and vector knowledge bases, that the agents connect to for grounding and execution. Within these layers, multi-agent design patterns dictate the interaction structures that enable agents to communicate, collaborate, or even compete to solve complex problems. The Orchestrator-Worker pattern involves a primary agent breaking down a user request and delegating sub-tasks to specialized worker agents, such as a code-writing agent or a data-analysis agent. The Blackboard pattern allows multiple agents to independently read and write to a shared contextual memory space, asynchronously solving pieces of a puzzle without direct point-to-point communication. Furthermore, Reflection and ReAct (Reasoning and Acting) compound patterns enable individual agents to critique their own intermediate outputs, execute a self-correction loop, and refine their execution strategy before finalizing a task. Testing these architectural patterns requires validating not just the final output, but the intricate web of intermediate interactions, data handoffs, and self-correction loops that occur across extended periods and multiple state changes. The Foundational Divide: Deterministic vs. Probabilistic Systems The defining friction point in transitioning from traditional microservice test automation to agentic AI testing lies in the dichotomy between determinism and non-determinism. This fundamental difference alters the entire philosophy of quality assurance and continuous integration. Traditional software engineering and testing frameworks are built entirely on the assumption of determinism. Given a specific input state, a well-defined microservice is expected to produce the exact same output and state transition every single time it is executed. This predictability allows engineering teams to manage reliability efficiently. For example, if a transient network error occurs, traditional microservices rely on infrastructure-layer patterns like exponential backoff retries or circuit breakers to ensure eventual consistency. In this deterministic world, software testing involves straightforward, boolean checks against known outputs. A unit test asserts whether a specific value matches an expected string, providing a clear, binary pass or fail outcome. Agentic systems inherently violate these deterministic assumptions. The foundational LLMs that power these agents operate probabilistically, generating responses by predicting the next optimal token based on vast matrices of contextual weights and sampling strategies like temperature configurations. Consequently, feeding the exact same prompt to an agentic microservice multiple times can result in subtle variations in phrasing, entirely different reasoning paths, or occasionally, destructive hallucinations. When this probabilistic core is wrapped in a microservice boundary and granted autonomy over external tools and APIs, the system’s execution becomes a highly dynamic, unpredictable trajectory rather than a static, linear pipeline. This non-determinism introduces profound production challenges that standard automated testing cannot resolve. An agentic pipeline that successfully completes a complex workflow 95% of the time is not demonstrating a "passing test suite"; rather, it is indicating a production incident occurring in one out of every twenty executions. This forces a shift in testing methodology from simple output validation to comprehensive behavioral and outcome validation. Evaluation CategoryTraditional Microservice TestingAgentic Microservice TestingPrimary Validation Focus Exact output matching (e.g., asserting HTTP 200 responses, strict JSON schema parity, and predictable database state mutations). Behavioral validation, probabilistic trajectory evaluation, and optimization of broader business outcomes over exact textual outputs.Execution Path Architecture Static and predefined; execution relies on explicit flow control, rigid branching logic, and linear task execution. Dynamic and adaptive; the agent autonomously plans the sequence of tool calls, API interactions, and recovery steps based on real-time context.Debugging and Reproduction Identifying and recreating specific input states and payload parameters to reproduce the exact error consistently. Capturing the entire reasoning context, which includes initial prompts, RAG retrieval snippets, tool call sequences, and intermediate agent thoughts.Reliability Mechanisms Managed primarily at the infrastructure layer using load balancers, API gateways, automated retries, and explicit code fallbacks. Managed at the cognitive layer using self-reflection patterns, output validation loops, prompt engineering guardrails, and human-in-the-loop oversight.Component Communication Rigid API contracts negotiated prior to runtime, utilizing protocols like REST, gRPC, or GraphQL with strict data schemas. Dynamic task negotiation and standardized context exchange using specialized AI protocols such as Model Context Protocol (MCP) and Agent-to-Agent (A2A). To secure these non-deterministic workflows, testing must evolve to focus on whether the agent achieved its intended goal, satisfied key criteria, and gracefully handled unexpected tool responses, rather than verifying if it produced a mathematically identical string of text. Standardizing Cognitive Communication: The A2A and MCP Protocols A critical vector for testing agentic systems involves the communication boundaries between the agents themselves and the deterministic services they rely upon. In early generative AI experiments, multi-agent systems were heavily siloed. Agents operated within a single vendor's runtime environment, communicating with external tools through bespoke, fragile connectors. Attempting to scale this approach resulted in massive context window bloat, as developers were forced to inject dozens of tool schemas directly into the prompt, resulting in severe token overhead and degraded reasoning performance. To resolve this fragmentation, the industry is rapidly standardizing around two complementary semantic communication protocols that have recently moved under the vendor-neutral governance of the Linux Foundation: the Model Context Protocol (MCP) and the Agent-to-Agent (A2A) Protocol. Understanding and simulating these protocols is paramount for integrating tests into a microservices CI/CD pipeline. Model Context Protocol (MCP) Introduced by Anthropic in 2024 and now managed by the Agentic AI Foundation, MCP functions as a normalized, standardized interface connecting LLM-powered agents to external data sources and deterministic tools. Instead of hardcoding API integrations into the agent's logic, MCP allows an agent to dynamically discover and request access to capabilities hosted on an external MCP server. The execution flow of an MCP interaction requires rigorous testing. First, a user issues a request that exceeds the agent's innate knowledge or requires an action. The agent determines it needs external information and sends a structured request to the connected MCP server. The MCP server authenticates the request, verifies permissions, executes the deterministic tool, and returns the structured result. Finally, the agent integrates this fresh context into its working memory to formulate an accurate response. Testing MCP integration focuses heavily on semantic contract validation. QA teams must verify that the MCP server properly exposes its tool schemas, that the agent formulates its requests in strict adherence to those schemas, and that the agent gracefully handles scenarios where the MCP server returns an error code or an unexpected data format. Agent-to-Agent (A2A) Protocol While MCP focuses on lowering the complexity of connecting agents to inanimate tools, the A2A protocol introduced by Google in April 2025 and now an open-source Linux Foundation project standardizes communication between active, autonomous AI agents, particularly those deployed across different external systems or organizational boundaries. A2A allows agents to interact as peers capable of negotiation, rather than treating each other as simple APIs. A2A operates on a client-server principle over JSON-RPC 2.0 transport. The flow begins with an A2A Client performing a discovery operation against an A2A Server to retrieve an "AgentCard," a standardized manifest detailing the remote agent's specific capabilities, skills, and authentication requirements. Once a connection is established, the client agent sends a message containing a task to the server agent. The receiving agent evaluates this task, executes its own internal cognitive loops, and returns a response, potentially utilizing Server-Sent Events (SSE) for streaming updates or asynchronous push notifications. Simulating and Mocking Agent Protocols Because A2A and MCP interactions introduce extreme non-determinism at the network boundary, testing multi-agent systems end-to-end for every minor code change is both financially cost-prohibitive and technically fragile. To execute reliable integration tests, engineering teams must leverage behavioral simulation and advanced mocking techniques. Mocking in the agentic context goes beyond returning static JSON payloads. Tools like MockAgentServer provide local mock servers specifically designed for simulating A2A endpoints. These simulators allow developers to define request expectations and mock complex AgentCard discovery phases without incurring live LLM inference costs or network latency. By defining strict simulation rules, a mock A2A server can intentionally inject probabilistic failures such as returning a vaguely worded refusal to perform a task or simulating a conversational loop, allowing engineers to verify that the consuming agent's error handling and reflection capabilities function correctly under duress. The following Mermaid diagram illustrates the complex sequence of testing a multi-agent architecture where A2A and MCP protocols intersect, highlighting where mock servers intercept communication for isolated integration testing. Deconstructing and Rebuilding the Testing Pyramid The traditional software testing pyramid, popularized by Mike Cohn, untangles the complexity of software testing by enforcing an efficient hierarchical structure. It demands a massive foundation of fast, isolated unit tests, a middle layer of integration tests, and a small apex of slow, fragile end-to-end (E2E) UI tests. This structure ensures that the majority of testing efforts are spent on verifications that provide rapid, reliable feedback to developers. However, when applied to agentic AI, this traditional pyramid fractures. Because agents rely on non-deterministic planning, testing isolated units of code provides a dangerous false sense of security. An API endpoint might pass all unit tests flawlessly, but if the AI agent hallucinates the parameters or decides to invoke the wrong tool entirely, the system fails. Trying to force agentic workflows into strict pass-or-fail unit tests inevitably leads to flaky CI/CD pipelines or teams quietly disabling their test suites. To bring order to the chaos of autonomous systems, the industry has evolved an Agentic Testing Pyramid. This new paradigm separates deterministic tool validation at the base from probabilistic cognitive evaluation in the middle, culminating in behavioral trajectory evaluation at the top. Layer 1: Tools and Semantic Contracts (The Foundation) The bedrock of the Agentic Testing Pyramid remains deterministic. Before an agent can even attempt to reason about a tool or service, the underlying infrastructure must be mathematically flawless. This layer utilizes classic unit and API testing to ensure microservices function perfectly when invoked with the correct parameters. However, agentic systems require an advanced addition to this layer: Semantic Contract Testing. In loosely coupled, API-first microservice architectures, schemas inevitably evolve. In a traditional system, a schema drift (e.g., changing a field name from userId to user_id) might cause a compilation error or a swift HTTP 400 Bad Request, allowing immediate detection. AI agents, however, are highly adaptable and simultaneously brittle. An agent encountering a changed schema might attempt to "hallucinate" a workaround, guess the missing parameters, or worse, map sensitive data to the wrong fields, leading to unpredictable and silent mutations. Semantic contract testing frameworks, such as Pact or Spring Cloud Contract, enforce explicit, version-controlled blueprints of communication between the agent (Consumer) and the external microservice (Provider). By validating response structures and enforcing strict tool input/output contracts including required parameters, typed outputs, and stable error codes, teams can prevent schema drift from silently breaking autonomous workflows. Next-generation AI-powered contract testing tools advance this further by analyzing actual API behaviors to automatically infer contracts and detect breaking changes without requiring manual test script maintenance. Layer 2: Agent Cognition and Decision Evaluations The middle layer of the pyramid shifts from evaluating code to evaluating the "brain" of the agent. The core validation metric here is cognitive routing: For a given prompt or complex goal, does the agent formulate the correct operational plan, and does it call the correct tools in the correct sequence with the proper semantic arguments?. Traditional programmatic assertions are useless here. Instead, developers must leverage evaluation frameworks that score the agent's decisions against predefined "ground truth" datasets. This involves calculating metrics such as Task Adherence, comparing the agent's intermediate outputs to the original query intent, and Tool Use Accuracy. Frameworks like Ragas calculate ToolCallAccuracy by executing a set of test prompts and comparing the agent's actual tool invocations against an optimal reference list, producing a statistical pass/fail score that indicates whether the agent made the right cognitive leap. Layer 3: Multi-Agent Trajectories and System Outcomes The apex of the Agentic Testing Pyramid evaluates the full, multi-turn lifecycle of the system. This layer assesses emergent behaviors, contextual memory drift over long sessions, and the coordination overhead between multiple agents. Because these evaluations require running the LLM through multiple, complex inference cycles, often interacting with external sandboxes, they are inherently slower and more expensive, justifying their position at the top of the pyramid. Evaluating outcomes requires sophisticated Trajectory Evaluation Metrics. A trajectory represents the complete sequence of actions, tool invocations, and state transitions an agent traverses to solve a problem. Platforms like the Vertex AI Gen AI evaluation service provide specialized metrics for this layer: Exact match: This metric demands strict adherence, requiring the agent to produce a sequence of actions that perfectly mirrors an expert-annotated reference trajectory.In-order match: This evaluates whether the agent's trajectory includes all necessary actions in the correct sequence, penalizing missed steps but tolerating extra, exploratory, or self-correction steps.Any-order match: Highly flexible, this metric verifies that the agent ultimately executed all required functions to achieve the goal, regardless of the specific sequence it chose to reach the outcome.Precision and recall: Precision calculates the proportion of actions taken by the agent that were actually necessary (punishing hallucinations and wasted tool calls), while recall measures the agent's ability to successfully discover and execute all the essential steps required by the reference solution. Metrics, Telemetry, and Evaluation Platforms Evaluating agentic microservices effectively demands a comprehensive matrix of telemetry that extends far beyond simple accuracy. An agent that perfectly completes a task but consumes an entire daily API budget to do so is a failure in a production environment. Therefore, enterprise testing strategies must balance intelligence with system performance, reliability, and cost. The Multidimensional Evaluation Matrix When transitioning agentic pipelines to production, testing telemetry must capture and analyze data across four critical dimensions : Evaluation DimensionCore Metrics & IndicatorsEvaluation MethodologyIntelligence & Accuracy Task Completion Accuracy, Logical Reasoning Quality, Multi-step Coherence, Grounding Faithfulness, and Contextual Awareness. Automated LLM-as-a-judge scoring, reasoning trace analysis, semantic similarity benchmarks, and human-in-the-loop review queues.Performance & Efficiency Time-to-First-Token (TTFT), End-to-End Wall-Clock Latency, Cost per Successful Task (compute time, token usage, API calls), and Resource Utilization. Distributed tracing via OpenTelemetry, token counting interceptors, latency monitoring dashboards, and payload size tracking.Reliability & Resilience Input Variation Robustness, API Failure Recovery (graceful degradation), Context Retention over extended sessions, and Long-session Memory Stability. High-volume stress testing, deterministic failure injection (simulating API timeouts), and contextual drift analysis.Responsibility & Governance Harmful Content Prevention, Adversarial Prompt Resistance, Privacy Boundary Compliance, PII Scrubbing, and Access Control Adherence. Automated red teaming, adversarial dataset injection, policy compliance checking, and vulnerability scanning. Advanced Agent Evaluation Platforms To capture this matrix of telemetry, the industry has matured rapidly to provide sophisticated tooling. The selection of an evaluation platform dictates how deeply testing can be integrated into the CI/CD pipeline and the observability stack. DeepEval: An open-source evaluation framework built natively into the Python testing ecosystem, deeply integrated with Pytest. DeepEval is engineered for teams requiring customized, off-the-shelf metrics, automated prompt optimization, and deep CI/CD pipeline integration. It allows developers to use standard testing paradigms (e.g., assert_test) to evaluate LLM applications at the component level, making it highly effective for shift-left testing.MASEval: A multi-agent native evaluation library released in 2026 that sits between agent frameworks and benchmarks. It provides a unified evaluation layer enabling framework-agnostic, system-level comparisons across any agent framework (like LangGraph or smolagents) without requiring users to rewrite orchestration infrastructure.LangSmith: Developed by the creators of LangChain, LangSmith provides industry-leading tracing and evaluation tightly coupled with the LangChain and LangGraph ecosystems. It excels at visualizing complex multi-agent traces and provides powerful annotation queues that allow product managers and QA engineers to conduct human review on edge-case interactions at scale.Arize Phoenix: While tools like DeepEval focus heavily on pre-production benchmarking, Arize Phoenix is an enterprise-grade platform centered on production machine learning monitoring and observability. It provides vendor-neutral, OpenTelemetry (OTel)-native instrumentation to detect post-deployment issues such as context drift and embedding anomalies. Teams frequently utilize a multi-layer stack, employing DeepEval for CI/CD pipeline gating and Arize Phoenix for continuous production telemetry.Braintrust and Comet Opik: Braintrust offers opinionated, structured evaluation pipelines specifically designed to gate CI/CD workflows and facilitate team collaboration, while Comet Opik focuses on automated prompt and tool optimization across a broader framework ecosystem. Site Reliability Engineering (SRE) for Non-Deterministic Pipelines The realization that an AI agent is a non-deterministic microservice brings an immediate operational imperative: the application of Site Reliability Engineering (SRE) principles. Multi-agent systems face distinct, hard production problems that separate successful enterprise deployments from fragile prototype demos. The foremost SRE challenge is cost unpredictability. Unlike traditional microservices that scale linearly with user traffic, agentic costs involve variable execution paths. A single edge-case input that triggers an agent to enter a confused retry chain or a continuous reflection loop can execute dozens of external tool calls and consume massive amounts of tokens, resulting in a single transaction costing orders of magnitude more than a nominal path. Furthermore, in multi-agent architectures, token consumption compounds across orchestration layers due to context multiplication where the findings of one agent are injected into the prompts of several others. To manage these systems reliably, engineering teams must deploy custom instrumentation and apply core SRE practices directly to agent pipelines: Service level objectives (SLOs) and error budgets: Organizations must define strict SLOs not just for system uptime, but for cognitive behaviors. This includes establishing acceptable output quality thresholds, maximum execution latencies, and strict cost-per-task ceilings. Error budgets create accountability, preventing the accumulation of reliability debt caused by flaky agent deployments.Agent-native distributed tracing: Mature distributed tracing for agentic workflows is vital. SRE teams must log every tool invocation, context handoff between agents, and internal retry attempt. This level of observability ensures that when a multi-agent system stalls, engineers can pinpoint whether the failure occurred due to a prompt misunderstanding, an MCP timeout, or a context parsing error.Graceful degradation: Agentic pipelines must be designed with fallback paths rather than all-or-nothing execution. If a specialized sub-agent fails to respond or produces a malformed output, the orchestrator agent should be engineered to bypass that specific insight, fall back to a simpler execution path, or return a partial result to the user rather than crashing the entire workflow or initiating a retry storm. Behavioral Chaos Engineering and Contextual Guardrails Because agentic microservices operate with autonomy, traditional security and penetration testing, which hunts for deterministic vulnerabilities like SQL injections or buffer overflows, is entirely insufficient. The attack surface of an agentic system expands drastically to include the agent's reasoning capabilities, its context window, and its probabilistic interpretation of instructions. This necessitates the adoption of Behavioral Chaos Engineering and Contextual Red Teaming. Dynamic Capability Mapping and Red Teaming Agentic AI red teaming efforts must evolve from testing static infrastructure to actively probing the behavioral boundaries of intelligent agents. This practice draws direct inspiration from chaos engineering in distributed systems, applying controlled turbulence to the agent's internal "mind" to ensure safety and robustness under real-world uncertainty. The threat model for agentic systems is multi-layered, heavily featuring input manipulation tactics such as prompt injection attacks, context poisoning, and goal hijacking. In multi-agent environments, vulnerabilities easily cascade across the network. For instance, if an attacker successfully poisons a document retrieved by a Researcher Agent, that poisoned context is subsequently passed to an Execution Agent, potentially resulting in unauthorized data exfiltration or fraudulent API executions. To combat this, automated red teaming frameworks execute dynamic capability mapping. Instead of running a static script, an autonomous "Profiler" red-team agent systematically converses with the target agent to map its capabilities. In documented enterprise security exercises using platforms like Prisma AIRS AI Red Teaming, Profiler agents have successfully extracted critical operational intelligence entirely through conversational interaction, discovering the target agent's available backend tools (e.g., withdraw_funds, execute_sql_query), mapping the complete database schema, identifying hidden authentication dependencies, and detecting the absence of rate limiting. This adversarial system reconnaissance validates whether tool-layer authorization can withstand conversational exploitation, proving that prompt-level security is insufficient without system awareness. Implementing Autonomous Guardrail Microservices To mitigate these cognitive vulnerabilities dynamically at runtime, architectures must integrate specialized guardrail microservices. These frameworks act as semantic firewalls, intercepting inputs before they reach the LLM and validating outputs before tools are executed. NVIDIA NeMo guardrails: A highly performant, enterprise-grade open-source toolkit optimized for GPU-accelerated environments. NeMo leverages Colang, a specialized modeling language designed to define strict dialogue state machines that govern how users walk through an AI interaction. It excels in complex conversational systems, offering robust content safety, topical boundary enforcement, PII detection, and strict enforcement of Retrieval-Augmented Generation (RAG) grounding. While powerful, its integration with the broader NVIDIA AI stack results in a steeper learning curve.Guardrails AI: A Python-native validation framework that prioritizes flexibility, ease of use, and autonomy in implementation. Utilizing Pydantic-style validation and its proprietary RAIL (Reliable AI Markup Language) specification, Guardrails AI allows developers to define fine-grained structural and semantic boundaries for LLM outputs. If an LLM returns data that violates a RAIL specification, the framework can automatically initiate a self-correction loop, re-prompting the LLM with the validation error to force a corrected response before the data ever reaches the broader system. By operating as independent microservices within the agentic architecture, these guardrail tools ensure that user-facing interactions and internal agent-to-agent data handoffs are rigorously monitored and scrubbed for policy compliance in real-time. The Agentic CI/CD Pipeline and Context Management In traditional, deterministic microservice development, the continuous integration and continuous deployment (CI/CD) pipeline operates essentially as an automated conveyor belt. Code is pushed, static analysis and unit tests execute, resulting in a binary pass or fail; a container image is built, and the artifact is deployed to production. In the era of Agentic AI, engineering teams are no longer just managing code; they are managing context. The configurations that steer an AI agent, including system instructions, prompt templates, tool schemas, and model hyperparameters, dictate the system's behavior entirely. Consequently, the traditional CI/CD conveyor belt must evolve into an Agentic Evaluation Loop, a continuous feedback cycle heavily reliant on statistical thresholds rather than binary assertions. Prompt Versioning as Infrastructure-as-Code Because agent performance is hyper-sensitive to subtle textual changes, managing prompts requires the same strict discipline as database schema migrations. Changing a seemingly benign system prompt variable from {{user_name} to {{user_id} can drastically alter an agent's reasoning pattern and its subsequent tool invocations. Best practices for agentic CI/CD dictate a rigorous approach to prompt versioning: Immutable versioning: Every prompt change must be assigned a unique version ID. Crucially, prompts must be versioned alongside their execution context, meaning the template structure, variables, and the specific model parameters (such as temperature and top-p) must be tracked as a single, immutable configuration. This ensures reliable rollback mechanisms and precise tracing of production outputs back to specific configurations.Environment management and rollbacks: Agents should never be deployed blindly. CI/CD pipelines must leverage feature flags and A/B deployments, running stable and testing environments simultaneously. If production health monitoring detects a spike in fault rates or latency, teams can seamlessly roll back to a known-good prompt version without requiring a full code redeployment. Integrating Evaluation Loops into CI/CD When a developer opens a pull request that modifies an agent's configuration, the CI pipeline must pause the conveyor belt and trigger an automated offline evaluation suite. Using frameworks like DeepEval, the pipeline executes the updated agent against a comprehensive "golden dataset" composed of historical user interactions, edge cases, and synthetic data. Because agents are probabilistic, tests rarely pass at 100%. Therefore, CI/CD pipelines must enforce statistical threshold-based gating. For example, a GitHub Actions YAML configuration utilizing DeepEval can be set to require an 85% Exact Match score for multi-agent trajectories and a 95% Contextual Relevance score. If the evaluation scores fall below the threshold, the merge is blocked. If the automated tests pass, the pipeline generates a quality report diff. For high-risk or ambiguous domains, this report is forwarded to an annotation queue (such as those provided by LangSmith) where human-in-the-loop reviewers provide final judgment before the agent is deployed. The Inversion of QA: Agentic Frameworks for Test Automation The ultimate, systemic evolution of the agentic testing strategy is the application of agentic capabilities to the Quality Assurance process itself. As applications grow increasingly complex with API integrations, dynamic user interfaces, and intricate microservice architectures, traditional test planning approaches that rely heavily on manual analysis, static documentation, and human intuition are failing to keep pace. Traditional automated testing tools depend heavily on static scripts that become brittle and break upon the slightest UI or codebase modification, generating massive manual maintenance overhead. Agentic QA Frameworks, such as those provided by platforms like Baserock and VirtuosoQA, represent a paradigm shift in test automation. These systems deploy AI agents to independently analyze application architectures, identify technical risk areas, execute testing workflows, and refine strategies without continuous human intervention. Operating on a framework based on MAPE-K (Monitor, Analyze, Plan, Execute, Knowledge), Agentic QA transforms the testing infrastructure into an autonomous entity. Application analysis agents automatically scan codebases, APIs, and user interfaces to comprehend the latest architectural state, data flows, and integration points.Risk assessment agents continuously evaluate this architecture to identify high-priority vulnerabilities. They dynamically prioritize testing based on business risk, allocating deep coverage to complex payment processing workflows while assigning lower priority to static documentation pages.Strategy generation agents then automatically generate and execute dynamic test scenarios that cover the identified risk areas, adapting to code changes on the fly and remediating minor test script failures in real-time. By learning from execution outcomes such as identifying frequent, flaky failures or recognizing redundant test paths, these autonomous testing agents continuously optimize the testing process, fundamentally transforming QA professionals from script writers into strategic supervisors of intelligent, self-healing systems. Conclusion The architectural transition from rigid, deterministic microservices to probabilistic, goal-oriented agentic systems represents a fundamental restructuring of enterprise software development. Organizations can no longer rely solely on binary unit tests, static API contracts, or traditional CI/CD pipelines to guarantee system stability and reliability. The inherent non-determinism of large language models, coupled with the autonomy granted to agents to execute external tools and negotiate with peer systems, introduces profound operational challenges ranging from cost unpredictability to cascading cognitive failures. Mastering agentic microservice testing requires engineering teams to completely deconstruct and rebuild their quality assurance methodologies. By establishing a new Agentic Testing Pyramid, teams can secure the foundation with AI-powered semantic contract testing to prevent schema drift. The middle layers must evolve to evaluate cognitive decision-making using specialized frameworks to benchmark tool-call accuracy and task adherence. At the apex, sophisticated trajectory evaluation metrics ensure that the multi-step, emergent behaviors of multi-agent interactions reliably achieve broader business outcomes. Furthermore, integrating continuous, threshold-based evaluation loops into CI/CD pipelines, enforcing immutable prompt versioning, and deploying behavioral chaos engineering alongside active guardrail microservices are no longer optional advancements; they are baseline requirements. The future of scalable, enterprise-grade AI relies not just on how intelligently an autonomous agent can act, but on how rigorously, systematically, and continuously those actions can be validated in a non-deterministic world.

By Viquar Khan

CORE

Prevent Duplicate API Calls With Idempotency: Patterns That Work

A failure pattern I have seen repeatedly in enterprise integrations: a payment request times out at the caller, succeeds downstream, and is then retried as though it failed. MuleSoft projects, this is especially easy to miss because a single inbound request may trigger payment, ERP, inventory, and messaging calls before the caller receives a final response. It succeeded 2 times. The customer was charged twice. Inventory was reserved twice. The ERP received two invoice requests. Nothing crashed. Every service did exactly what it was designed to do. The problem was simpler and more dangerous: the caller retried before anyone could prove whether the first request had already succeeded. That is the part people miss when they say, "just add retries." Retries are not reliability by themselves. A retry is a second attempt to perform a business operation. If the first attempt reached the server but the response was lost, a retry can create a duplicate order, a duplicate payment, a duplicate shipment, or a duplicate customer record. Idempotency is what makes retries safe. In enterprise integration, that distinction matters because the network cannot tell you the truth quickly enough. A timeout only tells you that the caller did not receive a response. It does not tell you whether the downstream system completed the work. The Incident: Timeout That Became Two Invoices Flow that looks normal: Plain Text Commerce API → MuleSoft order integration → Payment service → ERP A customer placed an order. The Commerce API sent POST /orders to the integration layer. MuleSoft validated the payload, called the payment service, then created an invoice in the ERP. The ERP was slow that morning. Not down. Just slow enough to produce a bad distributed-systems outcome. The first request reached MuleSoft. Payment succeeded. The invoice request reached the ERP. Then the Commerce API timed out waiting for the response. Its retry policy did what it had been configured to do: retry once. The second request was treated as a brand-new order. Payment ran again. The ERP created another invoice. Sequence diagram showing a timeout followed by a retry that charges the card twice and creates two ERP invoices: Figure 1: Before. The response was lost, not the work. The retry repeats every side effect, producing a duplicate charge and a duplicate invoice. From the perspective of each individual component, this was reasonable. The caller saw a timeout and retried. MuleSoft received a valid request and processed it. Payment received two valid payment instructions. The ERP received two valid invoice requests. The tech team may initially treat this as a timeout-tuning problem. Whereas it is an ownership problem: the API layer, integration layer, and system of record each assume another component will prevent the duplicate. Unless the business operation has one durable identity across those boundaries, none of them can reliably do it. That is the real job of idempotency: give the system a durable way to recognize the same intent when it arrives again. Idempotency Is Not "the API Returns the Same Response" The formal definition is usually presented as: repeating the same operation produces the same result. That is technically useful, but it is not enough for a production integration. For a business API, the practical definition is better. For one business intent, perform the side effect at most once, then return the recorded outcome for every valid repeat. That definition has three important parts. First, it is tied to business intent, not merely an HTTP request. A user buying two identical laptops should create two orders. A retry of the same "buy one laptop" action should not. Second, it protects side effects. Returning the same JSON response is meaningless if the system already charged the card twice. Third, it requires a recorded outcome. If the first request succeeded, the retry should receive the original successful response. If the original request was rejected, the retry should receive the same rejection. A unique database constraint helps, but it is only one layer. It does not automatically coordinate payment, messaging, invoice creation, and the response sent back to the caller. The dangerous period is the gap between "the work may have happened" and "the caller knows the result." Distributed systems spend a lot of time in that gap. A connection can reset after the downstream commit. A load balancer can close an idle connection. A worker can complete the work and die before it writes the response. A message can be delivered again after a consumer restart. You cannot eliminate every ambiguity. You can design so that ambiguity does not create duplicate business activity. Two-column comparison contrasting a retry that repeats the side effect against a retry that replays the recorded outcome: Figure 2: The retry is identical in both columns. Only the server's memory of the intent differs. The Implementation Path: An Idempotency Key and a Durable Record The implementation begins with an Idempotency-Key header. The client creates a unique key for one business attempt and sends the same key on every retry. HTTP POST /orders HTTP/1.1 Idempotency-Key: 0a5a98a0-7b40-4a8f-a5e2-7cf94e75a825 Content-Type: application/json The server stores that key with a request fingerprint, a processing state, the eventual response, and an expiry. Field Purpose idempotency_key Identifies one client business attempt operation Prevents a key for /orders being reused for /refunds request_hash Detects the same key arriving with different content status Tracks IN_PROGRESS, COMPLETED, or FAILED work response_status Replays the original HTTP status response_body Replays the original API response expires_at Allows safe cleanup after the retry window closes The key must be unique per operation. If a client reuses a key with a different payload, return a conflict. Never silently treat different requests as the same request. The Critical Rule: Reserve the Key Before the Side Effect The idempotency record must be created before payment, invoice creation, message publishing, or any other irreversible action. In a Spring service, the first durable action is an atomic insert. Java public OrderResponse createOrder(String idempotencyKey, CreateOrderRequest request) { String requestHash = RequestHasher.sha256(RequestCanonicalizer.canonicalize(request)); Reservation reservation = idempotencyService.reserve(OPERATION, idempotencyKey, requestHash); if (reservation.isReplay()) { IdempotencyRecord existing = reservation.record(); if (!existing.getRequestHash().equals(requestHash)) { throw new ResponseStatusException(HttpStatus.UNPROCESSABLE_ENTITY, "Idempotency key was reused with a different request payload"); } switch (existing.getStatus()) { case COMPLETED: return existing.replayResponse(); case IN_PROGRESS: throw new OrderProcessingException(existing.retryAfterSeconds()); case FAILED: throw new ResponseStatusException(HttpStatus.CONFLICT, "This idempotency key already failed: " + existing.getFailureCode()); default: throw new IllegalStateException("Unknown state " + existing.getStatus()); } } // We own the key. Every side effect below carries it as its business identity. try { PaymentResult payment = paymentClient.charge( request.customerId(), request.total(), request.currency(), idempotencyKey); ErpInvoice invoice = erpClient.createInvoice( request.orderReference(), payment.transactionId(), idempotencyKey); OrderResponse response = OrderResponse.created( request.orderReference(), payment.transactionId(), invoice.invoiceNumber()); idempotencyService.complete(OPERATION, idempotencyKey, HttpStatus.CREATED.value(), response); return response; } catch (RuntimeException exception) { if (OutcomeClassifier.isTerminal(exception)) { idempotencyService.markFailed(OPERATION, idempotencyKey, exception.getClass().getSimpleName()); } else { idempotencyService.markAmbiguous(OPERATION, idempotencyKey, exception.getClass().getSimpleName()); } throw exception; } } Two details in that method are easy to get wrong. The first is where uniqueness is enforced. Two requests can arrive at nearly the same moment, and both can pass an application-level "does this key exist?" check. The guarantee belongs in the database. SQL CREATE UNIQUE INDEX ux_idempotency_operation_key ON idempotency_record (operation, idempotency_key); The reservation then treats a duplicate-key exception as an ordinary concurrency outcome rather than an error, and commits in its own transaction so that a later downstream failure cannot roll away the evidence that the attempt was made. Java @Transactional(propagation = Propagation.REQUIRES_NEW) Reservation reserve(String operation, String key, String requestHash) { try { return Reservation.acquired(repository.insertInProgress(operation, key, requestHash)); } catch (DuplicateKeyException concurrentInsert) { // Another thread, pod, or retry won the race. Its record is authoritative. return Reservation.replay(repository.require(operation, key)); } } The second detail is the catch block. Marking a record FAILED because a downstream call timed out is how teams reintroduce the original bug: the next retry sees a terminal state, decides the work never happened, and starts a second charge. An ambiguous outcome is not a failure. Keep it distinguishable, and let a reconciliation job resolve the true state against the payment provider and the ERP. IN_PROGRESS Is a Real Production State Many implementations get the happy path right and fail during concurrent retries. The original request starts processing, but the client times out after two seconds and immediately retries. The original operation is still waiting on the ERP. What should the second request receive? Not another payment attempt. It should receive a clear, boring answer. HTTP HTTP/1.1 409 Conflict Retry-After: 3 { "code": "ORDER_PROCESSING", "message": "This order request is already being processed." } The caller can wait and retry with the same idempotency key. Once the first request completes, the next retry returns the stored result. That behavior is not glamorous. It is predictable. In integration systems, predictable is often more valuable than fast. MuleSoft: Enforce the Contract at the API Boundary The integration layer is a strong place to enforce this contract, because it sees the inbound request before it fans out to multiple systems. The important structural choice is that reservation is an insert, not a lookup. A SELECT followed by an INSERT is not safe under concurrent retries, so the flow attempts the insert first and interprets the unique-constraint error as "someone else owns this intent." XML <flow name="create-order-api"> <http:listener config-ref="httpListener" path="/orders" allowedMethods="POST"> <http:response statusCode="#[vars.httpStatus default 201]"/> <http:error-response statusCode="#[vars.httpStatus default 500]"/> </http:listener> <validation:is-not-blank-string value="#[attributes.headers.'idempotency-key' default '']" message="Idempotency-Key header is required"/> <set-variable variableName="idempotencyKey" value="#[attributes.headers.'idempotency-key']"/> <set-variable variableName="orderRequest" value="#[payload]"/> <ee:transform doc:name="Canonicalize request"> <ee:message> <ee:set-payload resource="dw/normalize-order-request.dwl"/> </ee:message> </ee:transform> <set-variable variableName="requestHash" value="#[dw::core::Crypto::SHA1(write(payload, 'application/json') as Binary)]"/> <try doc:name="Reserve idempotency key"> <db:insert config-ref="OrderDb"> <db:sql><![CDATA[ INSERT INTO idempotency_record (operation, idempotency_key, request_hash, status, created_at, updated_at, expires_at) VALUES ('CREATE_ORDER', :key, :hash, 'IN_PROGRESS', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP + INTERVAL '24' HOUR) ]]></db:sql> <db:input-parameters><![CDATA[#[{ key: vars.idempotencyKey, hash: vars.requestHash }]]]></db:input-parameters> </db:insert> <set-variable variableName="keyOwned" value="#[true]"/> <error-handler> <on-error-continue type="DB:QUERY_EXECUTION"> <set-variable variableName="keyOwned" value="#[false]"/> </on-error-continue> </error-handler> </try> <choice> <when expression="#[vars.keyOwned == true]"> <flow-ref name="process-order-side-effects"/> </when> <otherwise> <flow-ref name="resolve-existing-idempotency-record"/> </otherwise> </choice> </flow> The resolution flow is where the states earn their keep. A hash mismatch returns 422. A COMPLETED record replays the stored body with its original status. An IN_PROGRESS record returns 409 with Retry-After. XML <choice> <when expression="#[vars.record.request_hash != vars.requestHash]"> <set-variable variableName="httpStatus" value="#[422]"/> </when> <when expression="#[vars.record.status == 'COMPLETED']"> <set-payload value="#[read(vars.record.response_body, 'application/json')]"/> <set-variable variableName="httpStatus" value="#[vars.record.response_status default 200]"/> </when> <when expression="#[vars.record.status == 'IN_PROGRESS']"> <set-variable variableName="httpStatus" value="#[409]"/> </when> </choice> DataWeave: Canonicalize Before Hashing The same logical order can arrive with fields in a different order, with different casing, or with optional values that are absent in one attempt and empty in the next. If you hash raw payload text, semantically identical requests produce different hashes, and your idempotency layer quietly stops working. Normalize first, then hash the normalized output. Plain Text %dw 2.0 output application/json skipNullOn = "everywhere" fun canonicalText(value) = trim(value default "") match { case s if s == "" -> null else -> s } var lineItems = (payload.items default []) map (item) -> { sku: upper(trim(item.sku)), quantity: item.quantity as Number, unitPrice: item.unitPrice as Number } orderBy ((item) -> item.sku ++ "|" ++ (item.unitPrice as String)) --- { customerId: trim(payload.customerId), orderReference: trim(payload.orderReference), currency: upper(trim(payload.currency default "USD")), total: payload.total as Number, items: lineItems, shipTo: if (payload.shipTo == null) null else { line1: canonicalText(payload.shipTo.line1), city: canonicalText(payload.shipTo.city), region: upper(trim(payload.shipTo.region default "")), postalCode: canonicalText(payload.shipTo.postalCode), country: upper(trim(payload.shipTo.country default "")) } } Note what is excluded. Client timestamps, trace identifiers, and transport metadata do not belong in the fingerprint, because a legitimate retry carries new values for them. The goal is not cryptographic cleverness. The goal is to define what "the same order request" means in your domain, and to write that definition down in code. With the key reserved and the fingerprint stable, the retry path changes shape entirely. Sequence diagram showing a retry that finds a completed idempotency record and replays the original response without repeating payment or invoicing Figure 3: After. The retry still happens. It finds the completed record and replays the original response instead of repeating the work. Where Idempotency Belongs Do not stop at the public API if the flow crosses more than one boundary. The useful pattern is to propagate the key as correlation metadata so that every system on the path shares one business identity. Architecture diagram showing the business key propagated from the commerce client through MuleSoft, the idempotency store, the payment provider, the ERP, and the event bus: Figure 4: The same business key becomes the provider idempotency header, the ERP external document reference, and the event deduplication key. Each downstream system needs a compatible protection mechanism. Payment providers generally accept their own idempotency header. ERP systems usually support an external document reference with a unique constraint. Message consumers can record processed event IDs before applying the side effect. Databases should enforce unique business keys wherever the domain allows it. A single API gateway record cannot make an entire distributed transaction atomic. It does, however, give every downstream call a stable business identity, which is what turns an ambiguous retry into an answerable question. The Questions to Answer Before You Ship An idempotency design is incomplete until the team can answer these: What is one business intent in this API?Who creates the idempotency key, and is it unique per attempt rather than per session?Which fields define the request fingerprint, and which are deliberately excluded?What happens when the same key arrives with a different payload?What does a caller receive while the original request is still in progress?Which downstream side effects receive the same business key?How do you recover records stuck in IN_PROGRESS after a crash?How long are completed keys retained? The retention period should cover the realistic retry window, including asynchronous queues and client retry behavior. For many order APIs, 24 hours is a reasonable starting point; the right answer depends on the business process, not a generic framework default. The stuck-record question deserves more attention than it usually gets. A pod that dies mid-operation leaves a reservation nobody owns. Without a sweeper that reconciles those records against the payment provider and the ERP, your safety mechanism becomes a source of permanent 409 responses for a key the client will keep retrying. The Real Reliability Pattern Retry budgets protect a system from retry storms. Circuit breakers protect a dependency from overload. Timeouts prevent callers from waiting forever. Idempotency protects business operations when those mechanisms encounter uncertainty. That is why it belongs beside retries, not after them. A timeout is not evidence of failure. It is evidence that the caller does not know the outcome. My rule for enterprise integrations is simple: a retry must continue a known business operation, not create a new one. If the system cannot tell those two situations apart, retries are not a reliability feature yet.

By Manjeera Chanda