DZone Spotlight

Retries are one of the simplest ways to make a distributed system appear more reliable. A transient connection failure, overloaded replica, or short-lived network interruption can disappear after another attempt, which is why retry support exists in major RPC frameworks and cloud SDKs. The danger begins when every layer makes the same decision independently. A mobile client retries an API gateway, the gateway retries a service, that service retries another service, and the final dependency retries a database call. The original request has not become more important, but the system has multiplied the work required to fail. AWS describes a five-deep service stack in which three attempts at each layer can drive 243 calls against the database when the deepest dependency is failing. Google’s SRE guidance similarly warns that retries can amplify overload and contribute to cascading failure. When Reliability Logic Becomes Additional Load The common retry policy focuses on a single caller. A request fails, exponential backoff delays the next attempt, and jitter prevents large client populations from retrying at exactly the same instant. Those mechanisms remain important. AWS recommends backoff and jitter because immediate, synchronized retries can worsen overload, while gRPC exposes retry limits, exponential backoff, retry throttling, and server pushback for the same class of problem. The missing property is coordination. Consider three logical layers, each configured for three total attempts. If the lowest dependency rejects every request, a single logical operation can create up to 27 downstream attempts. Adding more independently retrying layers increases that multiplier exponentially. Backoff changes when those attempts arrive; it does not change the fact that separate components are authorizing additional work from the same original operation. A typical Spring service can accidentally create this behavior with perfectly reasonable local configuration: Java @Retry(name = "paymentService", fallbackMethod = "paymentFailed") public PaymentResult charge(PaymentRequest request) { return paymentClient.charge(request); } private PaymentResult paymentFailed(PaymentRequest request, Exception ex) { throw new PaymentUnavailableException(ex); } Nothing in this method indicates whether the incoming request has already consumed retries elsewhere. A gateway may already have retried the service, and paymentClient may apply another retry policy. Local resilience therefore becomes global amplification. A Retry Budget Changes the Decision A retry budget treats retries as limited capacity rather than an unconditional reaction to failure. Google documents two complementary controls in its overload handling: a per-request cap of three attempts and a per-client budget that permits retries only while retries remain below 10% of request traffic. In the example described by Google, the per-client budget reduces retry-driven traffic growth from almost three times the original request rate to roughly 1.1 times under the modeled overload condition. Finagle applies the same general idea through a shared RetryBudget, explicitly describing the budget as protection against the amplifying effect of many clients retrying. For a service chain, the useful abstraction is a request-scoped budget propagated with the operation. An internal header such as X-Retry-Budget can represent remaining retry permits. The header is an application convention rather than a standard HTTP field, its purpose is to ensure that downstream components consume from the same finite allowance. The retry decision can then become explicit: Java boolean canRetry(int remaining, HttpStatusCode status) { return remaining > 0 && (status.value() == 429 || status.is5xxServerError()); } int nextBudget(int remaining) { return Math.max(0, remaining - 1); } A caller starts a logical operation with a small budget, such as two retry permits. Every additional attempt decrements the value before forwarding the request. A downstream service receiving zero can still return a meaningful failure, but it cannot create more retry traffic for that logical operation. This model should not make every 5xx automatically retryable. Retry classification still matters. Validation failures, deterministic application errors, and non-idempotent operations can be unsafe or pointless to repeat. AWS recommends idempotent API contracts when operations may be retried and describes caller-provided request identifiers as a way to recognize duplicate intent. Propagating One Budget Across Service Boundaries Budget propagation belongs close to outbound transport logic so business methods do not manually manipulate retry metadata. A Spring interceptor can read the current budget and attach the decremented value to the next attempt: Java int remaining = retryContext.remaining(); if (remaining <= 0) { throw new RetryBudgetExhaustedException(); } request.getHeaders().set( "X-Retry-Budget", Integer.toString(remaining - 1) ); return execution.execute(request, body); The receiving service extracts the header once and places the value in the request context. Internal HTTP clients and RPC adapters then share that context. This is conceptually similar to distributed context propagation used by tracing systems. OpenTelemetry propagators inject and extract cross-cutting context through carriers such as HTTP headers, although retry-budget metadata can remain a dedicated internal header rather than telemetry baggage. A budget also needs to cooperate with deadlines. A remaining retry permit is useless when the logical request has only a few milliseconds left. Retry authorization should therefore require both budget and time: Java boolean retryAllowed(RetryContext context) { return context.remaining() > 0 && context.deadline().isAfter(Instant.now().plusMillis(100)) && context.lastFailure().isTransient(); } Server feedback should override generic retry enthusiasm. HTTP defines Retry-After so a service can indicate when a follow-up request should occur, including with 503 Service Unavailable, 429 Too Many Requests can also carry Retry-After. A budget answers whether another attempt is permitted, while server feedback helps decide when that attempt is appropriate. Measuring Whether the Budget Is Working Retry budgets are control mechanisms, so observability must expose both logical requests and physical attempts. Finagle distinguishes logical success from individual attempts and publishes metrics for retry budget availability, exhaustion, and request retry limits. Without that separation, retries can hide dependency instability because a successful second attempt makes the logical request appear healthy while infrastructure performs additional work. Useful telemetry should record the initial request count, retry attempt count, budget exhaustion count, retry success rate, response classification, remaining budget, and end-to-end latency. The critical ratio is retry amplification, which is total physical attempts divided by logical requests. A healthy value depends on workload characteristics, but a sharp increase during an incident indicates that resilience logic is becoming a load. Tracing adds the missing causal view. Each attempt can remain a child span of the same logical operation, with attributes such as retry.attempt, retry.remaining, and retry.reason. The resulting trace shows whether an operation failed because a dependency was unavailable, because the deadline expired, or because the shared budget prevented another attempt. That distinction is operationally important as budget exhaustion is often evidence that the system deliberately stopped adding pressure rather than evidence that the retry mechanism malfunctioned. Retry metrics also need to be interpreted alongside service saturation and rejection rates. A rising retry-success rate may initially indicate useful recovery from transient faults, but rising attempt volume combined with increasing backend saturation indicates a different condition. At that point, preserving capacity can be more valuable than pursuing another successful attempt. Google’s overload guidance explicitly recommends allowing failures to propagate when widespread backend overload makes additional retries unlikely to help. Conclusion Retries remain essential for transient failures, but retries without coordination can turn a partial outage into a traffic multiplier. Backoff, jitter, idempotency, deadlines, and server pushback address important parts of the problem that a retry budget adds the missing global constraint by limiting how much extra work one logical operation may create. Propagating that budget across service boundaries converts retry behavior from isolated local policy into distributed load control. The strongest resilience policy is therefore not “retry until success,” but “retry only while the failure is transient, the operation is safe, time remains, and the system can afford another attempt.”

By Uthej Mopathi CORE

The first warning sign wasn't an outage. It was a boring pull request. We changed one App Service setting. It was the sort of change that should have resulted in a small plan and a quick review. Instead, Terraform refreshed networking, private endpoints, DNS, Key Vaults, storage accounts, app services, and monitoring before showing what would actually change. Nothing was broken; that was the point. Terraform did exactly what it was designed to do: account for everything represented in state before calculating change. The problem was that our Terraform state had become a single, platform-sized boundary that every small change had to pass through, and one no team could fully own. If you have run a landing zone as a single Terraform configuration, you have probably had a version of that pull request. The instinct afterward is to blame size: the configuration has grown too large, so break it up. That instinct is wrong, or at least incomplete. Size is uncomfortable, but coupling is what actually hurts. Nothing in the change touched networking, DNS, or those key vaults. They were dragged into the plan because everything was bound together through one state. At first, that coupling just means slow plans and noisy reviews. Later, it raises a harder question: who actually owns this? Where the Coupling Shows Up Start with the plan. In a monolith, Terraform has to account for everything represented in the state before it can tell you what changed. You can target a single resource, but that is an escape hatch, not a way to run a platform. So the wait scales with the size of the estate, not your change. Both a one-line edit and a fifty-resource migration get stuck behind the same refresh before the diff appears. Provider upgrades show the same problem. A single root configuration pins one set of provider versions, so you cannot move networking to a newer azurerm version and leave everything else behind. Every upgrade becomes all-or-nothing, which means it keeps losing to smaller, safer priorities. Ours sat on azurerm 2.97 and only moved to the 4.x line once the upgrade could no longer be put off. The monolith had made the jump too big to schedule any sooner. The bigger concern is blast radius. One state file, one lock, one plan. A bad apply, a corrupted state, a destroy that catches more than you aimed at: whatever goes wrong can reach more of the platform than the change was ever meant to touch, because nothing in the layout is there to contain it. The dependency graph suffers too. Unrelated resources get sequenced together just because they share a graph. A network change might wait on unrelated compute, DNS on policy. The graph ends up reflecting accidental grouping rather than real dependencies. The result is clear. There is no small change. You cannot ship a DNS record or a new Key Vault without running the entire configuration through plan and apply. Every change is a platform change, carrying platform risk and requiring review, no matter how minor. These look like separate problems, but all come from the same design choice: too many unrelated concerns tied into one Terraform boundary. Where Coupling Becomes Ownership It is easy to call these operational annoyances: slow plans, awkward upgrades, risky applies, the tax you pay for a big configuration. But the same coupling appears in review and approval, where it stops being just an operational problem. Once too many concerns share the same state, pipeline, and approval path, the question is no longer only "how long did the plan take?" It becomes "who is accountable for the boundary this change is crossing?" Take private connectivity. A single private endpoint on Azure isn't handled by just one team. The application team owns the service behind it. The platform team manages the landing zone, subnet, and endpoint placement. Private DNS zones might be managed centrally or by another team. Security or governance may require the service to be private. How these map to teams varies, but in a monolith, everything ends up in the same state, pipeline, and plan. So "who owns this?" rarely has a clear answer. However you split teams, they are coupled through a single configuration that none can truly own. When the application team changes its service, the same config still carries platform connectivity and governance controls. You cannot draw ownership along your real organizational boundaries, because the code does not have them. Both slow plans and unclear ownership trace back to the same issue: shared concerns treated as if they belong to just one team. Figure 1: When Terraform boundaries stop matching ownership boundaries. The monolith gives Terraform one boundary. Organizations have several. The pain comes when small changes have to cross boundaries that no team fully owns. Reach for the Coupling, Not the Size The reflex now is to split the state and move on. But splitting a landing zone poorly can be worse than leaving it alone. If you split along the wrong lines, you trade one blast radius for tangled cross-state dependencies. You also lose the single plan that at least showed the whole graph in one place. For example, splitting private endpoints into one state and private DNS zones into another may look clean on paper. But if different teams deploy them without a clear agreement, every new endpoint becomes a coordination headache, not a smaller change. Moving files into separate folders does nothing if the same pipeline, credentials, and approval path still govern everything. Decomposition should follow actual coupling, not just line count. So the next question is not "how many states should we create?" It is "which boundaries are real enough for teams to own, deploy, and recover independently?" If your Terraform monolith hurts, do not start by counting files or resources. Look at what is actually being coupled. Slow plans and unclear ownership are both signs that your Terraform boundaries no longer match your real ownership boundaries.

Most modern applications do not function completely independently. For example, analytics, payment processing, user authentication, customer support, testing new features (experimentation), monitoring the app's performance, advertising, etc., are typically provided as third-party SDKs that enable those functions in your app. Using an SDK has its benefits; you don't have to build an entire piece of functionality yourself. When using an SDK, developers can download the software library, call the initialization method, then begin calling the API methods of the SDK to use its functionality. JavaScript import { analytics } from "third-party-sdk"; analytics.track("checkout_started", { productId: "123" }); In some cases, the amount of code required to add this type of functionality can be as little as a handful of lines of code. If the same functionality was built "from scratch", the time required could potentially be several weeks. However, with great convenience comes hidden complexity. The moment you allow a third-party SDK to run in your app, how well it performs and works (performance, reliability, security, and user experience) depends on what amounts to "someone else" doing something to your app. That is why third-party SDKs are important dependencies that affect how well your app will perform during production hours, instead of just being another library or module to include. SDKs Can Quietly Affect Performance The biggest reason front-end SDKs will show performance issues is that they typically run directly in your web browser. If you install a typical analytics SDK, it adds to your front-end bundle, it loads on page init, and then registers event handlers, makes requests over the internet, etc., as soon as there are interactions with your app. Although one SDK alone has little effect, if you use multiple SDKs for analytics, experimentation, customer service, session replay, ad tracking, and monitoring, your users will likely notice a difference. Therefore, teams need to evaluate whether individual "acceptable" SDK costs can compound into overall user-perceived degradation. In addition to measuring the cost of each SDK individually, teams need to look at the overall cost of loading the SDK(s), which can include: Bundled sizeTime to initializeNetwork requestsActivity on main threadOverall impact on Core Web Vitals This cost can be reduced by loading non-critical SDKs asynchronously or after the main application experience has loaded. A Third-Party Failure Can Become Your Failure Consider an application that will render the main page after initializing a recommendation SDK. JavaScript await recommendationSDK.initialize(); renderApplication(); If the third-party service has an issue, your application's overall appearance may slow or become unavailable, even if your backend is functioning properly. Thus creating unneeded coupling. Generally speaking, non-essential third-party services should be allowed to fail without affecting the primary user experience. For example, if you're unable to receive recommended products, you should still be able to browse through products; if analytics are failing, checkout should still function as normal; and if a support widget is unable to load, all other aspects of the webpage should continue to function normally. Applications should define clear fallback behavior for every external dependency. Timeouts are also important. Waiting indefinitely for a third-party service can turn a small external outage into a much larger product incident. SDK Updates Can Change Production Behavior Engineers typically spend considerable time evaluating large-scale framework updates; however, they may be less concerned about small third-party dependencies that make up much of their application codebase. This could potentially lead to issues. An SDK update can change how an application initializes, the format for making requests, which browsers an application supports, the default configuration, how data is stored, or how much JavaScript is downloaded during each session. Even if the public API hasn't changed, runtime behavior may still differ based on previous SDK versions. Therefore, dependency upgrades should follow standard engineering controls such as version pinning where applicable; automated testing; dependency review; and gradual deployment. The idea of automatically allowing all new SDK releases into production just because they have been classified as minor will create additional risk. Third-Party Code Expands the Security Boundary Every new SDK you add to your app will be a larger portion of all code making up the system. Browser apps make this especially important when SDKs can access page content, browser storage, cookies, user interaction, or even application data. You should know exactly which pieces of information will go out to an outside party. As an example, sending off an entire object to an analytics SDK could provide more information than was ever intended: JavaScript analytics.track("profile_updated", user); Some of the fields in the 'user' object might never have been intended for analytics. A safer approach is to explicitly select the information required for the event. JavaScript analytics.track("profile_updated", { accountType: user.accountType }); You'd be better off sending only the data you need for each specific event. The principle is simple: third-party integrations should receive only the data they actually need. SDKs Can Create Hidden Runtime Conflicts Not all third-party SDKs run independently. In addition to other actions such as modifying a browser's global objects, registering event handlers, intercepting web requests, and manipulating the DOM, third-party SDKs may also create new dependency conflicts that are incompatible with your current application code. Because of their nature, these issues can be difficult to reproduce because they typically depend on specific conditions (such as browser type, user environment, feature flags/feature toggle configuration, etc.) that cause them to occur only under very specific circumstances. Another reason why you should track correlation of failures to your integrations during production time is due to this. Also, when possible, initialize third-party SDKs in an isolated manner so that an error in initializing one service does not bring down the rest of the application. JavaScript try { await supportSDK.initialize(); } catch (error) { logError("Support SDK initialization failed", error); } If your optional service fails to start, then your application continues. Have an Exit Strategy The other, quite surprising, issue you might have when using an SDK is the difficulty in removing it. When there are numerous API calls in multiple layers of your app, making changes to which vendor you use as a service provider becomes extremely expensive. In this case, teams may want to develop their own internal abstraction layer on top of the external SDK. JavaScript tracking.track("checkout_started", data); Your application interacts with an internal 'tracking' interface, and then the internal tracking layer will interact with the external SDK. You still have a dependency on the vendor, but now all vendor-specific APIs are abstracted out of your codebase. Testing also becomes simpler, and you can easily add validation, filtering, error handling, and fallback logic. Monitor SDKs Like Production Dependencies Integrations with third-party tools should look similar in your observability dashboard as your internal services. Understanding when/why an SDK will fail; how long initialization takes; whether requests are timing out; and which specific integration(s) cause frontend errors/performance regressions helps teams understand when they have a problem. It is also beneficial to understand what feature of your application depends on each provider. This type of information greatly assists during an incident by providing a clear yes/no answer to an important question: Can I disable this integration and still run my core product? For critical integrations, the answer should already exist before an outage occurs. Conclusion Third-party SDKs are useful because they enable engineering teams to get things done in less time than would be required if the team had to build capability again, which has been developed by others who specialize in that area of development. However, when you add a new SDK to your project, you've added a new production dependency. This dependency can negatively affect your application's performance, reveal information about your application, break at unpredictable times, change with each upgrade, and ultimately become very hard to remove as it spreads across your codebase. Our objective is not to eliminate third-party SDKs. Our goal is to intentionally incorporate third-party SDKs into the project. Track how much performance is affected, track what amount of data is transmitted back to the provider, prevent failures from spreading through isolation, maintain control over upgrades, track the use of the service, and do everything possible to prevent tightly coupling core functionality to a service that the application does not control. A third-party SDK may take only a few minutes to install, but its production impact can last for years.

Most write-ups on building an MCP server focus on the protocol itself: defining tools, handling requests, wiring up a client. That part is genuinely straightforward. What gets skipped over far more often is what changes when the tool you are exposing operates on files rather than returning data. File processing introduces a specific set of security and reliability problems that a typical read-only API does not have to think about, and getting them wrong is easy to miss until something goes badly. This is a rundown of the decisions that mattered most while building an MCP server that exposes document processing tools, merge, convert, OCR, and similar operations, and why a few of the obvious approaches turned out to be the wrong ones. Why File-Processing Tools Are a Different Security Case A typical MCP tool that queries a database or calls a read-only API has a bounded, predictable attack surface. A tool that accepts a file, or worse, a URL pointing to a file, and processes it does not. Two problems show up immediately that a simpler API rarely has to deal with. First, any tool parameter that accepts a URL is a potential SSRF vector. An MCP client could be tricked, directly or through a compromised upstream model response, into passing a URL pointing at an internal service, a cloud metadata endpoint, or an otherwise unreachable internal address. If the server naively fetches whatever URL it is given, that request happens from inside your infrastructure with whatever network access your server has. Treating every incoming URL as untrusted input, resolving it before fetching, and explicitly blocking private IP ranges and metadata endpoints is not optional for a tool like this, it is baseline. Second, file processing is expensive relative to a typical API call. Merging PDFs, running OCR, converting between formats, these all consume real CPU and memory per request in a way that a database lookup does not. That changes how rate limiting needs to work, which is worth its own section below. Auth: Why API Keys Plus JWT, Not Just One or the Other A single long-lived API key is simple to implement and simple to leak. Once issued, it is valid until manually revoked, and if a key ends up in a log file, a committed config, or a client-side integration by accident, there is no time-boxing to limit the damage. The approach that held up better in practice: bcrypt-hashed API keys for the initial authentication step, then a short-lived JWT issued from that exchange for the actual session. The API key never gets passed around on every request, only at the start, and it is never stored in plaintext server-side, so a database compromise does not directly expose usable credentials. The JWT that follows has a real expiry, which bounds how long a leaked token stays useful and gives you a natural mechanism for revocation without needing to invalidate the underlying key. This is not a novel pattern. It is standard practice in plenty of API design. The point worth making is that it is easy to skip for an MCP server specifically, because the tooling and examples in most MCP documentation default to a single static key for simplicity, and that default quietly becomes the shipped implementation if nobody revisits it. Idempotency: The Requirement Everyone Forgets Until It Bites MCP clients retry. Network hiccups, timeouts, a model deciding to re-invoke a tool call, all of these mean the same logical request can arrive at your server more than once. For a read-only tool, that is harmless, you just return the same data twice. For a tool that processes and charges against a file, a duplicate request means duplicate processing, potentially duplicate output files, and depending on your billing model, duplicate charges for a single user action. The fix is an idempotency key attached to each request, generated client-side and checked server-side before any processing begins. If a request with a given idempotency key has already been handled, the server returns the cached result rather than reprocessing. This sounds obvious once stated, but it is very easy to build a working MCP server that passes every test in development, where retries are rare, and only discover the gap once it is handling real, occasionally flaky client connections in production. Rate Limiting That Doesn't Punish Legitimate Use Because file processing is CPU and memory intensive per request, generic per-minute rate limits borrowed from a typical REST API tend to either allow abuse or block legitimate batch workflows, and it is hard to tune a single number that avoids both. Someone processing twenty files in a genuine batch workflow looks identical, from a naive rate limiter's perspective, to a script hammering the endpoint. What worked better was tracking limits per API key with enough granularity to distinguish sustained high-frequency abuse from a legitimate burst of activity, rather than a single flat request-per-minute ceiling applied uniformly. This is a harder problem to get exactly right than it sounds, and it is one area worth revisiting periodically as real usage patterns become clearer, rather than treating the initial configuration as final. Audit Logging as a Design Decision, Not an Afterthought It is tempting to treat logging as something you bolt on once a security question actually comes up. For a tool that processes user files, that is backwards. Knowing which API key touched which file, when, and what operation was performed needs to exist from the first deployment, not added retroactively after an incident makes it obvious it should have been there. This matters for debugging as much as for security, since a surprising number of support questions end up being answerable directly from audit logs rather than requiring back-and-forth with the user. What Would Have Saved Time in Hindsight Two things, if starting over. The first is deciding on the auth pattern, API key exchange plus short-lived JWT versus a single static key, before writing a single tool handler, rather than starting with the simpler static key for speed and migrating later. The migration is not hard technically, but it touches every existing integration and every piece of client documentation, so the cost of delaying the decision is mostly organizational rather than technical. The second is building the idempotency check in from the first tool, rather than adding it once a duplicate-processing report surfaces. It is a small amount of code, a lookup and a cache write around the start of request handling, but retrofitting it means auditing every existing tool for where duplicate execution would actually cause a visible problem versus where it is harmless, which takes longer than just building it in from the start would have. Putting It Together None of these individually are exotic ideas. Short-lived tokens over static keys, treating URL inputs as untrusted, idempotency keys for retryable operations, audit logging from day one, all of these are well-understood patterns in API design generally. What is specific to building an MCP server for file processing is that the combination matters more here than it does for a typical read-only integration, because the failure modes are more expensive: a duplicated file, a leaked key with no expiry, an SSRF hole reachable through a tool parameter, or an untracked operation on a user's document. If you are building or evaluating an MCP server that touches files rather than just data, these are the questions worth asking early, before the first real client connects to it, rather than after.

A few months ago, I watched a senior engineer spend forty-five minutes reviewing a single pull request — a PR that an AI assistant had generated in under two minutes. The code looked clean. The tests passed. But she kept cross-referencing an incident postmortem from eight months earlier, muttering something about retry amplification. She caught a real production risk. The AI reviewer had flagged nothing. That moment stuck with me. We've spent years optimizing how fast we can write code. But we haven't seriously reckoned with what happens when review can't keep up. The Bottleneck Has Shifted A single engineer with AI assistance can now produce hundreds of lines of code, large refactors, infrastructure changes, and test suites — all within minutes. Review complexity, however, grows exponentially with change size and system interdependency. The core problem is no longer "Can AI write code?" It's "Can humans reliably validate what AI wrote?" Code generation speed increases. Human cognitive review capacity stays flat. That imbalance is quietly accumulating risk in engineering organizations everywhere. Why Current AI Reviewers Fall Short Most AI PR review systems today operate on static diffs, syntax-level reasoning, and shallow best-practice detection. They produce comments like: "Potential null pointer.""Consider renaming this variable.""Possible optimization opportunity." Occasionally useful. Rarely sufficient for production-critical systems. The structural problem is that these tools treat PR review as a language problem instead of a systems reasoning problem. They assume software correctness is inferable from local code semantics alone. In reality, production safety emerges from interactions between architecture, runtime behavior, operational history, and organizational context. The Shallow Review Problem in Practice Here's a concrete example. An AI assistant generates this database query optimization: Python # AI-optimized version def get_user_orders(user_id): return db.query(""" SELECT o.*, p.*, i.* FROM orders o JOIN payments p ON o.id = p.order_id JOIN items i ON o.id = i.order_id WHERE o.user_id = ? """, user_id) Typical AI reviewer comment: "Query optimized with JOIN to reduce round trips." What a senior engineer sees: "This will cause a Cartesian explosion. The orders table has 50M rows, items averages 8 per order. This returns 400M+ rows for power users. We had a nearly identical incident (INC-287) that took down the read replica. Needs pagination and selective columns." The difference isn't token count or model size. It's operational memory and causal reasoning. The Real Challenge Is Not Context Windows Many people assume the fix is larger context windows. Feed the model the whole repo, and it'll review like a senior engineer. But experienced engineers don't review code by loading entire systems into working memory. They use abstraction, selective attention, and compressed mental models. A senior engineer reviewing a Kafka retry change doesn't reread the entire messaging subsystem — they remember prior incidents, retry amplification risks, and historical outages. That's cognitive compression, not token recall. Modern LLMs are exceptional at syntax fluency, pattern completion, and probabilistic association — what you might call token intelligence. But effective PR review requires something deeper: causal reasoning, architectural abstraction, operational memory, risk forecasting. Call it cognitive intelligence — persistent contextual reasoning grounded in operational history and causality. The distinction matters because it changes what we need to build. What a Cognitive Review Architecture Looks Like Instead of: Plain Text Large Prompt + Large LLM → Review We need: Plain Text Structured Memory + Semantic Retrieval + Runtime Context + Specialized Review Agents + Reasoning Layer + LLM → Review The LLM should not be the memory. It should be the reasoning interface over structured engineering knowledge. Intent Reconstruction Before reviewing code, the system needs to understand why the change exists. Business intent, bug root cause, architectural motivation. Inputs include Jira tickets, PR descriptions, ADRs, incident reports, and commit timelines. Without intent, review quality stays shallow regardless of model size. Engineering Knowledge Graphs Human reviewers carry organizational memory: fragile services, latency-sensitive paths, scaling bottlenecks, previous outages, dangerous dependencies. AI reviewers need persistent semantic memory systems encoding the same — service relationships, API contracts, operational metadata, incident history, ownership boundaries. This creates an engineering cognition layer far richer than raw repository context. Multi-Agent Review Systems A single reviewer model is insufficient. Future systems will consist of specialized agents working together: Architecture Reviewer – dependency boundaries, coupling risk, architectural driftReliability Reviewer – retries, backpressure, idempotency, failover behaviorSecurity Reviewer – injection risks, auth issues, secret exposurePerformance Reviewer – memory growth, query amplification, scaling regressionsHistorical Regression Reviewer – correlation with past outages, postmortems, incident fingerprints This begins to approximate how experienced engineering organizations actually review software. Runtime-Aware Review Static analysis alone misses emergent runtime behavior. Future cognitive review systems will integrate observability telemetry, tracing data, production metrics, and traffic patterns. Compare these two responses to a retry configuration change: Traditional AI reviewer: "Code follows retry best practices." Cognitive AI reviewer with operational memory: "HIGH RISK: Similar retry configuration caused incident on 2023-09-15. This service processes 2M messages/hour at peak. 10 retries with exponential backoff = up to 17 minutes per message. Previous incident resulted in 8M message consumer lag and cascading downstream failures. Recommend: max 3 retries, circuit breaker, dead letter queue, idempotency check before db.save(). See ADR-089." That is a fundamentally different class of intelligence — and a fundamentally different class of safety. Engineering Memory Is the Missing Piece One of the biggest gaps in current AI systems is durable operational memory. Experienced engineers develop intuition through outages, failed deployments, debugging sessions, and production emergencies. These experiences become compressed heuristics: "This retry increase feels dangerous" — not because of syntax, but because of remembered causal relationships. Replicating this requires episodic memory systems, incident-aware reasoning, and causal knowledge graphs. Much of this mirrors practices long established in Site Reliability Engineering, where institutional learning from incidents is treated as critical infrastructure. Incident postmortems aren't just documentation — they're organizational immune system responses. Getting AI systems to genuinely learn from incidents rather than just pattern-match against them remains one of the harder open problems in this space. What Teams Can Do Today Fully cognitive review systems don't exist yet. But organizations can meaningfully improve AI-assisted review quality right now: Capture architectural knowledge in machine-readable form. Service boundaries, retry policies, timeout configurations, scaling assumptions — not just in wikis, but in structured formats AI systems can query.Link PRs explicitly to incident history. Build connections between code changes and the incidents they caused or prevented. This is organizational memory that AI systems can leverage today.Tag services with operational metadata. Criticality tier, traffic patterns, known failure modes, blast radius. Treat repositories as systems, not just files.Integrate observability into review pipelines. Connect production metrics and tracing data to code review. Runtime context dramatically improves review quality.Prioritize high-signal AI feedback. Review fatigue from noisy, low-signal comments is a real trust problem. Focus AI comments on incident-correlated patterns, architectural violations, and operational risks. The Trust Calibration Problem One concern I keep coming back to: bad AI reviewers are dangerous not because they miss things, but because they sound confident while missing things. They reduce human vigilance through automation bias. They generate fatigue through noise. They normalize shallow approval. Future cognitive review systems need to be not just more accurate, but properly calibrated — knowing when they lack sufficient context and escalating accordingly. An AI reviewer should be able to say: "I may not have enough confidence to validate this safely." That self-awareness may matter more than raw capability. The Road Ahead The next era of AI software engineering will not be defined by who generates the most code. It will be defined by trust, reasoning quality, and operational awareness. The future belongs to systems capable of understanding not just what changed — but why it changed, what it affects, and whether it's safe. That's the difference between code generation and engineering intelligence. And honestly, solving it seems harder and more interesting than anything we've built so far. Key Takeaways The bottleneck has shifted from code generation to code review and validation.Larger context windows alone won't bridge token intelligence and cognitive intelligence.Human-like review requires structured memory, causal reasoning, and operational awareness.Multi-agent architectures with specialized reviewers mirror how engineering teams actually work.Runtime-aware systems integrating production telemetry represent the next frontier.Engineering memory — learning from incidents — is critical for trust and safety.Teams can start today by capturing architectural knowledge and linking incidents to code changes. References Vaswani, A., et al. (2017). "Attention Is All You Need." NeurIPS.Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS.Shinn, N., et al. (2023). "Reflexion: Language Agents with Verbal Reinforcement Learning." arXiv.Beyer, B., et al. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media.Allspaw, J. (2012). "Blameless PostMortems and a Just Culture." Etsy Engineering.

When a QA team is asked, "How confident are we in this AI system?" the instinctive answer is to write more test cases. If 100 test cases gave us some confidence, surely 1,000 will give us ten times more. This instinct is deeply wired into traditional software testing, where every additional test case can, in principle, catch a bug the others missed. For AI systems, this instinct is not just inefficient — it is often mathematically wrong. Adding more test cases the wrong way can leave you with less real confidence than a much smaller, carefully sampled set. Understanding why requires looking at what a test case is actually measuring when the system under test is probabilistic rather than deterministic. This article walks through the statistical reasoning that should drive AI test suite design, and shows why smart sampling — not raw volume — is what actually buys you confidence in an AI system's behavior. What a Test Case Measures for Deterministic Software For traditional software, a test case answers a yes-or-no question: given this input, does the system produce the expected output? Each test case is independent evidence about a specific code path. Adding a new test case that exercises a previously untested branch genuinely adds new information, because the system's behavior on that branch was, until that test ran, completely unknown. This is why test coverage metrics — line coverage, branch coverage — are meaningful for deterministic software. Coverage tells you what fraction of the system's possible behavior has been directly observed at least once. Why the Same Logic Breaks for AI Systems An AI system does not have a fixed, enumerable set of behaviors the way a codebase has a fixed set of branches. Its behavior is a distribution — a probability of producing each possible output for a given input, and often a probability that itself shifts slightly across runs, contexts, and time. When you write a single test case for an AI system — one prompt, one expected answer — you are not observing "the" behavior of the system for that input. You are observing one sample drawn from a distribution of possible behaviors. Running that exact same test case again might draw a different sample from the same distribution. This changes the statistical meaning of a test case entirely. Here's where everything changes: a single AI test case is no longer a definitive answer — it is just one observation from a much larger behavioral distribution. In deterministic testing, one test case answers one question conclusively. In AI testing, one test case gives you one data point toward estimating a probability — and a single data point tells you almost nothing about a probability distribution. The Confidence Interval Problem Consider a concrete example. You want to know: does this AI-powered customer support assistant give a policy-compliant answer to refund questions at least 95% of the time? You write 20 refund-related test cases, run them, and 19 pass. That is a 95% pass rate — right at your target. Statistically, this result is far weaker evidence than it feels. With only 20 samples, the 95% confidence interval around your observed 95% pass rate is wide — the true underlying success rate could plausibly be anywhere from roughly 75% to 100%. Twenty test cases have not told you the system meets your bar. They have told you the system's true performance is consistent with a wide range that happens to include your bar. To narrow that confidence interval meaningfully — to actually distinguish a system that is truly 95% reliable from one that is truly 85% reliable — requires a sample size in the hundreds, not tens, for that specific scenario category. In practice, engineering teams commonly use 95% confidence when estimating AI system reliability, although higher confidence levels may be appropriate for safety-critical applications. This is the first place where "more test cases" and "smart sampling" start to diverge: you need enough samples per scenario category to say anything statistically meaningful about that category, and no amount of test cases in unrelated categories substitutes for that. There's a simple piece of math behind this that's worth internalizing, because it explains why the problem doesn't go away just by writing more tests. The width of a confidence interval shrinks in proportion to the square root of your sample size, not in direct proportion to it: Plain Text Confidence Interval width ∝ 1 / √n Doubling your sample size does not double your confidence — it only shrinks your uncertainty by about 30%. Quadrupling it cuts uncertainty in half. This is the quiet reason volume-based testing feels productive while delivering diminishing returns: the tenth test case in a category buys you real statistical ground, but the two-hundredth buys you very little compared to what it costs to write and run. What This Looks Like in Practice The contrast between the two approaches is easiest to see side by side. Plain Text TRADITIONAL TESTING 1,000 test cases spread evenly │ ▼ "Coverage" (looks thorough, says little about any one scenario) SMART SAMPLING Scenario A (high risk) → 120 samples Scenario B (high risk) → 80 samples Scenario C (medium risk) → 60 samples Scenario D (low risk) → 20 samples │ ▼ Confidence (narrow intervals where it matters most) The traditional approach optimizes for a number that looks reassuring on a dashboard. The sampling approach optimizes for statistical confidence where business risk is highest — and is honest about where confidence is intentionally looser. Where Volume Actually Hurts Here is the counterintuitive part. Teams often respond to this problem by writing more test cases — but they add them across many different scenario categories rather than deepening any single one. The result is a suite with 500 test cases, twenty scenario categories, and roughly 25 samples per category — still not enough to draw a confident conclusion about any individual category, while creating the appearance of a large, thorough suite. This is worse than it sounds, because a large test suite carries real costs. It takes longer to run, which slows down CI/CD feedback loops. It takes longer to maintain, since ground-truth answers for AI systems need periodic review as policies and knowledge bases change. And critically, it creates false confidence — a dashboard showing "500 tests, 98% pass rate" reads as strong evidence to a stakeholder, when the underlying statistics may not support that read at all for any specific scenario that stakeholder actually cares about. What Smart Sampling Looks Like Instead Smart sampling starts from a different question: not "how many test cases can we write," but "what decision do we need statistical confidence about, and how many samples does that decision actually require." The first step is defining scenario categories that map to real business risk — refund policy questions, account security questions, product availability questions — rather than categories that map to convenient technical groupings like "single-turn queries" versus "multi-turn queries." Risk-aligned categories are what stakeholders actually need confidence about. The second step is stratified sampling within each category: generating semantically varied inputs that probe the same underlying scenario from different angles — different phrasings, different levels of ambiguity, different amounts of context — rather than many near-duplicate test cases that differ only in superficial wording. Ten semantically diverse samples of a scenario carry more statistical information than fifty near-identical restatements of the same question, because the near-identical restatements are highly correlated with each other and do not independently sample the underlying distribution. The third step is allocating sample size deliberately by risk. A scenario category with high business consequence — anything touching financial transactions, medical guidance, or legal disclosures — warrants a large enough sample to produce a narrow confidence interval, potentially hundreds of cases. A low-consequence category, such as a cosmetic formatting preference, can be validated adequately with a much smaller sample. Treating every category with the same sample size wastes effort on low-risk scenarios while under-sampling high-risk ones. The fourth step is repeated sampling over time rather than only at initial test design. Because AI system behavior can drift, a sample that gave a narrow, confident interval six months ago does not guarantee the same interval holds today. Smart sampling treats the sample size and scenario allocation as something to periodically re-justify against current production data, not a decision made once and left untouched. The same sampling principles apply directly to retrieval-augmented generation (RAG) systems, which now sit behind most enterprise AI assistants. In a RAG pipeline, response correctness depends on both the generative model and the quality of what gets retrieved, so a scenario category isn't fully sampled unless it captures variation in retrieval outcomes too — cases where the right document is retrieved, cases where a close-but-wrong document is retrieved, and cases where retrieval comes back empty. Treating "RAG testing" as one category instead of a set of retrieval-quality-weighted sub-scenarios is one of the most common places teams under-sample without realizing it. The same statistical reasoning also applies to autonomous AI agents. Because agents make sequential decisions, validation must consider the probability distribution across complete workflows rather than evaluating each individual step in isolation. A Practical Illustration Suppose an enterprise AI validation team has a fixed budget of 300 test executions per CI/CD run — a real constraint, since each execution costs inference time and, for hosted models, direct API cost. A volume-first approach might spread these 300 across 30 scenario categories, roughly 10 samples each — statistically too thin to draw confident conclusions about any single category. A risk-based sampling approach might instead allocate 80 samples to the three categories touching financial and account-security actions, 40 samples each to five categories with moderate business consequence, and 10 samples each to the remaining ten low-consequence categories. The total sample budget is unchanged at 300, but the confidence intervals for the categories that actually matter to the business are now meaningfully narrower, while low-risk categories still receive baseline coverage rather than none at all. This is the essence of smart sampling: the same testing budget, reallocated according to statistical need and business risk, rather than spread evenly across categories regardless of consequence. The Broader Principle The deeper lesson here extends beyond test case counting. AI validation, as a discipline, has to import statistical thinking that traditional software testing rarely required, because traditional testing dealt with deterministic systems where a single well-chosen test case could conclusively answer a question. AI systems require thinking in terms of distributions, confidence intervals, and sample sizes — the vocabulary of applied statistics rather than the vocabulary of test coverage. Teams that continue to measure AI test suite quality purely by test case count will keep producing dashboards that look reassuring and mean less than they appear to. Teams that shift to measuring statistical confidence per risk-weighted scenario category will produce smaller, faster, and — despite being smaller — genuinely more informative test suites. AI systems are not validated by counting test cases. They are validated by measuring uncertainty. The future of AI quality engineering belongs to teams that measure confidence — not coverage. As enterprise AI systems become increasingly autonomous, statistical validation will become as fundamental to software quality engineering as code coverage is today.

End-to-end tests are crucial components of modern CI/CD pipelines, helping to ensure that changes do not cause regressions before deployment. Playwright offers network inspection, automatic waiting, browser isolation, tracing and cross-browser execution. While useful for improving automation, these features cannot replace the engineering work needed to develop, identify and monitor effective tests. Requirements, source code, page structure, APIs, selectors, authentication rules, test data can all change at the same time in a sprint. The reason for a Playwright test failure can include any of the following: regression in the application, the application's outdated locator, outdated test data, inability to reach the application, or instability of the timing. This test-case checkout sprint demonstrates the addressing of that problem by specialized agents, graph retrieval-augmented generation, execution evidence, and bounded repair, without obscuring real regressions. The Test-Case Sprint An e-commerce team plans a checkout update with three features: Promotional-code supportAutomatic refresh of expired authentication tokensA redesigned order summary The sprint defines seven acceptance criteria: Customers can apply valid promotional codes.Discounts appear in the order summary.Invalid or expired codes produce clear errors.Totals include discounts, tax, and shipping.Authentication refresh preserves the shopping cart.An order can be submitted only once.Successful submission displays an order identifier. The implementation changes several artifacts: src/pages/CheckoutPage.tsxsrc/components/PromoCodeForm.tsxsrc/components/OrderSummary.tsxsrc/services/auth.tssrc/services/checkout.tstests/checkout.spec.tsplaywright.config.ts The interface team also changes the test identifier of "place-order-button" to "submit-order." Post-change tests fail in 14 locations in Chromium, Firefox, and WebKit. With a conventional pipeline, failures are reported, but it is not possible to reliably identify which tests are to be repaired and which failures are product defects. Why One General-Purpose Agent Is Not Enough In the testing workflow, there are multiple tasks that call for various types of reasoning. Requirement analysis identifies the behavior(s) that need to be validated. Test planning maps behavior to preconditions, data, actions, and assertions. To generate code, one needs to know Playwright. Controlled browsers and artifact collection for execution. Traces, screenshots, network events, and repository changes are used for diagnosis [1]. Conservative decisions for repair must be made, and the original requirement must be retained. Assigning every responsibility to one unrestricted agent introduces several risks: Irrelevant context moves between stages.Large prompts increase cost.Failed actions may be repeated.Obsolete selectors may be regenerated.Application regressions may be misclassified as test defects.Repairs may weaken assertions merely to produce passing tests. This research separates these responsibilities into requirement, planning, generation, execution, diagnosis, and repair experts. Each expert operates under a specific tool policy, input schema, output schema, and validation contract [2]. The architecture does not require six different foundation models. Multiple roles can use specialized configurations of the same model. Building a Software Knowledge Graph Classical RAG usually retrieves independent text chunks through semantic similarity. Software evidence is relational. A uniform requirement can be implemented across several files. A page renders multiple components, and these components are comprised of interface elements. A test covers singular or multiple criteria of acceptance This study represents these relationships in a project knowledge graph. Possible node types include: User storiesAcceptance criteriaCommitsSource filesClasses and functionsPagesComponentsInterface elementsTestsExecution tracesDefects Representative relationships include: implementscontainsrendersdepends_onchanged_bycoversfailed_infixed_by A useful retrieval path for promotional-code testing could be: Apply valid promotional codeCheckoutPagePromoCodeFormPOST /api/promotions/validatePromotional-code inputExisting checkout test Another path could connect order submission with a historical defect: Checkout servicePOST /api/ordersSubmit-order buttonCheckout submission testDuplicate-order defect Semantic retrieval finds the artifacts whose language is related to the selected language [3]. Dependency paths are the basis for a structured retrieval. The recency feature avoids stale results from stale selectors and provenance is used to point to the repository revision and source of each artifact. Graph traversal does not gather irrelevant evidence, thanks to depth limits and token budgets. Routing the Task to Specialists Calling every expert for every task would increase latency and token cost. This study uses sparse routing to activate only the required specialists. The router considers: Requirement complexityChanged-file distributionHistorical defectsExecution riskAvailable evidenceRemaining pipeline budget Generation and execution experts may only be needed for minor text changes [4]. The checkout sprint impacts the following items: authentication, payment behavior, order totals and duplicate submission protection. Therefore, all six roles are activated in the router and the human approval for the repair if a high risk is involved is necessary. Each criterion must be broken down into observable behaviors by the requirements expert. Often, for security reasons and better planning, the planner creates scenarios of positive, negative and of retrieval. A valid promotional-code scenario might contain: Precondition: Authenticated customer with products in the cart Test data: Consider a promotional code, ‘SAVE10’ Actions: Open checkoutEnter SAVE10Apply the code Expected result: The order summary displays the discountThe final total includes the discount, tax, and shipping Cleanup: Remove the test order and release the promotional code Generating the Playwright Test For the Playwright code to be created, the generation expert uses the validated plan and retrieved graph evidence [5]. Locator selection follows a stability hierarchy: Accessible roleLabelStable visible textApproved test identifierCSS selector when stronger options are unavailable A generated test could look like this: JavaScript import { test, expect } from "@playwright/test"; test("applies a valid promotional code", async ({ page }) => { await page.goto("/checkout"); await page.getByLabel("Promotional code").fill("SAVE10"); await page.getByRole("button", { name: "Apply code" }).click(); await expect(page.getByTestId("discount-line")) .toContainText("SAVE10"); await expect(page.getByTestId("discount-amount")) .toHaveText("-$10.00"); await expect(page.getByTestId("order-total")) .toHaveText("$102.40"); }); The test validates a business outcome instead of checking only that the page remains visible. Static validation rejects: Fixed delaysUnsupported selectorsHidden execution-order dependenciesMissing importsWeak assertionsTests without acceptance-criterion mappings Accepted tests run in isolated browser contexts with deterministic setup and cleanup. Collecting Execution Evidence The execution expert runs the tests and collects: Playwright tracesScreenshotsVideosConsole messagesNetwork requests and responsesBrowser errorsTiming dataRetry outcomes Clerical failure is a way of separating between determinate failures and fragile and unpredictable behavior of the clerical. In like manner, cross browser execution will determine browser-specific parity but won't consider each browser difference to be an application defect. The 14 test-case failures form three groups. Case 1: Obsolete Test Identifier Seven tests fail with this error: Plain Text Timeout waiting for [data-testid="place-order-button"] Based on the traces, it is established that that the checkout page loads successfully. Graph retrieval connects the button component and acceptance criterion to the new `submit-order` identifier. The diagnosis expert classifies the failures as test defects. The repair expert proposes the smallest supported patch: JavaScript // Previous locator page.getByTestId("place-order-button"); // Repaired locator page.getByTestId("submit-order"); The patch changes no actions or assertions. It passes static validation, targeted execution, and the relevant checkout regression subset. Case 2: Authentication Regression Four tests fail after an authentication token expires. Network evidence shows: HTTP POST /api/auth/refresh -> 200 POST /api/orders -> 401 Refresh request is successful; order request still uses the expired token. This is possible because there is a correlation between the failure and the changes to the auth.ts and checkout.ts. An application defect is diagnosed by the diagnosis expert. No permission is given to the repair agent to change the tests. Relaxing the test or adding retries to it would hide the regression. On the other hand, the pipeline produces a defect report including the requirement, the trace, the events of the network, the files affected, as well as the version of the repository. Case 3: WebKit Timing Instability Some WebKit runs crash sporadically as the order summary is recalculating. No code changes for repeated runs yields passing and failing result. The original test checks the total immediately: JavaScript await applyButton.click(); await expect(orderTotal).toHaveText("$102.40"); The trace shows a visible recalculation state. The diagnosis expert classifies the failure as flaky synchronization. The repair waits for an observable state transition: JavaScript await applyButton.click(); await expect(page.getByTestId("summary-status")) .toHaveText("Updated"); await expect(page.getByTestId("order-total")) .toHaveText("$102.40"); This repair avoids a fixed timeout. Five repeated WebKit runs and the relevant regression subset must pass before acceptance. Bounded Repair and Governance Autonomous repair should never operate without limits. A patch is eligible only when: Diagnostic confidence exceeds the configured threshold.Evidence identifies a specific test defect.The acceptance-criterion mapping remains unchanged.Static validation passes.The targeted test passes.A relevant regression subset passes.The repair-attempt limit has not been exceeded. Application defects, ambiguous failures, payment changes, authorization rules, and security-sensitive workflows require human review. Every decision should record: Selected expertsRetrieved graph nodesModel outputsTool callsExecution evidenceFailure classificationProposed patchValidation outcomeHuman approval status This audit trail makes the workflow reproducible and reviewable. Test-Case Outcome After diagnosis: Seven failures are classified as locator-related test defects.Four failures are classified as application regressions.Three failures are classified as flaky synchronization problems.Ten tests receive validated repairs.Four application failures remain visible for developers.No assertions are removed or weakened.Every repair remains connected to its acceptance criterion. These values are illustrative and are not production measurements. Practical CI/CD Considerations A production implementation should: Build the graph incrementally.Cache reusable retrieval results.Pin browser, repository, model, prompt, and policy versions.Restrict expert tool access.Define token, execution-time, and repair budgets.Limit browser concurrency.Redact secrets before model access.Store generated tests and traces as reviewable artifacts.Require approval for high-risk repairs.Report assisted and autonomous outcomes separately. Throughout the procedure, the quality of the graph persists as a major dependency. A primary factor that misleads retrieval is the presence of non-uniform and obsolete relationships. Model variability, routing errors, test-data instability, integration cost, and privacy requirements also limit adoption. Conclusion Refresh request fails, but order request continues to use the previous (expired) token. This works because the failure is related to changes in the `auth.ts' and checkout.ts' files. An application defect is diagnosed by the diagnosis expert. Permits repair agent to change the tests (no). It would make the assertion less strict and add retries, thus burying the regression. On the other hand, the pipeline will create a defect report, which will contain the requirement of the trace, the events in the network, the affected files, and the repository revision. References J. Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” NeurIPS, 2024. C. S. Xia et al., “Agentless: Demystifying LLM-Based Software Engineering Agents,” 2024. D. Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” 2024. J. A. Pizzorno and E. D. Berger, “CoverUp: Effective High-Coverage Test Generation for Python,” 2025. S. Gu et al., “TestART: Improving LLM-Based Unit Testing via Co-Evolution of Automated Generation and Repair Iteration,” 2024.

Enterprise applications commonly face multiple data challenges. Some data requires transactional integrity and relationships, while other data prioritizes fast, predictable access. Sessions, counters, rate limits, temporary state, often-accessed objects, and coordination data may not benefit from the complexity of a relational model. In these cases, a key-value database's simplicity becomes an architectural advantage. This simplicity is especially valuable in distributed and cloud-native systems, where latency, throughput, plus scalability directly shape user experience and infrastructure costs. A key-value database offers a focused approach: identify data by a key and retrieve or update it efficiently. The challenge is selecting a technology that delivers this performance while meeting the operational maturity, ecosystem support, and governance standards required for enterprise applications. Valkey meets these needs successfully. Originating from the Redis OSS lineage and developed as a vendor-neutral open-source project under the Linux Foundation, Valkey delivers a high-performance key-value platform suitable for caching, application state, messaging, and primary data storage. Beyond being another database option, it lets organizations explore how key-value persistence fits into modern enterprise architecture and lets Java applications use its benefits without tightly coupling to a specific datastore. Why Key-Value Databases Matter Key-value databases use a simple data model in which each unique key identifies a value. This simplicity is effective when applications can directly locate the required data. By enabling direct reads and writes, key-value databases typically deliver low latency, high throughput, and a horizontally scalable operational model. In enterprise systems, this model suits scenarios such as distributed sessions, caching, counters, rate limiting, feature flags, shopping carts, temporary workflow state, idempotency keys, leaderboards, and frequently accessed application data. These workloads prioritize fast access by identifier over joins, ad hoc queries, or complex relational constraints. The main architectural advantage of key-value databases is their specialization for specific access patterns, rather than universal speed or simplicity. When the primary requirement is to retrieve the current value for a given key, adding a more complex persistence model can introduce unnecessary overhead. As part of a polyglot persistence strategy, key-value stores enable architects to align the database model with the workload, rather than forcing all workloads into a single database. Putting Valkey Into Practice With Jakarta NoSQL A key advantage of using Valkey in enterprise Java is that it does not require a new programming model. With Jakarta NoSQL and Eclipse JNoSQL, Valkey serves as another key-value implementation behind a consistent API and mapping model. Domain annotations remain unchanged, so switching between key-value databases usually involves only updating the driver and its configuration, not rewriting the application. This abstraction is valuable architecturally. The application relies on the Jakarta NoSQL contract, while Eclipse JNoSQL manages integration with the database. Although database-specific features may introduce some coupling, applications that use the portable API can switch key-value implementations with minimal impact. For this article, we will use a simple Java SE example. This persistence layer can later support a REST API, messaging consumer, scheduled process, or other enterprise architecture without altering the core database interaction. Starting Valkey The first step is to make a Valkey instance available. Docker provides a convenient way to start one locally: Shell docker run --name valkey-instance \ -p 6379:6379 \ -d valkey/valkey:latest With Valkey running, add the Eclipse JNoSQL Valkey driver to the Jakarta NoSQL infrastructure, which includes CDI, Eclipse MicroProfile Config, and Jakarta JSON Processing. XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-valkey</artifactId> <version>${jnosql.version}</version> </dependency> Configure the connection externally: Properties files jnosql.keyvalue.database=developers jnosql.valkey.port=6379 jnosql.valkey.host=localhost Since Eclipse JNoSQL integrates with Eclipse MicroProfile Config, you do not need to hard-code these values. They can be provided through configuration sources such as environment variables, in line with the Twelve-Factor App methodology. Mapping an Entity The mapping model for a key-value database is intentionally simple. Identify the class as an entity and specify the field that represents its key: Java @Entity public class User { @Id private String userName; private String name; private List<String> phones; // constructors, getters, setters... } Importantly, @Entity and @Id are part of the mapping abstraction, not Valkey itself. The domain model does not require Valkey-specific annotations. Using Jakarta NoSQL Eclipse JNoSQL provides KeyValueTemplate, a specialization of the Jakarta NoSQL Template API for key-value databases. This allows direct persistence and retrieval of entities: Java User user = User.builder() .phones(Arrays.asList("234", "432")) .username("username") .name("Name") .build(); KeyValueTemplate template = container.select(KeyValueTemplate.class).get(); User userSaved = template.put(user); System.out.println("User saved: " + userSaved); Optional<User> userFound = template.get("username", User.class); System.out.println("Entity found: " + userFound); For applications that prefer a repository abstraction, Eclipse JNoSQL integrates with Jakarta Data: Java @Repository public interface UserRepository extends CrudRepository<User, String> { } This approach allows the application code to focus more directly on domain operations: Java User user = User.builder() .phones(Arrays.asList("234", "432")) .username("username") .name("Name") .build(); UserRepository repository = container .select( UserRepository.class, DatabaseQualifier.ofKeyValue() ) .get(); repository.save(user); Optional<User> userFound = repository.findById("username"); System.out.println("User found: " + userFound); Notably, this code includes no Valkey-specific API in the entity or repository. Valkey is an infrastructure choice, while Jakarta NoSQL and Jakarta Data remain the application-facing abstractions. This separation guarantees the architecture remains reusable if the underlying key-value technology changes. Conclusion Key-value databases are highly effective for workloads that require direct access, low latency, and high throughput, rather than complex queries or relational navigation. This article examined how this model fits within enterprise architecture and how Valkey can integrate via Eclipse JNoSQL, allowing applications to avoid direct dependencies on vendor-specific APIs. By maintaining consistent entity mapping and using Jakarta NoSQL or Jakarta Data abstractions, switching key-value implementations becomes mainly a matter of infrastructure and configuration. This shift reflects a broader evolution in enterprise Java, as the platform expands its persistence capabilities beyond traditional relational databases. With Jakarta Persistence, Jakarta Data, Jakarta NoSQL, and tools like Eclipse JNoSQL, architects can choose the best data model for each workload while keeping familiar programming abstractions. Valkey enhances this ecosystem by providing a robust key-value option, making polyglot persistence both feasible and practical.

By Otavio Santana CORE

A detector I built was scoring 0.067 recall on temporal errors, meaning it caught about one in fifteen of the wrong dates it was supposed to find. Wrong dates are supposed to be the easy category: extract the years from the claim, extract the years from the source, compare. There is no semantics to get wrong. I assumed the extraction was broken and went looking for the bug. The extraction was fine. The benchmark was the problem, and not in a way that showed up anywhere in the code. The contexts had been written in the wrong voice. That's the part worth passing on, and it has nothing to do with hallucination detection. It applies to anyone who builds a synthetic evaluation set, which by now is most of us. A Detector That Failed Because of Prose Style The setup: a claim, a source context, and a question about whether the claim is supported. The detector is part of HallucinoType, an open-source package I maintain, and its benchmark was built the way most synthetic evaluation sets are built. Take a faithful claim, inject an error of a known type, keep the label. Two hundred fifty pairs, stratified across eight failure categories, thirty-five of them faithful so I could measure false positives. When I wrote the contexts for the date items, I wrote them the way a person naturally writes when they know the claim is wrong. Something like the treaty was signed in 1928, not 1938. Read that as a human, and it is unambiguous. Read it as a program that treats the context as a reference document, and the string 1938 is sitting right there in the source. The detector extracted it, matched it against the claim, found agreement, and passed the item. Every one of the thirty temporal items had this property. Seven of thirty numerical items did too. Rewriting the contexts as ordinary reference prose, the kind a retrieval system would hand you, moved temporal recall from 0.067 to 1.000. I changed no code. The numerical items did not move. They stayed at 0.600, which told me their misses had a different cause and saved me from congratulating myself on a fix that only worked once. The generalization is short enough to put on a sticky note. A context that argues with the claim is not the context a production pipeline supplies. I had unconsciously written my source documents in a fact-checking register, because I was thinking like an annotator rather than like a retrieval index. The register leaked the answer key into the input, and the system read it, exactly as it was built to. What makes this uncomfortable is that nothing about the benchmark looked wrong. The labels were correct, and the errors were real errors; any reviewer would have signed off on it. The defect lived in a stylistic property of the prose that no one thinks to specify, and it moved a headline number by a factor of fifteen. The Same Bug Wearing a Different Hat The same mistake showed up a second time, and I did not recognize it at first. A second detector in the same system checks whether a claim names the wrong person, company, or place. It works by extracting entities from the claim and looking for them in the context. If the entity appears in the context, the detector skips the claim, on the theory that the source confirms it. I upgraded the entity recognizer, the component that reads a sentence and tags which words are people, places, or organizations, from a regular-expression fallback to a proper statistical model. Recall fell from 0.200 to 0.067. A better component made the system worse, which is the kind of result that stops you mid-sprint. The recognizer was not at fault. The skip rule was. A source document can mention a person in a role that has nothing to do with the claim under evaluation, and still mention them truthfully. In one item, the claim misattributed who was second to walk on the Moon, and the context named the substituted astronaut correctly in a different sentence, doing a different thing. The weaker recognizer missed that mention and flagged the error. The stronger one found it, read it as confirmation, and waved the error through. Of twenty-seven items the detector wrongly skipped, twenty-six followed this pattern. The heuristic was never checking the right relationship. It asked whether the entity appears in the document when it needed to ask what the entity is doing in the sentence. Improving the model's ability to answer the wrong question just made it answer the wrong question more reliably. This is a hazard anywhere a rule sits on top of a learned component. Ablating downward, swapping in a deliberately weaker component to confirm the strong one is earning its cost, is something I do routinely. Ablating upward is rarer, and it tells you more: a rule that degrades when its inputs improve is a rule whose logic was wrong all along, and no amount of model quality saves it. A third instance, smaller but the same shape: a pattern for matching units of measurement was absorbing a trailing word, which let bare four-digit years slip past a filter meant to exclude them from numeric comparison. Fixing one regular expression moved numerical precision from 0.857 to 0.947. A lot of apparently semantic behavior turns out to be lexical. What the Headline Number Was Hiding None of these three defects were visible in the metric I would have reported at a demo. On the binary question of whether a claim is unsupported, the full system reached precision 0.991 and recall 0.986: almost nothing it flagged was fine, and almost nothing that was wrong got past it. Those are the numbers that go in an abstract. Averaged across the eight failure categories the system is supposed to distinguish, precision was 0.578 and recall 0.723. One category sat at 0.067 recall. Another fired on nearly everything, reaching 0.960 recall at 0.198 precision, meaning it claimed credit for errors that more specific detectors had already identified correctly. The binary number was not wrong. It was answering a question so coarse that every interesting failure averaged out of it. A system can be excellent at deciding that something is broken and close to useless at saying what broke. If the only number on your dashboard is the first one, you won't find out until the fine-grained output reaches someone who depends on it. None of this is new. It's the same argument as reporting per-class results instead of overall accuracy on an imbalanced dataset. Everyone agrees with it in principle and skips it anyway, because the aggregate is the number that makes the case for the work. Not Getting Fooled by Your Own Corpus Four practices came out of this, all cheap, none of them clever. Write your evaluation inputs in the register your production system receives. If your system reads retrieved documents, your test contexts should read like documents, not like annotations about documents. Voice is a feature your model can see, and the voice of someone who already knows the answer is a particularly dangerous one to hand it.Ablate upward, not just downward. Replace a component with a better one and check that every metric moves in the direction you expect. When something moves the wrong way, the rule sitting on top of that component is making an assumption you have not written down.Report per-stratum results next to the aggregate, always in the same table. Not in an appendix, not on request. If a category is at 0.067, that fact should be as easy to see as the number you are proud of.Hold out items you did not write. A corpus built by the same people who defined the categories will flatter the categories. Mine did. That is the single largest caveat on everything above, and no amount of internal rigor substitutes for a test set authored by someone else. The first one I would not have thought of before it cost me a day, and it's the one I now suspect is quietly wrong in a lot of synthetic eval sets. Injected-error benchmarks are easy to build, and their labels are correct by construction, which makes them feel safer than they are. The label being right does not mean the input is representative. The Register Your System Actually Speaks The failures worth writing up are rarely the ones where the model underperforms. They are the ones where the measuring apparatus was quietly reporting on something other than what you thought. A detector that scores 0.067 because the test data argues with itself is not a model problem. Neither is a rule that gets worse as its inputs get better, or an aggregate that averages away the only result that mattered. A bigger model fixes none of it. What fixes them is reviewing the evaluation harness as carefully as the thing it measures, defects and all. That's unglamorous work, and where most of my debugging time went. Probably where most of yours goes too.

Oracle Database 23ai introduced the powerful DBMS_DEVELOPER package, giving developers and database administrators a streamlined way to access database object metadata in JSON format. This feature represents a significant advancement in how we interact with database schemas, offering a more structured and programmatic way to extract and analyze metadata compared to traditional dictionary views or the older DBMS_METADATA package. In this article, we'll explore the capabilities of DBMS_DEVELOPER, focusing on its GET_METADATA function through detailed examples and practical implementation scenarios. Understanding DBMS_DEVELOPER The DBMS_DEVELOPER package was designed specifically for modern application development patterns, where JSON has become a universal data exchange format. Rather than returning metadata as DDL statements (like DBMS_METADATA), this package returns structured JSON documents that can be easily parsed, processed, and integrated into applications or DevOps workflows. Key Benefits Structured data format: Returns metadata as JSON objects that can be easily parsed Programmatic access: Perfect for integration with applications and automation scripts Versioning capabilities: Built-in ETag mechanism for tracking object changesConfigurable detail levels: Ability to retrieve basic, typical, or comprehensive metadata Setting Up Our Environment Let's set up a sample schema to demonstrate the package functionality: SQL CREATE TABLE customers ( customer_id NUMBER(10) CONSTRAINT pk_customers PRIMARY KEY, first_name VARCHAR2(50) NOT NULL, last_name VARCHAR2(50) NOT NULL, email VARCHAR2(100) CONSTRAINT uk_customer_email UNIQUE, join_date DATE DEFAULT SYSDATE, status VARCHAR2(10) DEFAULT 'ACTIVE' ); CREATE INDEX idx_customer_name ON customers(last_name, first_name); CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email FROM customers WHERE status = 'ACTIVE'; GET_METADATA Basics The core function of the DBMS_DEVELOPER package is GET_METADATA, which returns metadata about database objects in JSON format. Let's start with a basic example: SQL -- Using JSON_SERIALIZE for formatted output SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; The result is a structured JSON document containing comprehensive information about the table, including: Table name and schema Column definitions with data types and constraints Primary key, unique key, and foreign key information Index definitions An etag value representing the current state of the object This structured format makes it significantly easier to extract specific information programmatically compared to parsing DDL statements. NAME and SCHEMA Parameters The NAME and SCHEMA parameters work together to identify the specific database object. These parameters are case-sensitive and must match the object definition in the data dictionary. SQL -- Explicitly specifying schema SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'CUSTOMERS', schema => 'FINANCE') PRETTY) AS metadata; -- Using current schema (implicit) SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; When the SCHEMA parameter is omitted, the function uses the current schema. This behavior provides flexibility when working with objects across different schemas in your database environment. OBJECT_TYPE Parameter The OBJECT_TYPE parameter allows you to explicitly specify the type of object you're retrieving metadata for. While often optional (as the database can infer the object type from the name), it becomes necessary in cases where name resolution alone is insufficient. Currently, `DBMS_DEVELOPER` supports three object types: TABLEINDEXVIEW Let's examine metadata for our index and view: SQL -- Retrieving index metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', object_type => 'INDEX') PRETTY) AS metadata; -- Retrieving view metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', object_type => 'VIEW') PRETTY) AS metadata; The OBJECT_TYPE parameter becomes particularly important when dealing with objects that share the same name but have different types, such as packages and package bodies. LEVEL Parameter The LEVEL parameter controls the amount of detail included in the JSON output. Oracle provides three levels: BASIC: Minimal informationTYPICAL: Standard level of detail (default)ALL: Comprehensive metadata This flexibility lets you balance concise output with detailed information based on your needs. SQL -- Basic level metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'BASIC') PRETTY) AS metadata; -- All details SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'ALL') PRETTY) AS metadata; The output at the ALL level includes additional attributes such as segment information, compression settings, and physical storage details that aren't present at the BASIC level. ETAG Parameter One of the most powerful features of DBMS_DEVELOPER is the etag mechanism, which provides version tracking for database objects. The etag value changes whenever the object definition changes, making it invaluable for change detection. SQL -- Store the current etag value DECLARE v_metadata CLOB; v_etag VARCHAR2(100); BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA(name => 'ACTIVE_CUSTOMERS'); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; DBMS_OUTPUT.PUT_LINE('Current etag: ' || v_etag); END; / -- Modify the view CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email, join_date FROM customers WHERE status = 'ACTIVE'; -- Check if the object has changed using the stored etag SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', etag => 'A1B2C3D4E5F6G7H8I9J0') -- Previous etag value PRETTY) AS metadata; When you pass an ETag value that matches the current state of the object, the function returns an empty JSON document {}. If the object has changed, it returns the complete metadata with a new ETag value. Practical Scenario: Database Migration and Documentation Let's consider a practical scenario where DBMS_DEVELOPER proves invaluable: a large-scale database migration project with continuous schema changes. The Challenge You're leading a project to migrate a critical application database from on-premises to Oracle Cloud. The development team continues to make schema changes during the migration process, and you need to: Document the current state of all database objectsTrack changes between migration wavesValidate that objects were created correctly in the target environmentGenerate comprehensive documentation for compliance requirements The Solution Using DBMS_DEVELOPER, you can create a robust metadata management system: SQL CREATE TABLE schema_versions ( object_name VARCHAR2(128), object_type VARCHAR2(30), object_schema VARCHAR2(128), capture_date TIMESTAMP, etag VARCHAR2(100), metadata CLOB ); -- Procedure to capture all tables in a schema CREATE OR REPLACE PROCEDURE capture_schema_metadata(p_schema VARCHAR2) AS v_metadata CLOB; v_etag VARCHAR2(100); CURSOR c_objects IS SELECT object_name, object_type FROM all_objects WHERE owner = p_schema AND object_type IN ('TABLE', 'INDEX', 'VIEW'); BEGIN FOR obj IN c_objects LOOP BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA( name => obj.object_name, schema => p_schema, object_type => obj.object_type ); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; INSERT INTO schema_versions (object_name, object_type, object_schema, capture_date, etag, metadata) VALUES (obj.object_name, obj.object_type, p_schema, SYSTIMESTAMP, v_etag, v_metadata); COMMIT; DBMS_OUTPUT.PUT_LINE('Captured metadata for ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Error capturing ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name || ': ' || SQLERRM); END; END LOOP; END; / This solution provides several key benefits: Efficient change tracking: Using etags to identify exactly which objects have changedStructured documentation: Storing metadata in JSON format for easy extraction of specific attributesHistorical record: Maintaining snapshots of schema evolution over timeValidation capabilities: Comparing source and target schemas during migration During migration, you can extend this system to compare environments: -- Procedure to compare object between environments CREATE OR REPLACE PROCEDURE compare_object( p_name VARCHAR2, p_type VARCHAR2, p_source_schema VARCHAR2, p_target_schema VARCHAR2, p_target_db VARCHAR2 ) AS v_source_metadata CLOB; v_target_metadata CLOB; v_source_etag VARCHAR2(100); v_target_etag VARCHAR2(100); BEGIN -- Get source metadata v_source_metadata := DBMS_DEVELOPER.GET_METADATA( name => p_name, schema => p_source_schema, object_type => p_type ); -- Get target metadata via database link EXECUTE IMMEDIATE 'SELECT DBMS_DEVELOPER.GET_METADATA( name => :1, schema => :2, object_type => :3 ) FROM dual@' || p_target_db INTO v_target_metadata USING p_name, p_target_schema, p_type; -- Extract etag values SELECT JSON_VALUE(v_source_metadata, '$.etag') INTO v_source_etag FROM dual; SELECT JSON_VALUE(v_target_metadata, '$.etag') INTO v_target_etag FROM dual; -- Compare and report IF v_source_etag = v_target_etag THEN DBMS_OUTPUT.PUT_LINE('Objects match exactly'); ELSE DBMS_OUTPUT.PUT_LINE('Objects differ - detailed comparison needed'); -- Further JSON comparison logic could be implemented here END; END; / Conclusion The DBMS_DEVELOPER package represents a significant advancement in Oracle's metadata management capabilities. By providing metadata in JSON format, Oracle has created a more developer-friendly interface that aligns with modern application architecture patterns. Key takeaways include: JSON-based metadata is more programmatically accessible than traditional DDL statements The etag mechanism provides a reliable way to track object changes Multiple detail levels allow you to retrieve just the information you need The package is particularly valuable for documentation, migration, and change tracking While currently limited to tables, indexes, and views, the DBMS_DEVELOPER package has tremendous potential for expansion in future Oracle releases. Database architects and developers should consider integrating this powerful tool into their workflows, particularly for projects involving schema documentation, migration, or programmatic metadata access. As databases continue to evolve toward more autonomous and programmable systems, tools like DBMS_DEVELOPER will become increasingly central to efficient database management practices.

By arvind toorpu CORE