Software Design and Architecture

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen. Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel. In this blueprint, we will build a secure multi-tenant execution environment. We will containerize the agent runtime using Docker, provision a GKE cluster with Axion nodes, isolate the network, and orchestrate the execution layer using a robust Java backend. Step 1: Containerizing the Agent Runtime The first step is establishing a baseline execution environment. We want this Docker image to be as lightweight as possible to reduce the attack surface, while containing the necessary runtimes for the LLM to execute its logic. Dockerfile # Use a minimal Alpine base image to reduce attack surface FROM python:3.11-alpine # Create a non-root user for execution RUN addgroup -S agentgroup && adduser -S agentuser -G agentgroup WORKDIR /sandbox # Copy the execution wrapper script COPY --chown=agentuser:agentgroup execute_payload.py /sandbox/ # Enforce non-root execution USER agentuser # Prevent Python from writing pyc files and buffering stdout ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 CMD ["python", "execute_payload.py"] To make this functional, we need an entrypoint script that safely reads the LLM-generated code from an injected environment variable or a mounted volume, executes it, and captures the output. Here is a simplified execute_payload.py implementation: Python import os import sys import traceback def main(): # In a production environment, this payload might be injected via # a Kubernetes Secret or a secure sidecar proxy. encoded_payload = os.environ.get("AGENT_PAYLOAD", "") if not encoded_payload: print("Error: No payload provided.") sys.exit(1) try: # Execute the untrusted code within this isolated process # Security constraints are handled by the container and gVisor layers exec(encoded_payload, {"__builtins__": __builtins__}, {}) except Exception as e: print(f"Execution Error: {str(e)}") traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() Even if a malicious script breaks out of the Python runtime, it will find itself as an unprivileged user inside a minimal Alpine container. Step 2: Provisioning GKE With Axion and Agent Sandbox Google Axion (N4A) processors provide excellent performance per watt, making them ideal for running hundreds of concurrent, lightweight agent tasks. We will create a cluster and explicitly enable the sandbox feature. Shell # Create the GKE cluster with Sandbox enabled gcloud container clusters create agent-sandbox-cluster \ --region us-east4 \ --enable-sandbox \ --sandbox type=gvisor \ --release-channel regular # Create a dedicated node pool using Axion N4A instances gcloud container node-pools create axion-agent-pool \ --cluster agent-sandbox-cluster \ --region us-east4 \ --machine-type n4a-standard-4 \ --num-nodes 3 \ --node-labels dedicated=untrusted-agents \ --tags untrusted-workload Applying node labels ensures that trusted core microservices do not accidentally end up on the same physical infrastructure as untrusted agent execution environments. Step 3: Enforcing Network Isolation Compute isolation is useless if the untrusted code can scan your internal network or exfiltrate data to the public internet. We must deploy a strict NetworkPolicy to default-deny all egress traffic from our sandboxed namespace. YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-agent-egress namespace: isolated-agents spec: podSelector: matchLabels: app: agent-executor policyTypes: - Egress egress: # Only allow DNS resolution - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP # Allow outbound only to a specific internal API gateway if needed # - to: # - ipBlock: # cidr: 10.0.0.50/32 Step 4: Deploying the Sandboxed Workload With the network secured, we define the Kubernetes deployment. By setting the runtimeClassName to gvisor, Kubernetes routes the container lifecycle through the GKE Agent Sandbox rather than the standard container runtime. YAML apiVersion: apps/v1 kind: Pod metadata: generateName: dynamic-agent-task- namespace: isolated-agents labels: app: agent-executor spec: # Instruct GKE to use the Agent Sandbox (gVisor) runtimeClassName: gvisor # Ensure these pods only land on our Axion node pool nodeSelector: dedicated: untrusted-agents restartPolicy: Never containers: - name: execution-environment image: your-registry/agent-runtime:v1.0.0 env: - name: AGENT_PAYLOAD valueFrom: secretKeyRef: name: task-payload-secret key: payload # Drop all unnecessary Linux capabilities securityContext: runAsUser: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" volumeMounts: - name: temp-storage mountPath: /tmp volumes: - name: temp-storage emptyDir: {} Step 5: Orchestrating the Execution via Java Spring Boot To bring this architecture together, the control plane must dynamically spin up these sandboxed pods whenever an AI agent decides it needs to run code. In a modern distributed system, this is typically handled by a core backend microservice. Using the Fabric8 Kubernetes Client in a Java Spring Boot application provides a highly resilient way to orchestrate these ephemeral workloads programmatically. Java import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.client.KubernetesClient; import org.springframework.stereotype.Service; @Service public class AgentOrchestratorService { private final KubernetesClient kubernetesClient; public AgentOrchestratorService(KubernetesClient kubernetesClient) { this.kubernetesClient = kubernetesClient; } public String executeUntrustedCode(String tenantId, String pythonCode) { // 1. Create a Kubernetes Secret containing the code payload String secretName = createPayloadSecret(tenantId, pythonCode); // 2. Load the sandbox Pod template and inject the specific payload secret Pod sandboxedPod = kubernetesClient.pods() .inNamespace("isolated-agents") .load(getClass().getResourceAsStream("/k8s/agent-pod-template.yaml")) .item(); // 3. Launch the pod dynamically via the API server Pod runningPod = kubernetesClient.pods() .inNamespace("isolated-agents") .create(sandboxedPod); // 4. Await completion and extract the logs safely kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .waitUntilCondition(pod -> pod.getStatus().getPhase().equals("Succeeded") || pod.getStatus().getPhase().equals("Failed"), 30, java.util.concurrent.TimeUnit.SECONDS); String executionLogs = kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .getLog(); // 5. Clean up the ephemeral resources kubernetesClient.pods().delete(runningPod); kubernetesClient.secrets().withName(secretName).delete(); return executionLogs; } } The Defense in Depth Strategy This architecture relies on a strict defense in depth model. If an LLM hallucinates a malicious payload or a user deliberately attempts prompt injection to compromise the platform, the attacker faces multiple independent barriers. The code executes as a non-root user in a minimal Alpine environment with a read-only filesystem. Network access is completely blocked by native Kubernetes policies. Finally, any attempt to exploit kernel vulnerabilities is intercepted by the gvisor runtime boundary running on dedicated Axion hardware. By combining these layers, engineering teams can build and scale trustworthy Agentic AI platforms without risking the integrity of their core cloud infrastructure.

Branch networks no longer behave like quiet extensions of a single headquarters LAN. They terminate local user traffic, break out directly to the internet for SaaS, maintain persistent connections back to core systems, and increasingly host devices that are operationally important even when central resources are unavailable. NIST notes that the enterprise network landscape has shifted because of cloud services, geographic dispersion, and changes in application design, while zero trust guidance emphasizes that network location is no longer the primary signal of trust. In practice, that means a branch cannot be secured by treating the site-to-site tunnel as a blanket trust boundary. The branch edge has to make explicit policy decisions about which flows are allowed, which flows are encrypted, which flows are inspected, and which identities are entitled to touch which resources. Beyond the Old Perimeter The older perimeter model assumed that most meaningful risk arrived from outside the network and that internal traffic was comparatively trustworthy. That assumption breaks down quickly in distributed environments. NIST’s current network guidance explicitly calls out the limitations of perimeter-centric protection and VPN-centric access in environments that include cloud services, remote users, and branch offices, while NSA’s zero trust guidance frames lateral movement as a primary post-compromise technique that segmentation and granular policy are meant to contain. A modern branch design therefore needs layered control points close to the resource and close to the user, not just a tunnel back to a core firewall. That shift also changes how edge devices are treated operationally. Branch firewalls, VPN gateways, and routers are no longer simple plumbing. They are security control planes, and they are common targets. CISA issued Binding Operational Directive 23-02 specifically to reduce the risk from internet-exposed management interfaces, and NSA recommends encrypted administration, ACL-restricted management access, and dedicated management segments rather than broad reachability from production networks. Securing the branch therefore starts with the idea that the branch edge itself must be hardened, isolated, and observable before it is entrusted to enforce policy for anything else. Firewalls Define Intent A branch firewall is most effective when it expresses business intent instead of accumulating ad hoc port exceptions. NIST’s firewall guidance is still the right mental model: block inbound and outbound traffic unless it is expressly permitted, use stateful inspection to track valid sessions, and apply egress filtering so that spoofed or unexpected source traffic cannot leave the site. Where application awareness is needed, NIST also notes that application-proxy gateways can inspect protocol content and, in some cases, decrypt and re-encrypt selected traffic before forwarding it. That combination turns the firewall from a coarse packet filter into a policy engine that knows the difference between permitted business traffic and merely possible traffic. A concise nftables policy for a small branch can be deliberately narrow: Plain Text table inet filter { chain forward { type filter hook forward priority 0; policy drop; ct state established,related accept iifname "lan" oifname "wan" ip saddr 10.20.30.0/24 ip daddr 10.10.0.0/16 tcp dport 443 accept iifname "lan" oifname "wan" ip saddr 10.20.30.0/24 udp dport 53 accept iifname "lan" oifname "wan" ip saddr 10.20.30.0/24 tcp dport { 80, 443 } accept } } The shape of that ruleset matters more than the exact addresses. The first line admits only established or related traffic, which keeps return paths fast without making the policy permissive. The next rule allows a very specific branch-to-core application path over HTTPS. DNS is explicitly separated because name resolution is usually treated as infrastructure rather than open internet access. The final rule allows only web egress from the branch subnet, and the chain-wide policy drop turns every other flow into an intentional denial instead of an accidental omission. That aligns with NIST’s deny-by-default and egress-filtering guidance, and it scales far better than a firewall that starts from “allow any” and slowly adds patches. VPNs Protect the Path VPNs remain essential in branch networking, but their role is precise: protect traffic in transit across untrusted transport, not grant broad implied trust to the attached network. NIST’s IPsec guidance identifies gateway-to-gateway VPNs as the common model for linking a branch office to headquarters and notes that the model is operationally simple because it is largely transparent to end users. The same guidance recommends IKEv2 over IKEv1 because IKEv2 is simpler, faster, and more secure, and it lists modern algorithm choices such as AES-GCM and SHA-2 families as recommended options. It also states that tunnel mode is used for gateway-to-gateway deployments and that perfect forward secrecy should be used when resources allow. A stripped-down strongSwan configuration shows the right shape for a branch-to-core tunnel: Plain Text connections { branch-hq { version = 2 remote_addrs = 198.51.100.10 proposals = aes256gcm16-prfsha384-ecp384 local { auth = pubkey; certs = branch-gw.pem; id = branch-gw.example } remote { auth = pubkey; id = hq-gw.example } children { corp { local_ts = 10.20.30.0/24 remote_ts = 10.10.0.0/16 esp_proposals = aes256gcm16-ecp384 rekey_time = 50m start_action = trap } } dpd_delay = 30s } } The important details are the constrained traffic selectors and the modern cryptographic profile. local_ts and remote_ts keep the tunnel scoped to known subnets instead of turning it into a default route for every packet. rekey_time shortens the lifetime of key material, while dpd_delay enables liveness checking so dead peers do not leave stale state behind. strongSwan’s configuration model exposes exactly those selectors, proposals, and peer-liveness controls, which map cleanly onto NIST’s guidance for tunnel mode, IKEv2, and periodic key refresh. Just as important, NIST’s broader network guidance warns that VPN-based access has limits in the current enterprise landscape. A secure tunnel does not solve segmentation, visibility, or granular authorization by itself. IDS and IPS Reveal Drift Firewalls and VPNs are excellent at enforcing expected paths, but they are not enough to detect misuse inside those paths. That is where network IDS and IPS become decisive. NIST’s IDPS guidance recommends products that combine signature-based detection, anomaly-based detection, and stateful protocol analysis because each method compensates for the others. Signature-based methods are efficient for known threats but weak against novel variants and evasion; anomaly-based methods can detect unknown abuse but are noisy without careful profiling; stateful protocol analysis helps distinguish legitimate protocol behavior from malformed or abusive sequences. NIST also stresses that these systems require tuning and that prevention actions should often be tested in simulation or learning modes before being enforced inline. A practical Suricata rule can be very small while still expressing a meaningful branch policy: Plain Text drop tls $HOME_NET any -> $EXTERNAL_NET any ( msg:"Deprecated TLS from branch host"; tls.version:1.0; sid:1001001; rev:1; ) The rule follows Suricata’s standard structure of action, header, and rule options. In IPS mode, drop blocks the flow and generates an alert, while tls.version:1.0 turns a broad “bad crypto” idea into an enforceable control that stops unsafe client negotiations at the branch edge. That kind of rule is useful because it binds transport hygiene to observable protocol behavior instead of relying on application owners to update every endpoint perfectly. The placement of the sensor still matters. NIST explicitly warns that network-based IDPS cannot inspect payloads inside encrypted traffic such as VPN, HTTPS, or SSH unless traffic is analyzed before encryption or after decryption. In a branch, that usually means placing inspection logically behind the VPN gateway for branch-to-core traffic and beside the egress path for direct internet breakout. Identity Turns Access into Policy The most important change in branch security is that authorization can no longer be inferred from attachment alone. NIST’s zero trust architecture states that access to enterprise resources should be granted on a per-session basis with least privilege, and that policy decisions can vary by identity, device status, network location, time, and other environmental signals. NIST’s secure network landscape guidance pushes the same idea further by arguing that user identity alone is not sufficient and that contextual information about devices and services must be part of the decision. CISA’s zero trust maturity model reinforces that direction by describing automated access controls that consider identity, device risk, application, and data category, and that are time-limited. At the branch edge, the most practical implementation is usually 802.1X with EAP-TLS backed by RADIUS. IEEE 802.1X defines mutual authentication for LAN-attached clients and ports, while EAP-TLS provides certificate-based mutual authentication and key derivation. Once that identity has been established, RADIUS can return standard attributes that place the endpoint into the correct VLAN and attach the correct ACL. RFC 3580 specifies the exact tunnel attributes used for VLAN assignment, and a FreeRADIUS users file can express the authorization response very compactly: Plain Text [email protected] Tunnel-Type := VLAN, Tunnel-Medium-Type := IEEE-802, Tunnel-Private-Group-Id := "120", Filter-Id := "finance-restricted" That snippet is intentionally small, but the effect is powerful. A successful 802.1X session for the named identity receives a VLAN and an access filter rather than broad branch connectivity. The same pattern can be extended from a named user to directory-driven roles, device classes, posture states, and time-bounded administrative sessions. It is also the reason identity-based access belongs in the network discussion rather than only in the IdP discussion: the branch switch or wireless edge becomes the first enforcement point where verified identity is translated into concrete packet-level reachability. Conclusion A secure branch is not created by stacking appliances and hoping that defense in depth emerges automatically. It is created by dividing responsibility cleanly across controls that complement one another. The firewall establishes a deny-by-default policy and limits what can traverse the site. The VPN protects selected traffic across untrusted transport without pretending that encryption is the same thing as trust. IDS and IPS expose misuse, drift, and protocol abuse that still occur inside permitted paths. Identity-based access ensures that branch attachment results in the minimum reachability justified by the authenticated subject and device, not by the convenience of a subnet. When those controls are composed deliberately, the branch stops being a soft edge and becomes a constrained, observable, and policy-driven part of the enterprise security fabric.

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we'd agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas. What it didn't account for is that each replica needed almost three minutes just to pull a 14GB checkpoint and warm up CUDA kernels before it could answer a single request. The load balancer, seeing new pods report as running, immediately started routing traffic to them. For three minutes, a chunk of our users got 504s while perfectly healthy-looking pods sat there loading a model into memory. Nobody on the infra side had touched Docker that day. The incident wasn't a Docker bug. We assumed that container orchestration designed for web services would function the same way for processes that take minutes to become useful, rather than those that operate in milliseconds. Why LLM Containers Break the Usual Assumptions Packaging an LLM serving stack in Docker still makes sense for the same reason it always has; CUDA versions, driver compatibility, and Python ABI mismatches are miserable to manage across a fleet without a frozen artifact. But an LLM container carries baggage that a typical inference service doesn't. The weights are tens of gigabytes, not a few hundred megabytes. GPU memory is a single shared pool that one greedy container can quietly exhaust for everyone else on the box. And “ready” doesn't mean “process started”; it means the model is resident in VRAM and the CUDA graph is warmed, which can take minutes on a cold node pulling weights from object storage over the network. The Mistakes, in Order Our first version baked the model weights directly into the image, because it felt simpler: one artifact, one pull, done. In practice, it meant a 16GB image, painfully slow CI pushes, and a registry bill nobody wanted to look at. Worse, every time we bumped into a new fine-tuned checkpoint, we rebuilt and repushed the entire layer regardless of caching, because the COPY step touching gigabytes of weight files invalidates everything below it. Unlike a typical ML inference image, there's no meaningful caching win here at all; the layer is simply too big to ever be a cache hit across versions. We moved weights out to a mounted volume, fetched at container start from object storage, and never looked back. Second mistake, and this one actually cost us a production incident: we ran the container with Docker's default shared memory size. vLLM, which we used for serving, spins up worker processes that talk to each other over shared memory even on a single GPU. With the default 64MB /dev/shm, those workers would crash with cryptic bus errors under any real concurrency. The fix was almost embarrassingly small: Shell docker run --gpus all \ --shm-size=2g \ -e MODEL=mistralai/Mistral-7B-Instruct-v0.2 \ -e GPU_MEMORY_UTILIZATION=0.85 \ -e MAX_MODEL_LEN=8192 \ -p 8000:8000 \ llm-serve:latest The third mistake was more subtle and took longer to diagnose. vLLM's continuous batching reserves a large slice of GPU memory upfront for the KV cache, controlled by gpu_memory_utilization. We'd set that fraction high to maximize throughput, then bin-packed two replicas onto the same GPU to save cost. Under normal traffic, fine. During a burst of unusually long-context requests, such as someone summarizing a 6,000-word document instead of a tweet, the KV cache for that single batch ballooned, causing the container to run out of memory (OOM) mid-generation and taking down every other in-flight request in the same batch. This failure mode is more severe than a typical web service OOM because it not only drops the new request but also terminates queries that were already halfway through generating answers for paying customers. What We Actually Changed The readiness adjustment turned out to matter more than any Docker flag. We split liveness from readiness: liveness just checks that the process hasn't died; readiness fires a real, tiny generation request through the local API and only flips to healthy once that round trip succeeds. That alone killed the cold-start routing problem because the load balancer stopped trusting a merely alive process. We also gave up on bin-packing two replicas per GPU. In hindsight, treating GPU memory like it's as elastic as CPU or RAM was the actual root cause, not any single Docker setting. We implemented a model that uses one GPU, sets a conservative memory utilization ceiling, and enforces a request-level token limit at the proxy in front of the container, rather than inside it, because it is too late to make adjustments once the batch is already running. On the orchestration side, we stopped trying to scale-to-zero or scale aggressively off CPU-style metrics. Scale-to-zero is effective for web apps but doesn’t fit GPU-bound LLM serving, where cold starts can outlast traffic spikes. We kept a warm floor of replicas sized to baseline traffic and let a request queue absorb bursts instead of expecting new pods to materialize in time. It's less elegant than the autoscaling story everyone likes to tell, and it costs more in idle GPU time, but it's honest about what the hardware can actually do. What We Rejected, and Why We seriously considered dropping self-hosting altogether and routing through a managed inference API. For a side project, that's probably the right call — less to own, no GPU bin-packing headaches. We rejected it due to data residency requirements that prohibited sending raw text to a third party, and at our volume, managed pricing would quickly exceed our GPU costs. We also looked at Ray Serve and Triton early on, and they solve some of the issues more natively, but the team's Docker and Kubernetes muscle memory was strong enough that rebuilding on a new serving framework felt like trading one set of unknowns for another, at least for the first version. Key Takeaways Never bake multi-gigabyte model weights into the image — there's no caching benefit at that size, only slower pushes and bigger registry bills.Set shared memory explicitly; vLLM and similar multiprocess servers will fail under load with Docker's tiny default.Treat GPU memory utilization conservatively and avoid bin-packing replicas onto a single GPU unless you can guarantee a strict ceiling per container.Build a readiness assessment that performs a real generation, not just a process check; cold model loading will otherwise receive routed live traffic.Don't expect autoscaling to save you on cold-start timescales measured in minutes; a warm floor plus a queue is more honest than reactive scaling. Closing Thought None of these issues was really a Docker failure; the container did exactly what we told it to do. The failure was treating a multi-gigabyte, GPU-bound, slow-to-warm process like it was just another stateless web container that happens to need a GPU flag. I suspect that many teams will learn this lesson in the same way we did, during an incident on a Friday afternoon. Is it the right move to keep stretching Docker and Kubernetes to fit LLM serving, or is this the workload that finally pushes most teams toward purpose-built serving layers?

The Production Story Several years ago, my team made a decision that felt obviously correct: If a downstream call fails, retry it. More retries, more resilience. We set three retries on every integration touching our order-fulfillment platform, shipped it on a Thursday, and went home feeling good about our reliability posture. Six weeks later, retries were the single largest source of traffic in the platform. It surfaced during a routine inventory-sync slowdown. The inventory service got a little sluggish. Nothing dramatic. p99 crept from 200ms to maybe 1.2s. Our order API, sitting one layer up, started timing out and retrying. Three times each. The MuleSoft layer feeding the order API also had retries configured, so it retried the retries. By the time traffic reached the already-struggling inventory service, a single user click had turned into somewhere between nine and twenty-seven backend calls. The inventory service didn't recover. It got buried. We took a partial outage caused entirely by our own retry logic trying to save us. Figure 1: Retry amplification across API layers. One client request fans out to as many as 27 backend calls, while a retry budget keeps the downstream bounded. Why It Happened This part isn't obvious until you've watched it happen. Each retry decision was reasonable on its own, but together they were tearing the platform apart. Every team had configured retries looking only at their own layer. Three retries seemed fine in isolation. The trouble is that retries multiply across layers, and nobody owned the end-to-end number. Two retries here, three there, and suddenly one click is nine calls. The math was sitting in plain sight, and none of us had done it. What really bit us was how retries behave during a partial outage. When a downstream is healthy, retries are cheap, because failures are rare. When it's degraded, which is exactly when you're retrying the most, those retries pile more load onto a service that's already on its knees. So the system gets most aggressive at the worst possible moment. That's a feedback loop, and feedback loops like this end in outages. I've written before about retry storms and bounded reliability, and about how AI-generated DataWeave can fail quietly in production. This is the same family of problem. No single component is broken here. What's missing is a limit that spans the whole call path. Retries without a budget are really just a slow, polite way to DDoS yourself. The Bad Implementation This is what almost everyone ships first. I've shipped it myself. Java @Retryable( value = { RemoteServiceException.class }, maxAttempts = 3, backoff = @Backoff(delay = 200, multiplier = 2) ) public InventoryResponse checkInventory(String sku) { return inventoryClient.get(sku); } At first glance, this looks reasonable. Exponential backoff, a sane attempt count, a typed exception. Code review passes in thirty seconds. The problem is there's no awareness of anything beyond this one method. It retries during a full downstream outage just as eagerly as during a one-off network blip. It has no idea that the caller above it is also retrying. Worse, it happily retries errors that will never succeed. A 400, a validation failure, a duplicate-order rejection. You're burning retries on requests that were dead on arrival. DimensionBad (Naive Retry)Good (Budgeted Retry)Retry triggerAny failureOnly retryable failuresLimitPer-call attempt countFraction of total trafficBehavior under outageAmplifies loadSheds retries, stays boundedCross-layer awarenessNoneBudget shared end-to-endFailure modeRetry stormGraceful degradation The Good Implementation A retry budget flips the control. Instead of asking "how many times should this one call retry," you ask "what fraction of my total traffic is allowed to be retries?" The rule of thumb that's served me well: retries should never exceed 10% of your real request volume. If more than one in ten requests is a retry, something is genuinely broken, and hammering it harder won't fix it. It'll only dig the hole deeper. Here's a token-bucket budget that enforces this. Successful calls slowly refill the budget; each retry spends from it. When the budget is empty, you stop retrying and fail fast. Java public class RetryBudget { private final double retryRatio; // e.g. 0.10 = 10% private final AtomicLong tokens = new AtomicLong(); private final long maxTokens; public RetryBudget(double retryRatio, long maxTokens) { this.retryRatio = retryRatio; this.maxTokens = maxTokens; } // Every real request deposits a little budget back. public void onRequest() { tokens.updateAndGet(t -> Math.min(maxTokens, t + (long)(retryRatio * 100))); } // A retry is only allowed if the budget can pay for it. public boolean tryRetry() { return tokens.updateAndGet(t -> t >= 100 ? t - 100 : t) >= 0 && tokens.get() >= 0 && spend(); } private boolean spend() { return tokens.getAndUpdate(t -> Math.max(0, t - 100)) >= 100; } } The key behavior: under normal load, the budget stays full, and retries work as expected. Under a real outage, failures outpace successes, the budget drains, retries stop, and you protect the downstream instead of finishing it off. Wiring it into a Spring Boot client looks like this. Notice the two gates before any retry happens. The error has to be retryable, and the budget has to allow it. Java public InventoryResponse checkInventory(String sku) { budget.onRequest(); try { return inventoryClient.get(sku); } catch (RemoteServiceException ex) { if (isRetryable(ex) && budget.tryRetry()) { return inventoryClient.get(sku); // single budgeted retry } throw ex; // fail fast, don't amplify } } private boolean isRetryable(RemoteServiceException ex) { int code = ex.statusCode(); return code == 502 || code == 503 || code == 504 || code == 429; } Not every error deserves a retry. This distinction matters more than the budget math, because retrying a non-retryable error is pure waste. ErrorRetryable?Why503 Service UnavailableYesTransient, likely to clear504 Gateway TimeoutYesDownstream slow, may recover429 Too Many RequestsYes, with backoffHonor Retry-After, slow down502 Bad GatewayYesUsually transient routing issue400 Bad RequestNoRequest is malformed, will always fail401 / 403NoAuth won't fix itself on retry409 Conflict (duplicate order)NoRetrying creates a real data problem422 Validation ErrorNoDeterministic rejection The Architecture Pattern In MuleSoft, the same idea applies, and it's where I see the most damage because retries get configured at multiple layers without anyone counting. Keep the platform-level retry shallow and let your error type drive the decision. XML <until-successful maxRetries="1" millisBetweenRetries="500" doc:name="Budgeted Retry"> <http:request method="GET" config-ref="Inventory_HTTP" path="/inventory/{sku}"/> </until-successful> Then classify errors in DataWeave so the flow only retries what's worth retrying, and so a budget breach degrades cleanly rather than throwing: Shell %dw 2.0 output application/json var retryable = [502, 503, 504, 429] --- { shouldRetry: retryable contains payload.statusCode, action: if (retryable contains payload.statusCode) "RETRY_IF_BUDGET" else "FAIL_FAST" } The surprising part, when we rolled this out, was how rarely the budget actually engaged. Under healthy conditions, you'd never know it's there. It only shows its value during the bad fifteen minutes that used to turn into a bad three hours. Figure 2: The retry budget as a token bucket. Successful requests refill it, retries drain it, and once it falls below the 10% line, retries are disabled, and calls fail fast. A Real Production Example Picture a payment-processing flow calling an external gateway, with order-fulfillment and a Salesforce sync downstream. The gateway has a rough afternoon and starts returning intermittent 503s. Without a budget: every failed charge retries three times, order-fulfillment retries the payment call, and the Salesforce sync retries too. The gateway, already wobbling, gets three-to-nine times its normal load and falls over completely. A partial degradation becomes a full payment outage during peak hours. With a 10% budget: the first wave of retries is absorbed normally. As 503s climb, the budget drains within seconds. Retries stop, failed charges fail fast with a clear error, and customers see a retry-later message instead of a spinner. The gateway gets breathing room and comes back on its own. You take a small, honest failure now instead of a much bigger one you caused yourself. Metrics That Matter A budget you can't see is a budget you won't trust. These are the four numbers I put on a dashboard before I ship any retry change to production. MetricWhat it tells youHealthy rangeRetry ratio (retries / total requests)Whether retries are amplifying load< 10%Budget exhaustion eventsHow often the brake engagesRare, spikes during incidentsRetry success rateWhether retries actually help> 50%; if low, stop retryingDownstream p99 during retriesWhether you're worsening the outageShould not climb with retries If your retry success rate is low, that's the tell. You're retrying things that were never going to succeed, and the budget is doing you a favor by cutting them off. Monday-Morning Checklist Count your real end-to-end retry multiplier across every layer, not per service.Set a retry budget at roughly 10% of traffic and enforce it with a token bucket.Classify every downstream error as retryable or not, and never retry 4xx except 429.Cap retries to a single attempt at most layers; let the budget, not the attempt count, be your safety limit.Honor Retry-After on 429s instead of guessing backoff.Put retry ratio and budget-exhaustion metrics on a dashboard before you ship.Test it: degrade a downstream in staging and confirm retries actually stop. Final Thoughts Retries feel like reliability. Really, they're a loan against your downstream's capacity, and like any loan, they're cheap when you don't need them and expensive at the worst possible time. What makes the retry budget useful is that it ties retrying to the one number that should govern it: how much real traffic you're actually serving. I've made this mistake myself, and I've watched sharp teams make it too, because every individual decision looked correct in review. The fix isn't fancier backoff or smarter jitter. It's a ceiling. Decide up front how much of your traffic you'll allow to be retries, then hold that line even when every instinct is screaming at you to push harder. Especially then.

Most API security programs were built for predictable consumers: mobile apps, backend services, partner integrations, and the occasional script. Each of those calls your APIs in fairly bounded ways. AI agents do not fit that model. An agent does not just call an API. It decides which APIs to call, in what order, and often keeps going until it reaches a result. That autonomy is the point of using an agent, but it is also what makes it dangerous: a single misconfigured agent can generate thousands of requests in minutes, reach systems it was never meant to touch, or chain APIs together in a sequence no human ever designed. And importantly, these usually are not “hackers” in the traditional sense. They are systems doing exactly what they were permitted to do only at machine speed and scale. The token is valid. The request is well-formed. No known attack signature fires. That is precisely why the problem is easy to miss and hard to catch with the tooling most teams already run. The good news is that you do not need a new security stack to handle this. Most of the protection still comes down to fundamentals applied properly. The problem is that most organizations never applied those fundamentals with an autonomous, high-volume consumer in mind. This article walks through five controls that make a real difference, and more importantly, how to enforce each one at the API gateway, where it belongs. Where these controls live: Every control below is enforced at the same place: the API gateway sitting between the agent and your backend services. The gateway is the one choke point where you can see every call an agent makes, attach identity and context to it, count it, inspect its pattern over time, and record it. Treating the gateway as the enforcement plane for agents rather than trusting each backend to defend itself. This is the architectural decision that makes the rest of this practical. The examples use Apigee terminology (API products, quota, spike arrest), but the same primitives exist in most gateways. 1. Scope Agent Access to Least Privilege This is where most of the risk starts. In many setups, an AI agent is effectively treated like a backend service account. Once it is trusted, it quietly accumulates permissions, sometimes because it is easier, sometimes because nobody wants to risk breaking functionality. That approach does not hold up under autonomous behavior. An agent designed to help customers check order status does not need access to refunds, account updates, or admin operations. But in real systems, those boundaries are often missing or too loose: An order-status agent needs read access to two resources — nothing more. Plain Text Allowed: GET /orders/{id} GET /customers/{id} Denied by default: POST /refunds DELETE /accounts PUT /admin/* How to enforce it: Do not rely on the agent to request only what it should. Make the scope a property of the credential, enforced by the gateway. In practice, this means giving each agent its own client credential bound to an API product that contains only the endpoints it needs. In Apigee terms, the API product is the scoping boundary: if POST /refunds is not in the product the agent’s key is provisioned against, the gateway rejects the call at the VerifyAPIKey or OAuthV2 step before the request ever reaches a backend, and regardless of what the agent intended. This is enforcement by construction, not by policy the agent is trusted to honor. Scope at two levels. Coarse-grained scoping restricts the agent to a set of products or resource paths. Fine-grained scoping restricts the HTTP methods within them so an agent can be granted read on a resource without ever being able to write to it. The order-status agent gets read on orders and customers; it physically cannot issue a write, because no product in its grant exposes one. The anti-pattern to avoid: The single most common mistake is a shared service-account token reused across multiple agents. It collapses every agent into one identity, makes least privilege impossible (the token must be a superset of everything), and turns one compromised or misbehaving agent into your entire blast radius. Give every agent its own credential. The trade-off: Per-agent credentials and narrowly scoped products multiply the number of artifacts you manage. That is real overhead, and it is worth it — but plan for it with a naming convention and a lifecycle (who owns the credential, how it rotates, when it is revoked, when the agent is retired). 2. Put Hard Limits on Agent Behavior (Not Just Users) Rate limiting is usually thought of as traffic management. With AI agents, it becomes a safety mechanism, and the threat is not just malicious traffic; it is runaway behavior. A procurement agent might try multiple pricing sources, retry failed calls, and loop through suppliers. That is normal logic, until a bad response or an unbounded loop turns it into thousands of API calls in minutes. Two mechanisms, two problems. Teams often reach for a single rate limit and stop. Agents need two different controls that solve two different problems: Spike arrest smooths bursts and protects your backend from a sudden flood. For example, capping an agent at 30 requests per second so a tight retry loop cannot overwhelm a downstream service. It is about instantaneous rate.Quota caps business volume over a window — for example, N calls per day per agent, or a hard ceiling on a specific business action such as refunds per hour. It is about cumulative intent, not instantaneous rate. You want both. Spike arrest keeps a runaway loop from taking down a service in the next ten seconds; quota keeps a subtly wrong agent from doing ten thousand legitimate-looking operations over an afternoon. What you limit on matters: Per-user and per-IP limits fail for agents. One agent frequently acts on behalf of many users, and one user can spawn an agent that fans out across dozens of endpoints. Limit on a key that reflects the agent and its work. A composite of agent identity and workflow identity propagated through the call chain so the gateway can count correctly. Fail safely: When a limit trips, return 429 Too Many Requests with a Retry-After header, and make sure the agent framework treats that as a stop-and-back-off signal rather than a reason to retry harder. Add per-business-action ceilings for high-consequence operations (a refunds agent capped well below any plausible legitimate volume) so a logic error fails closed before it costs money. The trade-off: Set limits too tight and you break legitimate batch or fan-out workflows; too loose and they provide no protection. Baseline against observed normal behavior for each agent before enforcing, and start in a monitor-only mode so you can see what you would have blocked. 3. Don’t Trust “Valid Tokens” as Proof of Intent This is a common blind spot. A valid token proves identity. It says nothing about intent, context, or correctness. With agents, that gap is where misuse hides, because the questions that actually matter are not answered by authentication at all: Who triggered the agent?What task was it supposed to perform?Does this specific request align with that task?Is the behavior consistent with how the agent normally acts? Consider an agent that normally retrieves a handful of records and suddenly starts pulling large volumes of sensitive data from unrelated domains. Nothing in the token changes. Every request is “valid.” The behavior clearly is not. How to enforce intent: Carry the task context in the token itself, then check the request against it at the gateway. A delegated, on-behalf-of token (for example via RFC 8693 token exchange) lets you bind three things together: the user the agent is acting for, the agent doing the acting, and the declared purpose of the task. JSON { "sub": "user:4821", // the human on whose behalf "act": { "sub": "order-status-agent" }, // the agent "purpose": "order-status", // the declared task "scope": "orders:read customers:read" } Then enforce a context-aware policy at the gateway: native conditional logic, or an external policy engine such as OPA/Rego that rejects a request whose action does not match its declared purpose, even when the token is valid: Plain Text deny if request.action == "refund" and token.purpose != "refund" deny if request.path ~= "/admin/" and token.purpose != "admin" allow if request.scope covers request.path+method The trade-off: This requires token-exchange infrastructure and a policy set someone maintains as tasks evolve. The payoff is that “technically valid” stops being a free pass — the gateway now understands what the agent was authorized to do, not merely who it is. 4. Watch Behavior, Not Just Requests Traditional API security is signature-driven: invalid token, malformed request, known attack pattern. Agent traffic rarely looks like that. Most of the time, every request is syntactically correct, and that is exactly the problem. Signature-based detection is structurally blind to an attack made entirely of well-formed requests. What you need to watch is behavior over time. It helps to think about agent behavior across a few dimensions, and to baseline each one per agent identity: Velocity: request rate against the agent’s own normal, not a global threshold.Sequence: whether the agent is calling endpoints in an order it has never used before.Data volume: how much data a session pulls relative to its baseline.Resource novelty: whether it is suddenly touching resources or domains it never has.Delegation consistency: whether the on-behalf-of user and declared purpose still match the pattern of activity. The failure is almost never a single bad request; it is the pattern: Plain Text Normal : 15–30 customer lookups per day Abnormal : 5,000 lookups in one hour No single request is "wrong." The pattern is. Where detection runs: You have three broad options, trading latency for enforcement power. Offline analytics on gateway logs is easy to add but only catches problems after the fact. Streaming detection reduces that lag to near-real-time. Inline detection sits in the request path and can actually block at the cost of adding latency to every call. Many teams run inline detection for high-consequence agents and streaming for the rest. What to do on a breach: Decide the response ahead of time: alert only, throttle the offending agent, or quarantine its credential outright until a human reviews it. For an autonomous system, credential quarantine is often the right default; it contains the blast radius without waiting for someone to wake up. The trade-offs: Behavioral detection has real failure modes. Cold start: a brand-new agent has no baseline, so treat its first days conservatively. False positives: a legitimate batch job or a new feature can look like an anomaly, so keep a fast path to whitelist expected changes. And adversarial slow-drift: an agent (or whoever controls it) can creep behavior upward gradually to move the baseline, so anchor some limits to absolute business ceilings, not only to relative baselines. 5. Make Every Action Traceable End-to-End When something goes wrong with an agent, the first question is always the same: what exactly happened? If you cannot answer that quickly, you do not have enough observability. At minimum, you need to reconstruct which agent executed a call, which user it was acting for, what API it invoked, what decision drove the call, and what data it accessed or modified. How to plumb it. Propagate a single correlation identifier across the entire chain: agent, gateway, and backend using W3C Trace Context so every hop shares one trace: JSON traceparent: 00-4bf92f3577b3we20e0e4736-00f062b7-01 audit record (one per gateway hop): { "trace_id": "4bf9f3577a6a3ce929d0e0e4736", "agent_id": "order-status-agent-prod", "on_behalf_of": "user:4821", "method_path": "GET /orders/9931", "purpose": "order-status", "decision": "allow", "data_scope": "order:9931" } Two details separate real traceability from a pile of logs. First, every record must carry both the agent identity and the on-behalf-of user; an agent-only log cannot answer “whose request was this?” Second, capturing why the agent made a call (the reasoning step or tool-selection decision) requires instrumenting the agent side, not just the gateway; the gateway sees the call, but only the agent knows what prompted it. Telemetry spans on both sides, correlated by trace ID, gives you the full picture. The trade-off: Rich traces mean sensitive data and PII in your logs, so plan up front for redaction (masking sensitive fields before they are written), access control, and retention limits and budget for volume, because agent traffic produces far more log lines than human traffic. Putting It Together: One Request Through Five Controls Follow a single order-status agent call through all five, first when it behaves and then when it drifts. The legitimate call. A customer asks about an order. The agent receives an on-behalf-of token (Control 3) whose purpose is order-status and whose scope covers reads on orders and customers. It calls GET /orders/9931. The gateway confirms the endpoint is in the agent’s product (Control 1), that the call sits inside spike and quota limits (Control 2), and that the action matches the declared purpose (Control 3). Behavior is on-baseline (Control 4). The gateway writes an audit record tagged with the trace ID, agent, and user (Control 5). The call succeeds. The drift: Now the same agent, through a bug, a bad tool call, or manipulation, attempts POST /refunds and then starts pulling thousands of customer records. Control 1 blocks the refund outright: the endpoint is not in the agent’s product, so the gateway rejects it before any backend sees it. Even if it were reachable, Control 3 would deny it because the token’s purpose is order-status, not refund. The record-pulling spree stays syntactically valid, so Control 2’s quota trips first and returns 429s, and Control 4’s baseline flags the volume-and-novelty anomaly and quarantines the credential. Throughout, Control 5 leaves a complete, correlated trail so the post-incident question is answered in minutes, not guessed at. No single control catches everything. Together they turn an autonomous system from a trusted insider with a valid badge into an identity that is scoped, bounded, checked for intent, watched, and recorded. Final Thought Most of the risk around AI agents does not come from exotic attacks. It comes from familiar gaps: over-permissioned service accounts, missing or weak rate limits, blind trust in tokens, no behavioral monitoring, and poor observability. AI does not break these rules; it exposes where they were never enforced properly, because it exercises them at machine speed and scale. If you treat agents as just another integration, you will eventually run into trouble. If you treat them as autonomous identities that need tight governance from day one — scoped, rate-limited, intent-checked, behaviorally monitored, and fully traceable at the gateway, so the risk becomes manageable. The fundamentals have not changed. The scale and speed have. A useful place to start: pick one agent already running against your APIs, and check how many of these five controls it is actually subject to today.

As a data engineer, I’ve noticed business teams submitting intake forms, compliance documents, and project proposals that a tech team then manually validates against a set of predefined business rules stored in a database that gets updated quarterly. The time it takes to validate a single form is typically in the hours, and by the time you’ve validated the form, the submitter has moved on to other work. When I needed to validate project intake forms against 60+ business rules of financial, compliance, and other types of business rules and guidelines (some of them to be used in a deterministic way and others to be used in a more nuanced manner), I knew that a simple if-else logic-based manual review process would not scale. This article walks through how I developed an async, AI-powered validation API with AWS Bedrock Agents and Serverless Architecture to process and validate intake forms within 60 seconds without blocking the user. The architecture also manages cross-account authentication to get access to the AI-powered engine and shows failure recovery gracefully. Why Async? The Problem With Synchronous AI APIs Integrating AI into an API synchronously means users send a request, the server processes it, and returns results in one HTTP response, but many systems that use AI-powered validation take more than 30 seconds. The AI agent I built was taking anywhere from 30 seconds to 1 minute to evaluate all of the form fields for all the applicable rules and conditions. But the hard limit for the API Gateway is 29 seconds (HTTP timeout). One approach to make this API request work is to transform the synchronous request and response into an async request with a subsequent background processing step and poll the results from a separate endpoint. This can be implemented as follows: Client submits the form via POST, receives a request_id immediately (under 2 seconds)Validation runs asynchronously in the background (30–60 seconds)Client polls a GET endpoint with the request_id until results are ready By making the form submission step separate from the AI validation of that form in the background, users can continue working on other tasks instead of being stuck staring at a page waiting 30 to 60 seconds for the form to be validated. Architecture Overview As a data engineer, I was required to tackle three main challenges to create a production AI validation API: 1) the frontend application is deployed in a different AWS account, 2) AI agent-based form validation is extremely computationally expensive to run, and 3) business rules for this type of validation are likely to change from time to time without API code deployment. The architecture consists of five components: API Gateway (REST API): With Cognito Authorizer for cross-account JWT authenticationAsync Handler Lambda: It’s an entry point for the API. An Async Handler Lambda function is invoked by a POST request. It will store the form payload on S3, then trigger the Validation Lambda function and store an initial "processing" status in S3. The function immediately returns a request_id to the frontend client within 2 seconds.Validation Lambda: This function loads up all the rules for a given request from S3. It then builds up all the prompts for the Bedrock Agent and runs the Agent. The results of the Agent are then saved off in S3 for the Polling API.Polling Lambda: Handles GET requests and checks S3 for completed resultsRules Sync Lambda: Separate independent process to read validation rules from the data warehouse using EventBridge scheduler and sync to S3 for validation with AI model. Implementation: The Async Handler The async handler is the entry point. Its task is quite straightforward. It accepts the payload, stores it, triggers the Validation Lambda function, stores an initial "processing" status in S3, and returns the “processing” status with the request ID to the client. The function does all of this within a couple of seconds. Here is the core implementation: Python import json, boto3, uuid from datetime import datetime s3 = boto3. client(' s3') Lambda_client = boto3. client('Lambda') S3_BUCKET = 'my-validation-bucket' VALIDATION_LAMBDA = 'ai-validation-function' def lambda_handler(event, context): payload = json. loads (event. get ('body', "{}')) request_id = str(uuid.uuid4()) # Store initial processing status s3.put_object( Bucket=S3_BUCKET, Key=f'validation-output/(request_id)/status.json', Body=json.dumps({ 'request_id': request_id, 'status': 'processing', 'submitted_at': datetime. ttenew() .isoformat() }) ) # Fire-and-forget: invoke validation async pay Load ['_request_id'] = request_id lambda_client.invoke( FunctionName=VALIDATION_LAMBDA, InvocationType='Event', # Async invocation Payload=json. dumps (payload) ) return { 'statusCode': 202, 'body': json. dumps ({ 'request_id': request_id, 'status': 'processing' }) } In the above code snippet, I specifically invoke the validation lambda from the async handler by setting the InvocationType='Event'. This allows the async handler to return immediately to the frontend with the request_id for the submitted request. The Validation Lambda will then complete asynchronously and store the results in S3. Implementation: The Polling Handler The Polling Handler Lambda function manages the GET endpoint; it polls S3 for the updated status file and returns the current status of Validation Lambda processing: completed or failed. Here is the core implementation: Python def lambda_handler(event, context): request_id = event['pathParameters']['request_id'] try: status_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/status.json' ) status = json.loads(status_obj['Body'].read()) if status['status'] == 'processing': return {'statusCode': 200, 'body': json.dumps(status)} # Completed - return full results results_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/results.json' ) results = json.loads(results_obj['Body'].read()) return {'statusCode': 200, 'body': json.dumps(results)} except s3.exceptions.NoSuchKey: return {'statusCode': 404, 'body': 'Request not found'} S3 Decoupling: Using S3 as an intermediary between the validation Lambda and the polling handler allows for natural decoupling. The validation Lambda writes the results of the validation to S3, and the polling handler reads from S3 to return the latest status to the frontend. There is no shared state between the validation handler and the polling handler; there are no database connections, and there are no race conditions. Integrating the Bedrock Agent for Intelligent Validation An intelligent validation function would need more than just a set of rules to check for requirements and best practices. There are a lot of judgment calls that a human would make based on examples of how a policy or guideline would be applied in real life. To achieve that, the more effective way is to integrate with an existing AI function that is designed to handle a wide variety of scenarios and functions The Bedrock Agent architecture solved this by combining: Knowledge base: Containing policy documents, guidelines, and past examples of work for the intelligent validation to reference during the evaluation process.Dynamic prompts: The prompts for the AI model are built dynamically from the current validation rules. These are loaded from S3 as a JSON file and then injected with the current values for the specific field being evaluated.Structured output: Parse the assessment’s pass/fail status, confidence in the assessment, and a set of detailed recommendations made by the agent. The prompt for the AI agent is generated at runtime by the validation function. The rules are loaded from S3 earlier in the function's execution. Here is an example prompt: “Evaluate field [Project Justification] with value [user input] against rule: The justification must clearly describe the business problem being solved and include quantified impact. Reference the knowledge base for examples of approved justifications.” The AI returns a structured assessment of whether or not the field has passed validation, the confidence that the AI has in the assessment, and recommendations. Dynamic Rules Management: Keeping Rules in Sync Without Code Deploys Rules typically change on a monthly or quarterly basis by the business teams. To keep up with the current policy, the rules must be separate from the rest of the application code. To achieve that, I used Rules Sync Lambda, triggered daily by EventBridge: EventBridge fires at 6 AM daily.The Rules Sync Lambda queries the Data Warehouse (Redshift) for the current validation rules for the application.It also takes a copy of the most current version of the rules in S3 for purposes of rollback.It transforms and then uploads the new rules file to S3 as a new copy of the Validation_Rules.json file.Upon failure to update the rules in S3, a CloudWatch Alarm is triggered, which in turn triggers an SNS notification to the appropriate engineering team. The rules are managed as a database of rules (as opposed to being stored within the application code), which allows business analysts to easily update the rules on a quarterly basis without requiring any code changes or deployments. Cross-Account Authentication With Cognito In this case, the frontend application and the AI backend were set up in two different AWS accounts. When deployed within different accounts (as within an enterprise), cross-account authentication is required. Since the frontend application was already authenticated against a company’s SSO (Single Sign On) using Cognito, it was only a matter of how to reuse these tokens within another account without involving the Frontend team for changes. The solution was to create a Cognito Authorizer and attach it to a REST API created in the API Gateway. This API can then be set up to trust the User Pool from the frontend account. Below is a simplified representation of this configuration: API Gateway REST API with a Cognito Authorizer pointing to the frontend account’s Cognito User Pool ARN.CORS (Cross-Origin Resource Sharing) configuration for only that frontend domain.The frontend application is already authenticated with CognitoThe backend application accepts the tokens that the frontend application is using for authenticationThe frontend application simply sends the existing Cognito tokens that the frontend application already has created in the authentication process From the frontend team’s perspective, this was a simple implementation that required them to send the existing Cognito token with the request and to implement a polling loop for the GET endpoint. Results and Lessons Learned After deploying to production: Validation time: reduced from 2 -3 hours (manual) to less than a minute (automated)API response time for form submission: less than 2 seconds for GET API using an async pattern, meaning the frontend never has to wait for the backend60+ validation rules: per form, including both deterministic and AI-judgement rules Zero code deploys: for changes to the rules, which are stored in the database, sync daily Key lessons as a developer building this: Design for async from the start: Retrofitting a synchronous API to be async is very hard. If your AI inference takes more than 5 seconds, which is generally the case, then design your API to be async from day one.Use S3 as your state machine: S3 is the simplest, cheapest, and most reliable way to pass results between decoupled Lambdas. No databases, no queues, no DynamoDB for this pattern.Separate dynamic rules from code: Separate process for managing rules which are dynamic and change often to avoid deployment bottleneck Bedrock Agents are good for making judgment calls. If you have a deterministic check (is a field empty), then you can code that. But for a judgment call (does a justification make sense), then use an AI agent to make the call. Conclusion There is an entirely new way to approach the request lifecycle for APIs in this AI-powered validation API development. The asynchronous API with polling for validation is better than simply trying to work around the timeout limits of APIs. Bedrock Agents, along with S3 to manage the state of the workflow and EventBridge to synchronize rules on a daily basis from a database created by business users via a simple UI created by frontend team, while backend team does not need to write any code for new rules, all integrated together to form complex data validation system powered by AI-powered judgment calls while maintaining simple to deploy and scalable system. As a data engineer, there’s nothing quite like watching hours of manual work by a reviewer get compressed down into 60 seconds or less of automated work while maintaining the high level of evaluation that a business stakeholder expects.

In a lot of organizations, the real integration platform is a person. Someone exports orders from the ERP every morning and pastes them into the planning tool. Someone else re-types customer updates from the CRM into the invoicing system. It works until that person is on holiday or makes a typo in a price field or the volume doubles. Replacing that manual work with a synchronization service sounds like a junior-level task: read from system A, write to system B, schedule it, done. In practice, sync services are where many integration projects quietly fail. They fail not because moving data is hard, but because the edge cases are partial failures, retries that duplicate records, two systems that both think they own a field, and errors that nobody notices for three weeks. This article walks through the design decisions that separate a sync layer you can trust from one you learn to fear. The examples use Python and pseudo-SQL, but every pattern here is language-agnostic. Decision 1: One Source of Truth Per Entity The single most important design decision in any sync architecture is not technical. It is organizational: for every entity, exactly one system is allowed to win. Orders live in the ERP. The webshop may create them, but once created, the ERP's version is the truth, and the webshop displays what the ERP says. Customer contact details live in the CRM. The ERP receives updates from the CRM and never edits them locally. The moment two systems can both modify the same entity and both push their version, you have built a conflict generator. Last-write-wins will silently destroy data. Merge logic will grow into an unmaintainable swamp of special cases. The fix is almost never smarter conflict resolution. It is removing the conflict by assigning ownership. Write this down as a table before writing any code: EntityOwnerMay createMay updateOrderERPWebshop, ERPERP onlyCustomer contactCRMCRMCRM onlyProduct/pricingERPERPERP onlyStock levelERPERPERP only If you cannot fill in this table, you are not ready to build the sync. Any cell where two systems appear in the "may update" column is a design problem to resolve with the business first, not a technical challenge to code around. Decision 2: Idempotency, or Retries Will Hurt You Your sync will fail mid-run. The network will drop after 4,000 of 5,000 records. The target API will return a 500 halfway through. The scheduler will fire twice. None of these are exceptional; they are Tuesday. The only sane response to failure is retry, and retry is only safe when every operation is idempotent: running it twice produces the same result as running it once. The classic mistake looks like this: Python # Dangerous: creates a duplicate on every retry def sync_order(order): target_api.create_order( customer=order.customer_id, lines=order.lines, total=order.total, ) If this call succeeds on the target but the response is lost (a timeout, a crashed worker), the retry creates a second order. Someone ships it. The fix is to make every write carry a stable, deterministic key derived from the source record, and make the target treat that key as unique: Python # Safe: the natural key makes the operation idempotent def sync_order(order): target_api.upsert_order( external_id=f"erp-{order.erp_id}", # stable key from the source customer=order.customer_id, lines=order.lines, total=order.total, ) If the target system has no upsert endpoint, simulate one: look up by external_id first, then create or update. Wrap that lookup-and-write in one function and forbid every other code path from writing directly. The same rule applies to your own bookkeeping. Store sync state keyed by the same external ID, so a re-run of yesterday's batch is harmless by construction. Decision 3: Pull Changes, Don't Diff Worlds The naive sync reads all records from both sides and compares them. This works in the demo and collapses in production, where "all records" means 400,000 rows over a SOAP API that pages 100 at a time. You need change detection, and there are three workable tiers, in order of preference: The source has reliable updated_at timestamps or a change log. Store a high-water mark after each successful run and query only what changed since. This is the happy path; verify that the timestamp actually updates on every mutation, including the ones done by nightly batch jobs inside the legacy system. Legacy systems lie about this more often than you would expect.The source has no usable timestamps, but you can read all records cheaply. Compute a hash per record and compare against the hash you stored last run. Only records with changed hashes get pushed downstream: Python import hashlib, json def record_hash(record: dict) -> str: canonical = json.dumps(record, sort_keys=True, default=str) return hashlib.sha256(canonical.encode()).hexdigest() def detect_changes(records, stored_hashes): for r in records: h = record_hash(r) if stored_hashes.get(r["id"]) != h: yield r, h Neither is possible. You are down to full comparisons on a schedule. Constrain the entity scope aggressively and be honest with stakeholders about latency. Whichever tier you land on, keep the change detector separate from the writer. A queue between them, even a simple database table with pending / done / failed states, gives you retry, rate limiting, and an audit trail almost for free: SQL CREATE TABLE sync_queue ( id BIGSERIAL PRIMARY KEY, entity_type TEXT NOT NULL, external_id TEXT NOT NULL, payload JSONB NOT NULL, status TEXT NOT NULL DEFAULT 'pending', attempts INT NOT NULL DEFAULT 0, last_error TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), processed_at TIMESTAMPTZ, UNIQUE (entity_type, external_id, status) ); That UNIQUE constraint is doing real work: it prevents the same pending change from being enqueued twice, which keeps the queue idempotent too. Decision 4: A Silent Sync Is Worse Than No Sync Here is the paradox of a working sync layer: the better it works, the more people trust it, and the more damage it does on the day it silently stops. A sync that visibly fails gets fixed the same morning. A sync that dies quietly keeps its consumers confidently reading stale data. Sales quotes yesterday's* stock levels. Finance invoices from last week's prices. By the time someone notices, you are reconstructing three weeks of drift. Minimum viable observability for a sync service is four things: A heartbeat. Every run writes a completion record. An external check alerts when the most recent successful run is older than the expected interval. Do not rely on the sync alerting about itself; a crashed process sends no alerts.Drift metrics. Periodically count records on both sides and compare. The counts will never match perfectly in a live system, so alert on trend, not on exact equality.A dead-letter state. After N failed attempts, a queue item moves to failed and a human is notified with the payload and the last error. Infinite retry loops on a permanently broken record will otherwise clog the queue and mask new failures behind old ones.Readable logs per record. When finance asks why invoice 4482 shows the old address, you want to answer with one query, not a debugging session. None of this is sophisticated. All of it is regularly skipped, because on the day the sync ships, it works, and observability feels like polish. It is not polish. It is the feature that determines whether you find out about failure from a dashboard or from an angry customer. The Shape of the Whole Thing Put together, a trustworthy sync layer is small and boring: Plain Text [ Source system ] → change detection → sync_queue → idempotent writer → [ Target system ] ↓ heartbeat, drift checks, dead letters Two processes, one queue table, a handful of metrics. The value is not in the volume of code; most implementations of this design fit in a few hundred lines. The value is in the decisions encoded in it: one owner per entity, stable keys on every write, changes flowing through an inspectable queue, and failure treated as a normal input rather than an exception. Build it this way, and the sync becomes infrastructure nobody thinks about, which is the highest compliment integration code can receive. Build it as a quick script, and you have not removed the human integration layer at all. You have just changed whose Friday afternoon gets ruined.

Most JMeter test plans I’ve inherited share a common shape. Two hundred threads, one ramp-up, a flat plateau, and a results table that says “p95 was 480ms.” Somebody declares the system performant, the test plan goes into a Confluence page, and nobody runs it again until the next major release. The problem is that the test doesn’t model anything. The traffic shape is wrong, the user behavior is wrong, the data volumes are wrong, and the security controls aren’t being exercised. The system passes the test and then fails in production at peak load because production traffic doesn’t look like the test. This article is about the difference. How to design realistic load profiles, run distributed load that actually scales, and use the test data to find specific security-related bottlenecks (auth latency, encryption overhead, audit logging contention) that the simple test plan never surfaces. Why the Canned Test Plan Misses The default JMeter test plan does three things wrong: It uses constant load. Real traffic has spikes, valleys, and bursts. Constant load only tells you about steady-state behavior. The interesting failures happen during transitions.It uses uniform users. Every thread does the same thing. Real users have a mix of behaviors. Some browse, some search, some submit, some upload. A constant ratio is wrong; the ratio shifts by time of day.It tests on cached data. The first test run hits cold caches. The second run hits warm caches. By the time you’re looking at the results, everything is warm, and you’re measuring cache performance, not application performance. For a clinical system, these issues compound. The peak isn’t a steady 200 users; it’s a Monday-morning admission rush where every clinic is opening simultaneously, plus the lab batch results coming in from overnight, plus the medication reconciliation jobs running. The simple test doesn’t capture any of this. Designing a Realistic Profile The first move: instrument production. Look at actual traffic for 30 days. Pull out the patterns. What I look for: Request distribution by endpoint. What percent of traffic is GET /patients/{id}? What percent is POST /orders? The distribution is rarely uniform.Daily and weekly patterns. Healthcare systems have strong weekday patterns. Morning admission peaks, midday discharge peaks, evening lulls. Weekend patterns are different from weekday patterns.Burst characteristics. What’s the largest 5-minute spike in the last 30 days? How does p99 behavior change during the spike?User session shape. A user logs in, performs some actions, logs out. The actions aren’t random. There’s a typical sequence. From that, the JMeter test plan starts looking different. Instead of one thread group doing one thing, you have multiple thread groups with different behaviors: Thread group 1: Clinician users, doing chart review (heavy reads, light writes).Thread group 2: Admission staff, doing patient registration (medium writes, audit-heavy).Thread group 3: Lab system, posting results (high write volume, batch-shaped).Thread group 4: Reporting, doing aggregation queries (low frequency, expensive). Each thread group has its own ramp-up, plateau, and think times. The combined load looks more like production. XML <ThreadGroup> <stringProp name="ThreadGroup.num_threads">120</stringProp> <stringProp name="ThreadGroup.ramp_time">300</stringProp> <stringProp name="ThreadGroup.scheduler">true</stringProp> <stringProp name="ThreadGroup.duration">3600</stringProp> <!-- Clinician chart review pattern --> <ThroughputController> <stringProp name="throughput">85.0</stringProp> <!-- 85% of these threads do chart review --> </ThroughputController> </ThreadGroup> The Throughput Controller is what lets you mix behaviors within a thread group with realistic ratios. The assumption that burned us wasn’t volume — it was arrival shape. We had an inbound integration with an external system, and capacity planning assumed requests would arrive as they were submitted on the other side: a steady trickle across the day, the same shape as our own front-end traffic. The external system didn’t work that way. It accumulated submissions on its side and dumped the entire batch at once. The capacity model said 5,000 requests per hour; reality was that same 5,000 arriving in a burst measured in minutes. Nobody suspected the integration, because the daily totals matched the model exactly — steady-state capacity was fine, burst capacity wasn’t, and the constant-load test plan we’d been running had never exercised a burst at all. The fix was twofold: decouple arrival rate from processing rate by putting a message bus in front of the integration endpoint, so the batch lands in the queue at whatever rate it arrives and the system drains it at a sustainable pace; and rebuild the test plan to match — the integration thread group now fires its full daily volume in a short window, because that’s what production actually does. On the next run, the burst cleared without touching the rest of the system. The lesson: instrument the arrival pattern before designing the test, or production will run the experiment for you. Distributed Load A single JMeter instance maxes out somewhere around 1,000–2,000 threads depending on the test complexity. Above that, you need distributed load: multiple JMeter slaves driven by a master. The setup: Shell # On each slave node: jmeter-server -Djava.rmi.server.hostname=10.0.1.50 # On the master: jmeter -n -t test-plan.jmx -R 10.0.1.50,10.0.1.51,10.0.1.52 -l results.jtl The slaves run the test; the master aggregates results. The thing that breaks first in distributed mode is the aggregation. If your slaves are generating tens of thousands of samples per second and shipping them to the master, the master becomes the bottleneck, and your test results lag reality. Three settings that matter: mode=StrippedBatch in jmeter.properties. This compresses sample data before shipping to the master. Without it, the network between slaves and master saturates first.summariser.interval=30. Batches the summary updates rather than streaming every sample.Disable graphical listeners during the test. They consume memory and add overhead. Run with -n (non-GUI) and analyze the results file afterward. The other thing distributed mode breaks: tests that share state. If your test plan uses CSV Data Set Config to read user accounts, each slave needs its own copy of the CSV, and you need to make sure two slaves aren’t both using the same user concurrently. Either split the CSV across slaves or use a different uniqueness mechanism (UUID-based usernames, for example). Modeling Auth and Session Correctly Most simple test plans get authentication wrong. They either log in once at the start of the test (which lets the server cache too aggressively) or they log in on every request (which makes the test mostly about login throughput). Real users log in at the start of a session, perform many actions over 30+ minutes, and then log out. The test should match. Plain Text HTTP Request: POST /auth/login → captures access_token via Regex Extractor HTTP Header Manager → Authorization: Bearer ${access_token} Loop Controller (50 iterations of mixed actions) HTTP Request: GET /api/patients/{id} HTTP Request: GET /api/encounters HTTP Request: POST /api/notes Think Time: random 5-15 seconds HTTP Request: POST /auth/logout This shape exercises the actual session lifecycle. It also surfaces token-refresh issues if your access tokens expire mid-session. Most production auth bugs only show up in tests that have realistic session durations. For OAuth flows specifically, the JMeter HTTP Request can do the password grant or client credentials grant directly. For authorization code flows, you usually need a BeanShell sampler or JSR223 sampler to handle the redirect chain. Identifying Security Bottlenecks Here’s where realistic load testing earns its keep. Several of the bottlenecks I’ve found in production load tests are security-related, and they only show up under load: Authentication latency. Every request validates the access token. If the token validation calls back to an identity provider over the network, that’s a hop on every request. Under load, the IdP becomes the bottleneck. The fix is local token validation (JWT signature check rather than introspection) for the hot path.Authorization decision latency. ABAC policy evaluation can be expensive. If the authorization service is calling out to a database for attributes on every request, that’s database load proportional to traffic. Caching the policy decision at a session level (with appropriate TTL) is a meaningful win.Audit log contention. Every PHI access generates an audit event. If the audit log is a synchronous database write, the audit log table becomes a hot spot. The fix is asynchronous audit (write to a queue, batch insert from the queue) or partitioning the audit table by time.Encryption overhead. TLS handshake cost matters at scale. If your load balancer terminates TLS and connection reuse is poor, you’re paying a handshake on every request. Connection keepalive on the client side and sufficient backend connection pool size on the server side are the relevant levers.Rate limiter contention. Rate limiters that use a centralized store (Redis is common) can themselves become a bottleneck. Every request reads and writes the limiter state. Under high load, the Redis instance becomes the gating factor. A pattern I’ve seen play out: the audit log turns out to be the bottleneck. During a ramp test, throughput plateaued at roughly 60% of projected peak, and latency started climbing on read endpoints that should be cheap. Nobody suspected audit, because audit is “just an insert.” But every PHI access wrote a synchronous insert to a single audit table, and the table didn’t have the right indexes for how it was being used. The compliance queries that ran against it — who accessed which patient, over what date range — had no covering index, so each one scanned an enormous and constantly growing table, holding locks and I/O while thousands of inserts per minute queued up behind it. Every request in the system paid for that contention, because every request carried a synchronous audit write in its path. The database wasn’t saturated overall; one table was. The fix was twofold: index the audit table for its real query patterns and partition it by time so scans and index maintenance stayed bounded; and move the audit write itself out of the request path — inserts go to a queue and land in batches, so a slow audit table can no longer slow down a chart view. On the next ramp, the same load cleared projected peak with margin. The audit requirement didn’t change — every PHI access was still fully logged — but the logging stopped competing with the requests it was logging. Test Data That Doesn’t Lie A test that runs against a database with 500 patient records doesn’t tell you anything about a system that will run against 5 million. Database performance is non-linear: index efficiency, query plan choice, and table scan behavior all change at scale. The test environment should have: Production-scale data volumes. Not real PHI; synthetic data at production scale.Production-shape data distribution. If 10% of patients have more than 100 encounters and 1% have more than 1,000, the test data needs that shape.Realistic relationships. Patients have encounters, encounters have orders, orders have results. The relational density affects query performance. The synthetic data generation is its own engineering problem. Tools like Synthea (an open-source synthetic patient generator) produce realistic enough data for most testing. For specific use cases, you may need to generate your own. Don’t use de-identified production data. De-identification is harder than it sounds, and a flawed de-identification means PHI is now in a non-PHI environment. Synthetic is the safer answer. Reading the Results The metrics that matter, in priority order: Error rate. If the test is producing 5xx responses, that’s the first thing to fix. Performance numbers from a test where 10% of requests are erroring don’t represent anything.p95 and p99 latency, by endpoint. Average latency is misleading. The user experience is shaped by the tail. p99 latency that’s 10x p50 latency tells you there’s contention somewhere; it just doesn’t tell you where.Throughput per endpoint. If you ramp load from 100 to 1,000 RPS and throughput plateaus at 600 RPS, that’s the system’s actual capacity. Latency above that point goes vertical.Resource utilization on each tier. CPU, memory, network, disk on the application servers, database, cache. The bottleneck is whichever resource saturates first. If application CPU is at 95% but database CPU is at 30%, you scale the application tier. If it’s the other way around, the application tier scaling won’t help. A common misread of these numbers: application-server CPU pinned at 90% while database CPU sits comfortably around 40%, so the team does the obvious thing — scales the application tier horizontally and re-runs the test. Same throughput ceiling, except now more application servers are pinned. The application CPU isn’t doing application work; it’s churning on connection-pool waits, timeouts, and retries, because the database is the actual constraint. The misleading part is that database CPU looks healthy. The real problem is contention — sessions stack up waiting on locks and I/O for a handful of hot rows, and waiting doesn’t burn CPU. Utilization tells you which tier is busy; it can’t tell you why it’s busy, and busy-waiting on a downstream constraint looks identical to real work on a CPU graph. The wait-event statistics on the database tell the true story in about five minutes, once someone finally looks. The fix isn’t more application servers — it’s resolving the row contention, after which the original server count clears the target load. The lesson generalizes: utilization identifies the bottleneck only when the bottleneck is throughput-bound. Contention hides behind moderate utilization, and you find it in wait statistics, not CPU graphs. Don’t call a bottleneck until you’ve seen what the busy tier is actually busy doing. What to Do With the Results The output of a load test should produce one of three actions: No action. The system handles projected peak load with margin. Document the capacity, archive the test plan, set a calendar reminder to re-run before the next major release.Tuning. A specific bottleneck is identified, the fix is in configuration or code, and the next test run validates the improvement.Architectural change. The bottleneck is structural, and the fix is significant. The load test produces the case for the work; without the test data, the architectural change is hard to prioritize. The mistake I see most: load tests that run, produce numbers, and then sit in a Confluence page with no action. The test isn’t valuable for its own sake. It’s valuable for the decisions it enables. If no decisions came out of the last test, the test was probably not asking a useful question. What I’d Do Differently If I were standing up a load testing program from scratch: Run the simple test first to validate the harness, then throw it away. The first useful test is the one with realistic profiles. Run the test against a production-scale environment, even if that’s expensive. Tests against under-scaled environments produce misleading results. Include the security stack in the test path. Don’t bypass authentication, authorization, or audit logging to “isolate the application.” The security stack is part of the application’s performance. Set explicit pass/fail criteria before the test, not after. “Acceptable” is what you said before you saw the results, not what you negotiated after. Run the test on a regular cadence, not just before releases. Capacity changes as the system evolves. The test that passed six months ago doesn’t necessarily reflect today’s system. The version of JMeter testing I’d put in front of any production system is the one where the results actually inform decisions. Most JMeter setups don’t get there. The ones that do are the ones where the test was designed to model production, not to produce a number for a release checklist.

The Failure You Have Probably Already Seen An enterprise AI agent is deployed against production data. It answers the first ten questions confidently and correctly. Then, on the eleventh question, it produces an answer that looks reasonable but is completely wrong. The team investigates. The model is fine. The prompt is fine. The tool integrations are fine. The problem is buried in the data itself. A field the agent relied on has drifted. A join it assumed existed no longer holds. A quality signal that used to be reliable has silently degraded. This is not a rare edge case. It is becoming one of the most common failure patterns in enterprise AI systems moving from prototype to production. And it points to a simple, uncomfortable truth: most enterprise data infrastructure was built for a consumer we no longer have. I have spent the past couple of years designing agentic AI systems against production data at Fortune 500 scale. What follows is the runtime governance pattern I now design around, and the failure modes it protects against. Who this article is for: This article is for data engineers, platform architects, AI engineers, and governance teams building enterprise agents that depend on production data. It focuses less on prompt design and more on the runtime data controls required to make agent answers reliable. Twenty Years of Data Built for Humans Every large enterprise data platform in production today was designed for human consumption. Analysts, business users, data scientists, and BI teams. Those consumers share a common trait: they exercise judgment. A human analyst looking at a broken dashboard notices it. A data scientist opening a table with unusual distributions asks a colleague. A finance user reviewing a report questions the number when it does not match their gut. Enterprise data governance evolved to support this consumer. Documentation lives in wikis. Quality is enforced by expected-value alerts that a human triages. Lineage is captured at the ETL job level, not the field level. Access is granted through role-based permissions and refined by manual data stewardship. All of this works when a human is at the end of the pipeline. An AI agent is not that consumer. An agent has no judgment. It processes what it is given and returns an answer. If the data is stale, the agent produces a stale answer with high confidence. If the lineage is broken, the agent cannot trace why. If a quality signal exists only as a wiki page, the agent cannot use it. The Four Gaps Most Enterprises Have Across the AI-in-production work I have seen, the same four gaps show up almost every time. Gap 1: Machine-Readable Data Contracts Most contracts exist as documentation, not as programmatic constraints. An agent cannot ask a Confluence page whether it is safe to trust a field. Data contracts need to be enforced at the platform layer, with schema, type, freshness, and quality guarantees expressed as executable rules. Gap 2: Use-Case-Aware Quality Fitness A dataset that is 95 percent complete may be fine for a marketing dashboard and completely wrong for a clinical AI model. Traditional data quality checks are use-case-agnostic. Agentic AI requires quality signals that answer a different question: is this data fit for this specific decision, right now? Gap 3: Field-Level Lineage That Updates in Real Time When a pipeline changes, human consumers get an email. Agents get a wrong answer. Lineage systems need to update as pipelines evolve and expose change signals in a form agents can consume, not just visualize. Gap 4: A Discovery Layer Agents Can Query Most catalog systems are designed for humans to browse. Agents need a machine interface to ask questions like which tables contain the concept I care about, and which of them is authoritative for this domain. Design Principles for Agentic Data Governance Closing these gaps does not require rebuilding the entire data platform. It requires making governance executable in the same path where the agent retrieves data, evaluates context, and produces an answer. Three design principles matter most. Start with the decision, not the data. For each production AI use case, define what a wrong answer looks like and work backward to the data requirements that would prevent it. This surfaces the specific quality signals, lineage nodes, and freshness constraints that matter. Make governance runnable, not readable. Every governance artifact your agents depend on should be programmatically executable at inference time. If a rule cannot be checked in code, an agent cannot use it. Documentation is useful for humans, but for agents it is invisible. Instrument for continuous evaluation. A governance framework that only fires at deployment is not enough. Models drift, data drifts, and use cases evolve. The governance layer needs to continuously evaluate agent outputs against real-world outcomes and flag drift before it becomes damage. Reference Architecture: Runtime Data Governance for AI Agents A practical implementation usually introduces a lightweight runtime governance layer between the agent and the underlying data platform. The goal is not to slow the agent down. The goal is to give the agent a reliable way to ask whether the data behind an answer is safe to use. At a minimum, this pattern includes five components: a data catalog that exposes authoritative sources, a contract registry that stores schema and business rules as executable checks, a lineage service that tracks upstream dependencies at the field and metric level, a quality service that publishes freshness and fitness signals, and an agent guardrail service that evaluates these signals before the agent responds. Runtime flow: User question → Agent → Semantic/data resolver → Governance service → Catalog, contract registry, lineage service, and quality service → Pass/Warn/Block decision → Agent response. Layer Responsibility Example Signal Catalog Identify authoritative datasets and business definitions. Certified source for booked deal value. Contract registry Validate schema, data types, null thresholds, and business rules. Discount variance must use the approved baseline method. Lineage service Track upstream source, transformation, and metric dependencies. Metric changed because a new source was added. Quality service Publish freshness, completeness, anomaly, and fitness scores. Dataset refreshed within SLA and passed threshold checks. Agent guardrail Block, warn, or allow the answer based on governance signals. Answer allowed only if lineage and contract checks pass. The agent should not directly trust a dataset simply because it can access it. Before answering, it should evaluate the data path, the contract status, the freshness window, the lineage change history, and the use-case-specific fitness score. If any critical check fails, the agent should either decline to answer or return the answer with an explicit data reliability warning. How the Runtime Governance Check Works In practice, the check is a short pre-answer step. The agent does not need to understand every governance rule directly. It needs a stable contract with a governance service that can evaluate the data path and return a decision. The user asks a business question.The agent resolves the requested metric, entity, dataset, or semantic concept.The agent calls the governance service with the resolved data assets and intended use case.The governance service checks catalog certification, contract status, lineage changes, freshness, completeness, and use-case fitness.The service returns a pass, warn, or block decision with machine-readable reasons.The agent answers, adds a caveat, escalates, or declines based on that decision. What a Machine-Readable Data Contract Actually Looks Like The abstract idea of a data contract only becomes real when you can point to one that an agent can actually consume. Here is a compact YAML example for a deal variance metric, expressing schema constraints, business rules, freshness expectations, and quality thresholds in a single artifact: YAML contract: dataset: deal.discount_variance schema: - field: discount_variance_pct type: decimal(18,2) required: true calculation: approved_discount_baseline_v2 - field: source_system type: string allowed_values: [crm_v3, revenue_hub] freshness: sla_hours: 24 breach_action: warn quality: completeness_threshold: 0.95 anomaly_score_max: 3.0 lineage: change_window_days: 30 on_upstream_change: require_review With this in place, an agent can call a single governance endpoint before responding, receive a machine-readable pass, warn, or block decision, and either answer confidently, answer with a caveat, or decline. The rule is not buried in a wiki page. It is live at inference time. Example Runtime API Pattern The runtime call does not need to be complicated. A minimal request can identify the metric, dataset, use case, and decision context. The response should be small enough for the agent to use directly in its control flow. JSON POST /governance/evaluate Request: { "metric": "deals.discount_variance_pct", "dataset": "deals.discount_variance", "use_case": "deal_desk_agent_review", "decision_context": "discount_variance_explanation" } Response: { "decision": "warn", "reasons": ["upstream_lineage_changed", "freshness_within_sla"], "agent_action": "answer_with_caveat" } In the agent workflow, this response becomes a control decision. A pass allows the agent to answer normally. A warn allows the answer but requires a reliability caveat. A block prevents the answer and routes the request to review, remediation, or a safer fallback path. Pseudocode: Turning Governance Into Agent Control Flow Python decision = governance.evaluate(metric, dataset, use_case) if decision.status == "block": return decline_with_reason(decision.reasons) if decision.status == "warn": return answer_with_caveat(query, decision.reasons) return answer(query) This is the core shift: governance is no longer a document the team reads during design review. It becomes a runtime dependency that the agent uses to decide whether to answer, qualify the answer, or stop. Runtime Checks an AI Agent Should Perform Before Answering Is this dataset or metric certified for the requested business domain?Has the schema changed since the agent workflow was last validated?Did all required fields meet completeness and validity thresholds?Is the data fresh enough for the decision being requested?Has any upstream lineage changed within a defined risk window?Does the requested answer depend on a metric with multiple calculation methods?Should the agent answer, warn, escalate, or decline based on the governance outcome? This does not require a heavyweight approval workflow for every query. In many cases, the runtime check can be a fast metadata call that returns a simple decision: pass, warn, or block. The important design principle is that governance must be available in the same execution path as the agent response, not in a separate documentation process that only humans can interpret. Failure Modes and Runtime Controls Failure mode What causes it Runtime control Stale answer Dataset missed its refresh SLA. Freshness check with warn or block behavior. Wrong metric Multiple calculation methods exist for the same business concept. Contract and semantic registry validation. Silent lineage change An upstream source or transformation changed after validation. Field-level lineage check within a defined risk window. Misused dataset The dataset is accessible but not certified for the requested domain. Catalog certification and use-case fitness check. Incomplete evidence Required fields fail completeness or validity thresholds. Quality service decision with explicit failure reasons. A Concrete Example From the Field On one enterprise AI project in a regulated environment, we deployed an agentic assistant to help analysts explore a large deal registration and booking dataset. Early testing looked solid. Several weeks into production, the agent began returning confidently wrong answers about a specific discount variance metric. The model had not changed. The prompt had not changed. What changed was an upstream ingestion job that added a new source that computed discount against a different price baseline. A human analyst would likely have questioned the number because it felt off. The agent did not. It saw a valid number in a valid field and reported it as authoritative. The fix was not in the model. We added a machine-readable contract for the approved discount baseline, a lineage signal for recent upstream changes, and a runtime check the agent could call before answering. After that, the same failure could not recur silently. The agent either answered correctly or flagged that the underlying data had changed and required review. The lesson was not that agents are unreliable. It was that agent reliability is a property of the data layer, not the model layer. Once we treated the governance layer as an active runtime dependency instead of static documentation, the entire class of silent-failure risk collapsed. Implementation Considerations Cache low-risk governance decisions to reduce latency, but recheck high-risk metrics at runtime.Separate warn rules from block rules so agents can still answer safely when risk is explainable.Version data contracts alongside pipelines, semantic models, and metric definitions.Log every agent answer with the governance decision, reasons, dataset version, and lineage snapshot used.Start with high-risk metrics and regulated workflows before expanding the pattern across the broader data estate. Why This Belongs in the Architecture, Not the Prompt Prompt engineering can reduce some surface-level errors, but it cannot solve a missing contract, stale dataset, broken lineage path, or ambiguous metric definition. Those failures sit below the model. They need to be handled in the platform architecture, where data access, metadata, quality, lineage, and policy decisions are available at runtime. For teams building enterprise AI agents, the practical takeaway is straightforward: treat runtime governance as part of the agent stack. If an agent can call a retrieval service, vector index, SQL endpoint, or workflow tool, it should also be able to call a governance service before committing to an answer. The next generation of enterprise AI reliability will not come only from better models. It will come from data platforms that can tell agents, in real time, whether an answer is safe to give. About the Author. Avinash Maddineni is a Lead Data Engineer with 15 years of enterprise data infrastructure experience across healthcare, financial services, energy, and travel. He builds agentic AI and data governance systems at Fortune 500 scale and is founder of PureStrokeAI (USPTO provisional patent filed May 2026).

Recently, I made a comment about the idea of there being a “best” monitoring tool: In fact, let’s get this out in the open: There simply isn’t a singular “best” monitoring tool out there any more than there’s one singular “best” programming language, or car model, or pizza style.* There isn’t a single tool which will cover 100% of your needs in every single use case. The comment got some pushback, both privately and in a few forums, and so I wanted to dig into what I meant and why I feel that way. But before I do that, I want to set the record straight: I stand by what I said about there being no “best” monitoring tool. But I was flat-out lying about the other stuff. The hills I’m willing to die on are: The best programming language is PerlThe best car is the 1967 Ford Mustang 390 GT/AThe best pizza style is deep dish, and I’m partial to getting it from Tel Aviv Kosher Pizza in Chicago With that cleared up, let’s get back to monitoring and observability. Zoom Zoom What really got me thinking about the false concept of a “best” monitoring tool was a video my son shared with me, comparing a Lucid Air Sapphire, a Bugatti Chiron, and a Tesla Plaid to see which was the fastest production car of all time. Disclaimer: I am NOT a car person. My son Kaleb (who is 21, in his 2nd year of university to become a mechanical engineer, and on two different Baja SAE teams) very much is. After watching the video, Kaleb pointed out that if the race (which was on a quarter-mile track) had been a half-mile instead, the Bugatti would have won, hands down. This was because (reasons. I honestly couldn’t follow the things he was saying at this point. I leave it to the reader to imagine Star Trek-like technobabble) My point in all this is that “best” — the fastest car in this case — is highly subject to other variables. The type of track (this was on a track that had been pre-treated with VHT, which makes it super sticky and affects traction), the distance, even things like altitude and weather — these all can impact the ultimate outcome. But I’m talking about more than external factors. What is “best” can be affected by the ultimate use case. Cost is the easiest one that comes to mind. Yes, a Bugatti might be fastest. But possibly the “best” car is a Honda Civic because you value cost and reliability over speed. Or perhaps a Kia Sedona might be “best” because you need more seats. Or a Ford F150. Or a Ryder 16-wheeler. This all relates to monitoring and observability in ways that are both important and, sadly, novel to a lot of IT practitioners. We get so caught up in the “speeds and feeds” aspect of tools and solutions — how many flows per second, how many traces per collector, maximum ingest before backpressure occurs — that we often fail to stop and say “Do I need that? Will I ever need that?” I shared this with my friend Kevin Sparenberg (another car guy, who writes occasionally on ), and he added: Monitoring tools are like any other tool. You have your favorite hammer, but it’s not appropriate in every scenario. You don’t (or shouldn’t) use a sledge for setting a nail, nor would you use a claw hammer for forging. You might have a favorite, but you need to consider the type of work you need accomplished. (Besides being a car guy, he’s a home repair weekend warrior. This is just one of the many reasons why we’re friends.) “Best” Isn’t Always Best Part of the blame can be placed at the feet of vendors. I’ve worked at a few, and it’s rare to find one that has the tools to help customers quantify volume and cost before implementation. Most simply say “let’s just get this installed, and we’ll see where it lands, and we can tune from there,” blithely ignoring the way the level of effort (not to mention political maneuvering in the C-suite) to implement a new tool makes sunk-cost fallacy a near-certainty. But that’s only part of the blame. The other part rests at our feet — the monitoring engineers who need to better shoulder the responsibility of understanding and speaking for the needs of our organization. Because if we don’t, who will? Much of this comes back to things I’ve already ranted about: If we don’t have a plan for the monitoring and observability data being collected, any cost is going to seem to be too much. If you have a plan, you’ll know exactly how much the data is worth, and be able to evaluate the cost of a tool. Learn to speak the language of business, to frame things NOT in the technical terms that you find familiar and comfortable, but in terms that make it clear to the business why a new tool is needed and the value it will provide. Solve the problems your organization is actually having. Once again, it’s easy to get swept up by a vendor’s vision. But if your company isn’t having any of the problems that vendor vision describes, it’s all wasted time and money. Kevin added another nuance. Hyper-focusing on a single metric isn’t just sloppy; it can lead to real problems down the road: I would even go so far as to revisit that zero-to-60 metric. What about zero-to-60-to-zero? Sure any car can get to 60, but which car is able to do that AND return to status quo quickly. If you need a good “bad” example, look at the Gen 1 for Dodge Viper. As my dad said, “plenty of giddy-up, virtually no whoa.” How does that translate to monitoring and observability? Think about alerting. LOTS of tools are able to detect and trigger alerts based on extremely specific (and sensitive) triggers. But far fewer have the controls to detect and stop alert storms. My point in all this is to remember that “best” always has to be weighed against YOUR values: Your real-world, actual business needsYour cost-to-benefit ratioYour team’s skillsetsYour tolerance for toil and effort during the transitionYour willingness to support one more tool in perpetuity…and so on. Good Enough is Usually Good Enough There is a story that is equal parts old, hilarious, and fake. It involves a hapless young man (it’s ALWAYS a dude) who decides to mount a JATO rocket to his car to see just how fast he can go. And in the ensuing chaos, this young man (supposedly) earned himself a posthumous Darwin Award. (Once again, I have to emphasize that this story is 100% fake and was even debunked on the very first episode of MythBusters) However, one of the aspects of the story that makes it funny (at least in my opinion) is the sheer ridiculousness of it all. Sure, lots of folks want a fast car. Many of those folks are willing to spend a little extra for a car that’s a little faster than the norm. Fewer (but not zero) people might also be willing to go to great lengths to acquire not only a fast car, but “the fastest” car. But strapping a rocket to the top of an old car — one that was clearly never intended to be used this way? That is some serious janky automotive slapstick. It tickles our funny bone by evoking those grainy sepia-tinted images of early 20th century “flying machines” that were nothing more than 2 umbrellas strapped to a piston. As the final image of the JATO story fades in our mind, we can almost see the end-card saying “You just couldn’t leave well-enough alone, could you?” Likewise, we need to foster a habit of self-restraint and technical reflection in our monitoring discipline. We have to recognize when our excitement about a tool’s ability to process 8 million log messages a second clouds our own ability to step back and say “why do I even HAVE 8 million log messages a second?” And if there’s a good reason for that volume, follow up with the question “Why do I need to send every single one of those messages across the internet to a vendor’s storage?” I’m not saying there’s nobody in the world who might need that. You might have a good reason. I just want to suggest you take a second to consider things before you end up creating your own JATO-powered observability disaster. The Solution Is Both-And, not Either-Or This is the thing most vendors don’t say, at least admit within earshot of investors and members of the board, because it flies in the face of the marketing hype and sales pitches they’ve worked so hard to craft. You need more than one monitoring and observability solution. You need to decide which systems (and more fundamentally, which data on those systems) use each tool. You’re going to have to split your budget (time, effort, skills, money) between those tools. This is an ugly but unavoidable truth. Over 27 years working with monitoring and observability tools, the number of times I’ve seen a company that had exactly one monitoring solution is: zero. Even the ones that insist they do, after a little digging, have at least a few pockets of the environment that use something else, whether that’s a class of system (mainframes, minis, Windows NT servers, 6509’s); an organization or team that just went their own way or was acquired but never fully integrated; or a location that — due to their distance from the main organization, either functionally or geographically — has to maintain their own set of tools. And most orgs don’t have “at least a few pockets”. They have a full suite of overlapping solutions. You are going to have — more likely you already DO have — multiple monitoring solutions in place. This is one of those tech realities which is simple, but not easy — like supporting more than one operating system (whether servers or desktops); or moving from branch-based development to feature flags; or building a multi-language, multi-cloud app. It’s a cost of doing real business in the real world. Observability, like life itself, is messy. It also, to quote Ian Malcom, finds a way. So will you. Here’s how: Plan and prepare to identify data types based on complex criteria: it might be a combination of location, system type, data type, and even time frame. Expect that, based on those parameters, you’ll then filter and transform the data before sending it in the correct direction. Also expect that some data types are so valuable, you’ll end up sending the same data in more than one direction. But also prepare to put boundaries in place so you aren’t doing that all the time, because that gets expensive fast. As I said, you need to be ready to support multiple tools, but you should have a plan in place for how you’ll keep track of those tools, identify their primary use case, and even set boundaries on the things they will NOT be permitted to monitor, so your stable of solutions doesn’t explosively get out of control. One way to do that is to set, for every data type or use case, a definitive choice for a primary tool that handles that data, and a secondary that you use as a gut-check. For larger organizations or more important data sets, you might have a tertiary, but draw the line there. Another way (complementary to the first) is to understand that some tools are cheap and do a lot of things mostly ok, so you can spread them like peanut butter across the enterprise; while others are expensive (in time, effort, or money) and only do certain things well. So you should spread THOSE like caviar — only on the systems where they’ll do the most good. Finally, differentiate between management tools that also have a monitoring component and true monitoring and observability solutions. You shouldn’t get rid of management tools, but you also shouldn’t make them the primary source of truth for enterprise monitoring information because they are usually so vendor- or system-specific, and it will lead (again) to that explosion of tools I cautioned against earlier. A Brief Buyer’s Guide A natural question to ask next is “how do I know WHICH tools to get?” Once again, my buddy Kevin has some wise observations: Is it even worth mentioning bake-offs? Do people even do that anymore? Maybe tool A has these features, but tool B has these other ones. And they both share some other capabilities. We need all of it, but can’t get it in one package. There’s nothing wrong with that. Leon’s point about having multiple tools is on target. Bias your decisions to picking the right tool for the right job (but at the same time, try not to collect too many tools. This ain’t Pokémon). Don’t buy for “this tool has this neat feature we don’t need, but maybe someday we’ll want.”. Buy (or deploy) for what you need now and in the near future. IT (and the business) is always growing and evolving. What you THINK is important today may be irrelevant in six months. Hopefully not, but thinking too far ahead — the infamous “five year plan” is just a waste of your effort and time. Taking a Victory Lap The point of this blog is pretty simple: There’s no such thing as “best”, and that goes for everything from cars to programming languages all the way to observability solutions. But more essential is my point about WHY there isn’t a single, specific “best” — it’s because context matters. Use case matters. To be more nuanced, there probably is a “best,” but what is best is extremely particular to you and your circumstances. So the lesson in all this is to make sure you are clear about those circumstances, and that you’re always weighing them against the “we’re the best” marketing hype you’ll hear from many vendors.