Microservices

A live production integration case study. Introduction and Purpose of This Article This article is written for mid- and high-level managerial and technical decision-makers. I am the author of the open-source Java library MgntUtils. The article presents an analysis of a real integration of the stack trace-filtering feature from that library into a live commercial production environment. A few important clarifications up front: This is not a side-project pilot and not a lab demo. The feature was integrated into a production service of a company that serves a high volume of real customers. Due to legal constraints, I am not at liberty to name the company.This is not a how-to article for implementers. If you came looking for code samples or logging-framework wiring, please see the dedicated articles listed in the Disclaimer below.MgntUtils can be used in Java projects and in other JVM-based languages such as Kotlin. Before diving into the production numbers, it is worth stating briefly what the feature does and why those numbers matter. Server-side stack traces are usually full of framework and infrastructure noise — proxies, filter chains, containers, thread pools, and similar boilerplate — while the few lines that actually explain the failure are easy to lose in the pile. The MgntUtils filtering utility keeps the application frames and the exception / Caused by chain, and collapses that noise. The result is a much shorter stack trace without losing the information you actually need. When those stack traces are later consumed — sent to an LLM for analysis, or opened by an engineer — that reduction can mean: Substantial AI token savingsTypically more accurate AI root-cause answers, because the model has less framework noise to latch onto and hallucinate aboutA meaningful productivity boost for human triage The rest of this article focuses on what was observed after integrating this feature in production: the measured benefits, how to interpret them, and the integration experience itself — including gotchas that only surfaced in a real live environment, as opposed to a pilot project. Disclaimer This article deliberately does not discuss the technical design of stack trace filtering or the technical details of the integration. Each of those topics has its own dedicated article: Filtering Java Stack Traces With MgntUtils Library DZone: https://dzone.com/articles/filter-java-stacktrace-mgntutilsDEV Community: https://dev.to/mgantman/java-stacktrace-filtering-utility-1c1i Zero-Code-Change Stack Trace Filtering for Spring Boot: An Infrastructure-Level Integration DEV Community: https://dev.to/mgantman/zero-code-change-stacktrace-filtering-for-spring-boot-an-infrastructure-level-integration-3fk5 Production Results and Benefits Below are the observations and conclusions from monitoring the live production system after the feature integration. The feature had been running for about a month, and filtering was also temporarily turned off for comparison. What the Production Environment Looked Like Anonymized sketch of the deployment (enough to judge fit, without identifying the company): High-traffic JVM/Spring Boot service in a commercial production estateStructured JSON logging to a major observability platformObservability billing dominated by per-event (not per-byte) pricingIn a typical production day, that service emitted on the order of ~70,000+ log events carrying a stack trace That is a large stream of stack trace payloads — expensive if fed to an LLM, and tiring if engineers open them by hand. Stack Trace Volume Reduction Range in Production Filtering was measured across production stack traces with filtering on vs off. Observed size/token reductions typically fell in roughly the ~75%–95% range: Toward the high end (~90–95%): framework-heavy request-handling traces (long security/container/proxy tails)Toward the lower end (~75%+): more application-dense traces, where a larger share of frames is your own code The average reduction on a typical trace in this environment was about ~91%. The table below is a real before/after example — shown so you can see what that looks like in practice: MetricUnfilteredFilteredReductionLines19518~91%Bytes~22,200~1,900~91%Input tokens (approx.)~6,300~540~91%Application framesall (buried in noise)all (kept)no signal lost Every application frame in the business call path was retained; what disappeared was framework and infrastructure noise (proxies, filter chains, container/thread-pool frames, and similar boilerplate). Stack traces tokenize poorly for LLMs — package separators, generated class names, and (File:line) markers all split into extra tokens — so the token reduction tracks the size reduction closely. Root-cause readability was unchanged. In both versions, the failure was identifiable from the application frames and the exception message. Filtering did not remove diagnostic signal; it removed the large majority of the payload that never helped. What Improved AI analysis: cheaper and more accurate (when exceptions are analyzed). For every exception sent to an LLM, the stack trace input payload shrank by roughly ~75–95% depending on the trace shape (~5,800 tokens saved on a typical ~91% trace). That saving repeats for every analyzed event. In an environment where tens of thousands of stack traces are emitted per day, any AI triage, clustering, or “explain this error” pipeline pays that tax over and over unless the noise is stripped first. Cost is only half of the AI benefit. Filtering also improves answer quality. The removed frames are framework and infrastructure boilerplate — identical across many errors and unrelated to the application failure. When those frames remain in the prompt, models often latch onto them and hallucinate a root cause in the noise. With them collapsed, the model is steered toward the application frames and exception message that actually explain the failure — so analysis is not only cheaper, but typically more accurate. Sensitivity calculator (illustrative — not this company’s AI spend). If your org analyzes exceptions with an LLM, you can size token cost roughly as: Plain Text annual token saving ≈ (exceptions analyzed per year) × (tokens saved per exception) × (model input price per token) Using ~5,800 tokens saved per exception (average on a typical ~91% trace) and an illustrative model input price of $3 per 1 million input tokens: Analyzed exceptions / dayApprox. tokens saved / dayApprox. saving / year5,000~29M~$32K50,000~290M~$318K250,000~1.45B~$1.6M Plug in your own analysis volume, your place in the ~75–95% reduction range, and your model pricing. The production measurement that is firm is the observed per-exception reduction range, with application frames preserved. Secondary AI upside: More errors per context window. Because a typical filtered stack trace is so much smaller (~540 tokens vs ~6,300 in the example above), many more distinct exceptions fit into a single model call. That is a capability change, not just a cost saving: cross-error analysis — clustering failures, or asking “what went wrong in the last N hours?” — becomes practical instead of blowing the context window on framework noise. It is secondary to the per-exception token and accuracy benefits, but it matters for any AI workflow that looks at more than one error at a time. Human triage productivity. Engineers reading a filtered typical trace see the full application call path at the top (~18 lines in the example above) instead of scrolling through ~195 lines to confirm there is no hidden nested cause and to piece the business path together. For on-call and incident review, that is a direct readability win. What Changed in Log Volume — and What Did Not It helps to separate event count from bytes per event. Event count did not change. A stack trace is still one log event whether it is 195 lines or 18. If your observability vendor bills per event (or per indexed log line item), filtering does not reduce that charge. In this production environment, that was the dominant billing model — so there were no savings on a per-event bill. Bytes per stack trace event did change. Each filtered stack trace was roughly ~75–95% smaller than its unfiltered counterpart (commonly ~90% for framework-heavy traces). There is a real reduction in stack trace payload size. How much that shows up in total log volume is not deterministic. Overall space / ingested-byte savings depend on what share of all logs are stack traces: Plain Text overall byte reduction ≈ (stacktrace share of total log volume) × (~75–95% reduction on those stacktraces) In this company’s environment, stack traces were only about ~1% of total log volume — which is unusually low (an anomaly for many systems, but what we observed here). Cutting ~90% of that 1% yields only a fraction of a percent of total logs, which is easy to lose inside normal day-to-day traffic variance. That is why aggregate ingested-byte charts did not show a clear step when filtering was toggled. In another organization where stack traces are a much larger share of log volume, the same per-trace cut would produce a more visible space saving. Those savings are real in principle, but variable by workload and not the main point of this case study. The main point here is consumption cost. The firm, repeatable benefit we are highlighting is what happens when a stack trace is analyzed by an LLM or read by an engineer: large payload reduction, same diagnostic signal. Treat log-space savings as a possible secondary effect, sized by your own stack trace-to-total-logs ratio — not as the success criterion for this feature. How to Read These Results as a Decision Maker QuestionAnswer from this production caseDid filtering remove useful diagnostic information?No — application frames and exception chain structure remained.How large is the per-exception reduction?Roughly ~75–95% across production traces (often ~90%+ on framework-heavy request traces).Does that reduce per-event log billing?No — event count is unchanged.Is there space / byte saving?Yes per stack trace (~75–95%); overall only if stack traces are a meaningful share of total logs (here ~1%, so barely visible).Where is the upside for AI analysis?Far fewer tokens and less hallucination on framework noise — cheaper and typically more accurate.AI context-window upside?More exceptions fit in a single context window — useful for clustering or “what failed in the last N hours?” analysis.Other upside?Time saved when humans read errors.Who should adopt it?Teams that already (or soon will) send production exceptions to LLMs at volume, and/or teams whose engineers routinely open noisy stack traces. The production evidence supports a clear, bounded claim: when stack traces are consumed, filtering delivers a large, repeatable reduction in payload size with no loss of application signal. Per-event log bills do not drop. Overall log-space savings may exist but depend on stack traces’ share of total volume — and are not the primary reason to adopt the feature. Integration Experience I started from an implementation I already had in the MgntUtilsUsage side-project repository — a runnable Spring Boot demo of MgntUtils features, meant to emulate real-life apps as closely as possible. It was a very good starting point. Still, as I worked through the live commercial integration, a few gotchas surfaced that a single-JVM demo simply does not force you to confront. Gotchas That Showed Up in a Real Production Environment 1. Feature Toggle Storage Across Multiple Containers My demo app runs in a single JVM. A real production service typically runs on several containers that scale in and out. In the demo, the on/off flag for stack trace filtering lived in memory — which is fine for one process, and useless once you have more than one. In a multi-container environment, you need an external, shared flag holder that every instance can read. Redis (or an equivalent shared store available to all containers) is a good candidate. 2. JSON Logging Adapters, Not Only the Classic Logback Pattern When I first modified the Logback configuration, my demo mainly used conventional Logback pattern-based adapters. A real production app will most likely also use a JSON encoder for external logging systems such as Datadog (and similar platforms). That special adapter has its own throwable-handling path, so wiring the filter there is a must — otherwise you can end up with filtered console output locally and unfiltered stack traces in the system that actually matters. 3. Hardening the Fail-Safe Path A fall-back option already existed for the case where anything goes wrong inside the filtering path. For production, that fail-safe had to be hardened a bit further to make it as bullet-proof as possible: if filtering ever fails, the system must still emit a full standard stack trace and must never drop the log event. 4. Logback Is Not the Only Popular Logging Framework This company uses Logback, so that is what the production integration targeted. But Logback is not the only widely used option — my own favorite, for example, is Log4J. For the dedicated integration article (linked in the Disclaimer), I also had to provide Log4J instructions, even though Log4J was not used in this particular environment. Anyone planning an org-wide rollout should assume more than one logging stack may need to be covered. Effort, Timeline, and Outcome All in all, the integration was smooth, and the side-project was close enough to the final result in the real app. About 4–5 hours to get an integrated version up and running in the staging environmentAbout one day of observing staging to make sure there were no unexpected behaviorsThen deployment to production, with about another day of close monitoring before declaring the feature live So roughly half a day of integration work, and about 1.5 working days of testing / staging observation / production monitoring. Not a single bug was found. There are two contributing factors for that: The stack trace-filtering feature itself is mature and battle-tested — I am tempted to say it has no bugs, but let’s just say it is highly stable and reliable.The integration itself is simple enough. The next integration should be even faster, since this one is now well documented (including the dedicated Spring Boot integration article linked in the Disclaimer). If you are interested in integrating this feature into your project, the detailed integration instructions are in the article Zero-Code-Change Stack Trace Filtering for Spring Boot: An Infrastructure-Level Integration. If you are interested in support for the integration, feel free to contact me at or through my LinkedIn profile. Conclusion This case study supports a simple decision: Adopt stack trace filtering if your organization already analyzes production exceptions with LLMs at a meaningful volume, or if engineers routinely open noisy stack traces during triage and on-call. In those cases, the live evidence is clear: typically about ~75–95% less stack trace payload (around ~91% on a typical trace), with application frames preserved — cheaper AI analysis, typically more accurate answers, and easier human reading. Do not adopt it expecting your per-event observability bill to drop, or expecting a large automatic cut in total log volume. Event count does not change. Overall byte savings depend on how large a share stack traces are of all logs — and that varies by organization. Consumption cost is the main point; log-space savings are secondary and workload-dependent. On effort and risk: in this live commercial integration, getting to staging took about half a day of work, followed by roughly a day and a half of staging observation and production monitoring. No bugs were found. The feature is mature, the integration is simple, and the demo-to-production gaps (shared toggle, JSON logging adapters, fail-safe hardening, and covering more than one logging framework) are now documented. If that profile matches your environment — high exception volume that is actually consumed by AI or by people — this is one of the cheaper, lower-risk improvements available. If exceptions are mostly logged and rarely looked at, the benefit will be thin, and that is an honest reason to pass.

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

If different Docker Engine versions are running simultaneously in a Docker Swarm cluster, this may lead not to an obvious service outage but to a more subtle scenario: partial traffic degradation on individual nodes. In this case, the issue appeared on one of the manager nodes, Traefik started reporting an unavailable status (health=0) for the router-app service, and the cause, according to the working hypothesis, was related to differences in iptables rules and overlay networking between Docker 28.1.1 and 28.2.2. On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time. Context At the time of the incident, the Docker Swarm cluster consisted of 5 manager nodes, several dozen worker nodes, and approximately 40–50 services. External HTTP traffic passed through Traefik, deployed on each of the five manager nodes, and was then routed by Traefik to the backend application containers. One of the key services was router-app, responsible for building public transportation routes on the frontend. It was one of the critical entry-point services with the highest SLA, and any disruption to its availability could have led to severe penalties from the customer, so any deviation in its availability required an immediate response. Grafana dashboards showed application availability through Traefik health-check statuses. Those statuses were generated based on HTTP health-check endpoints implemented by the developers for most services, primarily the most critical ones. This was enough to quickly localize the problem at the ingress traffic level. The cluster had one important characteristic. Some nodes were running Ubuntu 20.04 (focal), while others were running Ubuntu 22.04 (jammy), and different APT repositories were pulling different Docker Engine versions. As a result, after another scheduled Docker update on the nodes, the production environment ended up with a mix of nodes running 28.1.1 and 28.2.2 at the same time. The docker node ls screenshot additionally confirmed that mixed versions were present not only on worker nodes but also on manager nodes, including swarm4 and swarm5. How the Incident Manifested The incident was detected not through user complaints and not through a general service outage, but through Traefik monitoring. The triggered alerts showed partial unavailability of one of the router-app containers, after which the health-check dashboard confirmed that the issue affected not the entire service but one of the manager nodes. This is important because the red blocks on the dashboard did not indicate a complete outage of router-app. It meant that Traefik on one of the manager nodes started receiving health=0 when checking that service’s backend endpoint, while the other manager nodes continued to see the backend as healthy. In practice, it looked like this: traffic through one of the manager nodes stopped reaching the router-app containers correctly, but Traefik, running on all five manager nodes, automatically excluded requests to the unhealthy entry point. As a result, from the outside the incident appeared as partial degradation rather than full unavailability. That is exactly what made the situation tricky. Fault tolerance limited the impact of the incident, but the underlying cause remained inside the cluster and continued to affect one of the entry points. What Docker Showed After localizing the issue to one of the manager nodes, it became clear that the cause should be sought not in router-app itself but in the network path between Traefik and the backend containers. At the same time, docker service ps did not show a widespread service failure, and the containers still appeared as running. The next useful signal came from the dockerd logs on swarm5. Repeated messages appeared there, including Peer delete operation failed, neighbor entry not found, and errors related to deleting FDB and neighbor entries for the VXLAN interface vx-001001-5lk08. For example: Plain Text Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.366807202Z" level=warning msg="Peer delete operation failed" error="could not delete fdb entry for nid:5lk08r7jjvtq5idqggzeygmlv eid:4e7d63d00fffaa6be7ce6362f47acd7912f7c11e5ac6e018393722decc16c210 into the sandbox:neighbor entry not found for IP 10.170.0.37, mac 02:42:0a:1b:14:2c, link vx-001001-5lk08" Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.765092537Z" level=warning msg="error deleting neighbor entry" error="no such file or directory" ifc=vx-001001-5lk08 ip=10.170.0.138 mac="02:42:0a:1b:14:65" Such messages were highly consistent with problems in Docker Swarm’s overlay network. In essence, Docker was trying to delete network records that were no longer present in the tables, which usually points to desynchronization of network state at the VXLAN, FDB, or neighbor-table level. By themselves, these messages still did not provide a complete explanation, but they pushed the investigation in the right direction. It became clear that the problem was not in the application’s business logic but in the network layer on one of the nodes. Additionally, docker node inspect self --pretty on swarm5 showed that from Swarm’s point of view the node looked normal: State: Ready, Availability: Active, Raft Status: Reachable, Leader: No, while Engine Version was already 28.2.2. This was an important point: the control plane still considered the node healthy, even though at the traffic-flow and network-state level it was already behaving differently. Diagnostics The investigation was carried out at the node level. The tools used included docker node ls, docker version, apt-cache policy docker-ce, as well as ip link show, bridge fdb show, ip neigh show, and comparisons of iptables chains across different nodes. The key fact became visible after docker node ls. The cluster was not homogeneous in terms of Docker Engine version: some manager nodes and some worker nodes were already running 28.2.2, while the others remained on 28.1.1. This led to the assumption that the issue might be at the iptables rules level. After that, iptables had to be compared separately on healthy and problematic nodes. On swarm5, a full rules dump was collected using the combination of iptables -S, iptables -t nat -S, and iptables -t mangle -S. Those rules showed the DOCKER, DOCKER-FORWARD, DOCKER-INGRESS, and DOCKER-USER chains, as well as ACCEPT, DROP, and DNAT rules for traffic through docker_gwbridge, published ports, and ingress routing.... To test the hypothesis, not only the problematic swarm5 but also the first manager node, swarm1, was compared, where a stable stack with Docker 28.1.1 had long been running. On swarm1, the output of iptables -S and iptables -t nat -S showed the expected picture: the DOCKER and DOCKER-INGRESS chains contained a full set of ACCEPT and DNAT rules for all published ports (80, 8080–8082, and dozens of internal service ports) with symmetric dport/sport pairs, while DOCKER-USER effectively boiled down to a clean RETURN. Taken together with the dump from swarm5, this reinforced the conclusion that on nodes running 28.1.1, the iptables configuration for ingress and routing was consistent, and the differences seen on 28.2.2 were related not to manual changes but to the behavior of Docker Engine itself. After that, iptables had to be compared separately on other nodes running different Docker versions. On nodes with 28.1.1, the DOCKER and DOCKER-USER chains and the associated rules were in the expected state, whereas on nodes with 28.2.2 some of the required rules were missing or the chains were reduced to a minimal RETURN. This explained the observed behavior well. The services remained running, Swarm did not appear broken, but external traffic and part of the overlay routing through a specific manager node were working incorrectly, causing Traefik on that node to report health=0 for router-app. It is worth noting separately that journalctl -u docker and docker service ps did not provide a simple direct cause for the incident. They did not show a picture of a general failure, so the conclusion had to be assembled from several sources: Traefik monitoring, dockerd logs, Docker versions, and the state of iptables on different nodes. Fix Once the main hypothesis had narrowed down to mismatched Docker Engine versions, the solution was fairly straightforward: return the cluster to a homogeneous configuration by rolling back to version 28.1.1 as the fastest solution. The rollback was performed for swarm4, swarm5, and all worker nodes where version 28.2.2 had already been installed. To do this, a specific package version was pinned via apt, then Docker was restarted, and the installed version was verified. One version of the commands looked like this: Shell apt-cache madison docker-ce | grep 28.1.1 apt-get install docker-ce=5:28.1.1-1~ubuntu.22.04~jammy \ docker-ce-cli=5:28.1.1-1~ubuntu.22.04~jammy \ containerd.io && systemctl restart docker && docker --version Additionally, it made sense to check the package sources and, if necessary, remove conflicting APT entries so that the nodes would no longer receive an unsuitable Docker version from another repository. In practice, it looked like this: Shell sudo rm /etc/apt/sources.list.d/download_docker_com_linux_ubuntu.list sudo apt update After the rollback, docker version was checked again, as well as the DOCKER and DOCKER-USER chains. After Docker Engine had been unified to 28.1.1 on both manager and worker nodes, the issue disappeared. From the perspective of external behavior, this was confirmed immediately. Health checks in Traefik returned to the green zone, and the partial unavailability of router-app on one of the manager nodes could no longer be reproduced. Root Cause Based on the available data, the most well-founded working version is this: in this environment, Docker Engine 28.2.2 formed or applied iptables rules related to DOCKER, DOCKER-USER, FORWARD, ingress, and overlay networking differently. In a mixed cluster, this led to one of the manager nodes no longer forwarding traffic correctly to the router-app backend containers, even though from the perspective of the control plane and service state this did not look like a direct failure. It is important here not to overstate what the data allows. This case does not prove a universal upstream bug in Docker 28.2.2 for all Swarm installations, but it does show that even closely related Docker Engine versions can affect the cluster’s network plane differently, especially when different Ubuntu distributions and different package sources are present in production at the same time. What Follows From This The first conclusion is simple: Docker Swarm is sensitive to Docker Engine version mismatches. If some manager or worker nodes have been updated while others have not, this can lead not only to version drift as an organizational problem, but also to practical issues with traffic, published ports, and overlay routing. The second conclusion is that after updating Docker, it is necessary to check not only docker version but also the node’s network behavior. The minimum set includes iptables -L DOCKER -v -n, iptables -L DOCKER-USER -v -n, checking published ports, ingress/overlay state, and health checks from the edge proxy. The third conclusion is that it is useful to maintain a single baseline stack across all nodes. One Ubuntu LTS distribution, unified repositories, and the same update order reduce the chance that cluster state will remain formally healthy while part of the network traffic is already being handled incorrectly. The fourth conclusion concerns update order. In our case, that was exactly how it happened, but it is worth noting separately. When Docker is updated in a Swarm cluster, it is better to update worker nodes first, then manager nodes, and the leader last, while after each stage separately checking the node’s behavior under real traffic conditions (service availability, correct routing, and published ports). In our case, enhanced monitoring was in place, so no additional manual checks of node behavior in traffic were required: if any part of the infrastructure became unavailable, we would promptly receive an alert. The final conclusion relates to monitoring. In this case, Traefik not only helped limit the impact of the incident by routing around the unhealthy node, but also provided the first precise signal that the problem was localized to a specific entry point rather than existing at the level of the entire service or the entire cluster.

Compliance Checkbox vs. Architectural Constraint Most data platforms treat audit-readiness as a downstream concern. The pipelines are built, the warehouse is populated, the dashboards ship, and only then does someone ask how the platform would respond to a regulator's request to reconstruct account balances as of a date eighteen months ago, or to prove that a reported figure hasn't been altered since submission. At that point, the answer is usually assembled after the fact: cross-referencing backups, reconstructing state from scattered logs, or worse, discovering that the required history was never captured at all. This reactive posture is what "compliance checkbox" architecture looks like in practice. The alternative audit-ready by design treats three properties as non-negotiable architectural constraints from the outset, not features added later: lineage, point-in-time reconstruction, and immutability. The distinction matters because a constraint enforced at the architecture level cannot be silently bypassed under deadline pressure the way a bolted-on compliance script can. Three Architectural Constraints, Defined Lineage Every data point must be traceable to its origin, and every transformation it passed through must be recorded as a first-class artifact of the pipeline, not reconstructed later from job logs or tribal knowledge. Lineage that lives only in a wiki page or a data dictionary is documentation, not architecture; lineage that lives in pipeline metadata, enforced by the platform itself, is a constraint. Point-in-Time Reconstruction A regulator's question is rarely "what does the data look like today"; it's "what did the data look like as of a specific past date, and can you prove it?" A platform designed for point-in-time reconstruction can reproduce the exact reported state as of any historical timestamp, not just restore from the nearest backup window. Immutability Once a record has been reported or submitted, it should be architecturally incapable of silent modification. This doesn't mean data can never be corrected; it means corrections are new, versioned, timestamped events layered on top of history, never in-place overwrites of it. The Audit-Readiness by Design (ARD) Maturity Model To evaluate whether a given data platform or pipeline is genuinely audit-ready by design, it helps to score it across the same three dimensions on a four-level maturity scale from bolted-on compliance to constraint-native architecture. This is deliberately structured the same way infrastructure maturity models work: each level represents a materially different failure mode under regulatory scrutiny, not just a stylistic difference. Dimension Level 1: Bolted-On Level 2: Retrofitted Level 3: Designed-In Level 4: Constraint-Native Lineage Manual documentation only; no code-level trace Logging added after pipelines built; partial coverage Lineage captured by pipeline metadata at build time Lineage is a required schema element; pipelines fail to deploy without it Point-in-Time Reconstruction No historical state; only current snapshot exists Periodic backups allow coarse-grained rollback Versioned tables enable reconstruction to any recorded checkpoint Any timestamp is reconstructable to the transaction level, by design Immutability Tables freely overwritten (UPDATE/DELETE in place) Soft-delete flags added; underlying rows still mutable Append-only storage for regulated tables Immutability enforced at the storage layer; mutation is architecturally impossible A platform's ARD maturity is not a single score but a profile across the three rows; it's common to see a platform at Level 3 on Immutability while still at Level 1 on Lineage, and that gap is usually exactly where audit findings originate. The model is most useful as a gap-identification tool during architecture review, applied per regulated data domain rather than to an entire platform at once, since different domains (e.g., transactional reporting vs. internal analytics) typically warrant different target levels. Design Patterns That Support Each Constraint Event sourcing: Storing state as an append-only sequence of events rather than mutable current-state tables gives lineage and immutability simultaneously; the event log is both the audit trail and the source of truth.Table versioning/time-travel storage: Storage formats that retain prior versions of a table as of any commit or timestamp directly support point-in-time reconstruction without requiring separate backup infrastructure.Append-only ledgers for regulated tables: Rather than updating a row, a correction is written as a new row referencing the one it supersedes; the history is never destroyed, only extended.Metadata-driven pipeline orchestration: Lineage capture built into the orchestration layer itself (rather than added as a separate logging step) ensures lineage cannot be skipped, since the pipeline cannot run without emitting it. Common Pitfalls Lineage tracked only in documentation: A data dictionary or architecture diagram is not evidence a regulator can independently verify against the running system.Silent backfills: Correcting historical data by overwriting it in place destroys the very history the platform may later be asked to prove.Soft-delete mistaken for immutability: A boolean "deleted" flag on an otherwise mutable row provides none of the guarantees of true append-only storage.Backup cadence mistaken for point-in-time capability: Nightly backups allow rollback to the nearest backup window, not reconstruction of the exact state as of an arbitrary past timestamp. A Composite Example Consider a generalized (composite, non-attributable) regulatory reporting platform for a financial services organization. An architecture review using the ARD model found the platform at Level 3 on Immutability (append-only storage for core ledger tables) but Level 1 on Lineage transformation logic lived in scheduler scripts with no captured metadata trail. When a regulator later requested a full transformation history for a reported figure, reconstructing it took several weeks of manual log archaeology rather than a direct query. Applying the ARD model earlier in the platform's design would have surfaced this specific gap: a strong Immutability posture masking a materially weaker Lineage posture well before it became a live audit finding. Conclusion Audit-readiness that is designed in behaves fundamentally differently under regulatory pressure than audit-readiness that is bolted on: one is a property of the architecture that cannot be quietly skipped, the other is a checklist item that depends on someone remembering to run it. Treating lineage, point-in-time reconstruction, and immutability as architectural constraints and using a structured model like ARD to find the gaps between them turns audit-readiness from a recurring fire drill into a property the platform simply has.

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs

The evolution from monolithic applications to microservices transformed enterprise software by decomposing business capabilities into independently deployable services. REST APIs, asynchronous messaging, and service discovery enabled systems that scaled both organizationally and technically. Although this model remains effective for deterministic business logic, the emergence of AI agents introduces a different execution paradigm. Instead of invoking predefined endpoints, an agent receives an objective, reasons about available capabilities, selects appropriate services, and dynamically composes a workflow. This shift changes service boundaries from business functionality to decision-making and capability orchestration. Why This Matters Traditional microservices assume that applications already know which services to invoke. An Order Service calls Inventory, Payment, and Shipping because the workflow is explicitly encoded during development. An AI agent, however, begins with an intent rather than an execution path. A request such as "purchase the least expensive laptop available and deliver it tomorrow" requires evaluating inventory, pricing, promotions, shipping constraints, and fraud policies before any API is called. The workflow is determined during execution instead of implementation. A conventional orchestration service typically resembles the following implementation. Java public OrderResponse checkout(OrderRequest request) { Inventory inventory = inventoryClient.reserve(request); Payment payment = paymentClient.authorize(request); Shipping shipment = shippingClient.schedule(request); return new OrderResponse(payment, shipment); } The implementation is deterministic because every dependency is known beforehand. Adding another payment gateway or shipping provider requires modifying orchestration logic, gradually increasing coupling between services. As enterprises integrate AI-driven workflows, continuously extending predefined execution paths becomes increasingly difficult. Agent Services replace hardcoded dependencies with capability discovery. Rather than directly invoking an Inventory Service, the runtime identifies which registered capability satisfies the current intent. Java public Tool resolve(Intent intent) { return toolRegistry.stream() .filter(tool -> tool.supports(intent)) .findFirst() .orElseThrow(() -> new ToolNotFoundException(intent.name())); } The registry enables services to advertise capabilities instead of exposing only procedural APIs. Existing microservices remain responsible for inventory reservation, payment authorization, or shipment scheduling, but the responsibility for deciding which capability should execute moves into an intelligent coordination layer. New business capabilities can therefore be introduced without rewriting orchestration code. This distinction fundamentally changes API design. Traditional REST endpoints expose operations such as /reserveInventory or /authorizePayment. Agent-oriented systems instead expose semantic capabilities like "find lowest cost supplier," "recommend shipping option," or "detect payment risk." These descriptions allow planning engines to reason about business objectives instead of matching endpoint names. Reasoning requires an additional architectural component capable of translating natural language into executable plans. This responsibility belongs to an Intent Router, which functions similarly to an API Gateway but routes requests based on semantic meaning rather than URLs. Java public ExecutionPlan plan(String goal) { Intent intent = classifier.classify(goal); Tool tool = registry.resolve(intent); return planner.create(tool, goal); } The classifier converts an objective into structured intent, the registry discovers an appropriate capability, and the planner generates an execution strategy. Once planning completes, downstream execution remains deterministic. Large language models participate only during reasoning, while conventional microservices continue enforcing validation rules, transactional consistency, and domain constraints. Separating planning from execution preserves enterprise reliability while introducing adaptive behavior. This separation also dispels a common misconception that AI agents replace microservices. Business logic continues to belong inside deterministic services because payment authorization, inventory consistency, pricing calculations, and compliance rules require predictable execution. Agent Services instead provide an intelligent layer responsible for selecting, coordinating, and sequencing those services according to business objectives. Rather than replacing existing architectures, they extend them with decision-making capabilities that previously existed only inside application code. Consequently, service boundaries begin shifting away from business entities toward reusable decision engines. Instead of embedding procurement, logistics, or fraud decisions inside multiple applications, organizations can expose these responsibilities as independent Agent Services that orchestrate existing microservices. The underlying APIs remain stable while reasoning evolves independently, enabling enterprise systems to become progressively more adaptive without sacrificing the deterministic foundations that made microservice architectures successful. Taking Memory Into Account Memory becomes the next architectural concern once planning is separated from execution. Stateless REST requests work well for isolated transactions, but agents frequently solve objectives through multiple reasoning cycles. Intermediate decisions, retrieved knowledge, user preferences, and execution history must persist beyond a single request. This context is operational rather than transactional. Business entities continue residing in relational databases, while the agent memory layer preserves reasoning state that enables future decisions to remain consistent. Java public AgentContext update(String sessionId, Observation observation) { AgentContext context = repository.load(sessionId); context.append(observation); repository.save(context); return context; } Rather than storing business records, the memory layer continuously enriches execution context with observations generated during planning. Future reasoning cycles consume this accumulated context instead of repeatedly querying downstream services, reducing redundant tool execution while maintaining continuity across long-running workflows. As objectives become more sophisticated, a single agent rarely owns every required capability. Instead of directly invoking multiple APIs, an agent can delegate specialized responsibilities to another agent while maintaining overall coordination. This interaction is based on expertise rather than ownership, allowing procurement, logistics, compliance, or fraud agents to evolve independently while sharing the same underlying microservices. Java AgentResponse response = logisticsAgent.execute( new AgentTask( "Optimize shipping route", context)); Delegation transfers structured objectives instead of procedural API calls. Each agent independently plans its assigned task before returning a deterministic result. Existing Inventory, Payment, and Shipping services remain unchanged, while the coordination layer becomes modular and extensible. Observability Implications Observability must also evolve because traditional distributed tracing explains service execution but not decision making. Understanding why an agent selected one capability over another is equally important as measuring latency or availability. Reasoning traces therefore become first-class telemetry alongside conventional application metrics. Java Span span = tracer.nextSpan() .name("agent.plan"); span.tag("goal", goal); span.tag("selectedTool", tool.name()); span.tag("confidence", score.toString()); span.end(); Capturing planning metadata allows engineering teams to correlate business outcomes with reasoning quality. An operation may succeed technically while producing an incorrect recommendation because the planner selected an unsuitable capability. Monitoring therefore expands beyond response times to include tool selection, planning confidence, execution cost, and reasoning latency. Autonomous planning also introduces governance challenges. Traditional services authorize callers before executing business logic, whereas Agent Services must additionally validate that planners invoke only approved capabilities. Every tool should expose explicit permissions and execution policies so that reasoning engines remain constrained by enterprise governance regardless of how plans are generated. Java public ToolResult execute(AgentTask task) { policyEngine.authorize(task.agent(), task.tool()); return toolExecutor.run(task); } Separating authorization from planning ensures deterministic policy enforcement around probabilistic reasoning. Existing identity providers, audit systems, and compliance frameworks remain applicable because execution ultimately flows through governed business capabilities rather than unrestricted model outputs. A Final Word The transition from microservices to Agent Services is therefore not a replacement of proven architectural principles but their natural evolution. Microservices continue delivering transactional consistency, persistence, and deterministic business logic, while Agent Services introduce planning, semantic routing, capability discovery, memory, and adaptive orchestration. The architectural boundary shifts from exposing operations to exposing decisions, allowing intelligent planners to compose existing services according to business objectives rather than predefined workflows. Enterprise platforms adopting this layered approach preserve the reliability of mature microservice ecosystems while gaining the flexibility required for AI-native applications, making Agent Services the next logical abstraction for software systems where reasoning becomes as important as execution.

The Micro-Enterprise Bottleneck: When Core Delivery Collides With Operations The Business Case: The Friction of the "Comfort Gap" I have three primary alter egos. Early in the mornings, I teach Spanish. Nothing fancy, just a simple, online session, focused on one student at a time, sharing and imparting what I learned and how I learned, to help them benefit from knowing Spanish as their second language. The rest of the day is spent in my Enterprise Architecture work — from consulting, to product development, to strategic solutions, and you know… all the standard corporate jargon. And then late at night, I imagine mysteries and write fiction. All that is fine. But then one of the most awkward conversations I have to have occasionally is telling my student: “Hey, so… you’ve used 10 classes and only paid for 10 classes… physics dictates we cannot proceed without a renewal.” Awkward, right? One morning where I needed to have that exact conversation, I thought to myself, “Ha! Let me hire an operations manager to handle these. I just need to see the details on the Kanban board later.” But then, I hit the budget committee. Ahem. Which was just me, looking at my own bank account. The committee quickly decided that hiring a manager for an ultra-small-scale business means I’d be working entirely to pay them, leaving me with Rs. 0 and a lot of regret. The Solution Philosophy: Pragmatic Lifestyle Engineering So, in real-world businesses, this is where they bring in an Enterprise Architect. I thought, “hey, that’s me!” I looked at the problem through an engineering lens and realized that manual administrative work is the technical debt of real life. If a system requires me to manually check a spreadsheet and manually make a reminder, then the system is broken! After all, why spend 10 minutes a week doing something manually, when you can spend an hour over the weekend, over-engineering a serverless cloud pipeline to do it for you — for free? But how do you build an automated system that handles the “money talk” with the cold, polite neutrality of a machine, that ensures absolute accuracy so you don’t falsely accuse a student of not paying, and… runs with a grand total operating cost of exactly zero rupees? Fig. 1. The reality of operational scale Deconstructing the Solution: Three Core Architectural Pillars First thing to consider in a multi-million-dollar platform is the core of the business problem. What pillars are going to hold up this house? It is the exact same way a structural architect might think before drawing a blueprint. Decoupled State Management (The "Database" Illusion) Let’s take the data storage layer first, because, well, there is data and it needs to be stored. In an enterprise, what would this be? Potentially an RDS instance or a distributed NoSQL cluster. In the current use case, I found the perfect low-latency “read/write replica” for a non-technical admin interface. It is easily accessible on my phone, simple to update manually if and when required, and most importantly, it has zero hosting costs. What is it? It is an engineering sin that makes an architect shudder. It is Google Sheets. But don’t dismiss it as a glorified spreadsheet. Look at it pragmatically as a lightweight, highly available distributed state machine. Strict Temporal Bounding (The Data Inflation Filter) We solved the data and storage layer. Now let us look at a potential problem that could come up at this stage. Let us look at tracking this attendance event over time, correlating it to the problem statement at hand. Imagine if the code blindly counts every class a student has ever attended since day one; the data volume will burgeon, and the execution will come to a grinding halt. For all you know, the historic data can even corrupt my current cycle numbers. To mitigate this, we introduce the pattern of setting a strict dynamic time window. The API needs to get hard boundaries based on the last transaction date. Now what if you have a recurring calendar invite? The second boundary that gets passed to the API then is the attendance data only up to the current millisecond. If this is not in place, then we are basically looking at a catastrophic data bug. If we don’t define the time array, a student who took a break three months ago might suddenly get an automated email screaming that they owe money for classes they took in some past life. We need accuracy, not a tracking crisis, remember? Idempotency and State Gates (The "No Spam" Rule) No one likes spam. That brings us to a crucial enterprise pattern — idempotency. What does it mean? Well, simply that no matter how many times a given operation is executed, the side effect is only applied once. I wish this were the case for medications that have side effects, but that is out of an enterprise architect’s scope. What did I do here for this idempotency? A simple gate column in the spreadsheet with a binary value for ReminderSent. The engine strictly evaluates this Boolean flag before firing an email. Once the threshold is hit, and the email is sent, the pipeline instantly flips the state to True. Think of this as the safety valve. GitHub Actions runs this automatically every evening. Without this state gate, once a student’s package expires, my headless cron engine will politely, coldly, and relentlessly spam their inbox every single evening at 7 pm until they pay me or block my email address. Fig. 2. The Architecture Building the Serverless, Zero-Cost Stack Ok, enough of the talking; let us orchestrate the cloud ecosystem. Now, we are allowed to use only the free-tier resources. Ladies and gentlemen, put your hands together for the trio — Google Workspace APIs for data and logic, GitHub Actions, our ephemeral runtime environment, and Node Mailer over SMTP. Accelerating Development via a Local AI Agent Stack One of the biggest challenges we face as adults is context switching. I’d skip elaborating on that for all of our sanity. I built this stack without the additional burden of context switching by spinning up a local AI environment on my humble 8GB CPU on a basic home laptop running a Windows operating system. Just good old Ollama, the Continue extension in VS Code, and Gemma. The benefit of a local agent is that it allows an architect to quickly generate boilerplate code, test logic boundaries, and iterate without needing premium cloud tokens. Engineering the Pipeline: Key Code Implementations I chose TypeScript for the engine’s core implementation to leverage its strict typing system. When you are mapping dynamic spreadsheet cells to operational parameters, strict types are your first line of defense against runtime metadata errors, especially when handling complex student data structures. Enforcing Temporal Boundaries in API Queries If we look at the piece of code below, we see the strict temporal bounding pillar, which we spoke about earlier, in action. The date boundaries are dynamically calculated on the fly, with the student’s last payment date defining the lower bound and the exact current moment becoming the upper bound. This configuration payload is now handed over to the Google Calendar API query to extract only the relevant window of attendance events. TypeScript const res: any = await calendar.events.list({ calendarId: process.env.GOOGLE_CALENDAR_ID, singleEvents: true, orderBy: "startTime", maxResults: 2500, pageToken, timeMin: lastPaymentDateISO, // Strictly drops anything before this timestamp timeMax: nowISO, // Strictly drops anything in the future }); By offloading this filter to the API gateway, we are protecting our serverless memory footprint and preventing legacy historical data from leaking into the current cycle calculations. Mitigating Notification Spam via Idempotency Check Gates After isolating the precise attendance window, the engine now evaluates the current state of the record. The logic gate is straightforward, but absolute at the same time. Gate A – The Quota Breach – Does the total number of attended classes meet or exceed the pair threshold?Gate B – The Idempotency Check – has a reminder already been dispatched for this specific cycle? If and only if both gates evaluate to true, the communication layer fires up. The cold, polite notification goes out. And immediately, the engine executes a state synchronization back to the persistence layer. TypeScript const meetsQuotaLimit = currentLessonsCount >= s.classesPaidFor; const isReminderNotSentYet = !s.reminderSent; console.log(`↳ Quota Met (Count >= ${s.classesPaidFor}): ${meetsQuotaLimit} | Is Reminder Pending: ${isReminderNotSentYet}`); // Update Column H with the exact calculated count first await updateLessonsUsed(s.rowNumber, currentLessonsCount); if (meetsQuotaLimit && isReminderNotSentYet) { if (s.email) { // Step 4: Dispatch email notification message await sendEmail(s.email, s.student, currentLessonsCount, s.classesPaidFor); console.log(`↳ Outbound alert dispatched cleanly to ${s.email}`); // Step 5: Persist ReminderSent back to TRUE await updateReminderSentStatus(s.rowNumber, "TRUE"); console.log(`↳ Spreadsheet statuses permanently updated to TRUE.`); } else { console.log(`⚠️ Email notice skipped: Student is missing an email address.`); } } else { console.log(`↳ Conditions not met. Sheet column counters updated, no emails dispatched.`); } } console.log("\nProcess finalized successfully!"); Infrastructure as a Service: The GitHub Actions Cron Engine Great code is completely useless without an operational home. Since our core constraint when we started was an operational budget of exactly INR 0, spinning up a dedicated AWS EC2 instance or an Azure VM was entirely out of the question. That is where a knight in shining armor came to my rescue — GitHub Actions. This is not just any CI/CD tool; it serves as a highly capable serverless, headless execution environment. Securing the Infrastructure Without an Enterprise Vault You turn around and see the elephant in the room. Security. Let us address that then. To make this pipeline functional, the runner needs access to highly sensitive credentials. Let’s see — my Google Service Account private JSON keys, my personal SMTP email app passwords. In an enterprise, this would either be solved by pulling secrets dynamically from HashiCorp Vault or AWS Secrets Manager. I achieved the exact same security boundary by injection-mapping these sensitive parameters directly into my execution runtime environment via GitHub Repository Secrets. This adheres strictly to one of the fundamental golden rules of software architecture: No secrets ever touch source control. So sorry, you won’t find a single credential sitting in my source repository. Navigating Cloud Scheduler Nuances (The Asymmetric Minute Strategy) I had this all set up and was waiting for the line to appear on the workflows tab of GitHub Actions that my job had run at exactly 7 pm that evening. But hey, what is engineering without a few infrastructure curveballs? GitHub Actions handles both scheduled and manual workflows based on what is set up in your configuration. Now, this is a shared, multi-tenant free queue, and millions of developers configure their cron jobs to run at flat intervals like :00 or :30. This causes massive platform resource contention. The background event bus gets heavily backed up, leading to severe delays or entirely skipped jobs. While I still haven’t learned how to bypass real-world traffic jams in Bangalore, fixing this cloud traffic jam was far easier in comparison. I deliberately shifted my cron pattern completely away from peak times to an asymmetric, off-peak minute (:33 or :37). This is one way to optimize reliability in shared cloud infrastructure. But mind you, it still won’t fire down to the exact second mentioned in your YAML file; shared platform queues will always have a slight propagation lag. YAML on: schedule: # Runs every day at 13:33 UTC, which corresponds to 7:03 PM IST (Indian Standard Time) - cron: '33 13 * * *' workflow_dispatch: # Allows you to also trigger it manually from the GitHub UI whenever you want Fig. 3. Schedule on GitHub Actions Conclusion: Reclaiming Creative Bandwidth Through System Design Let us look at the ROI here, because isn’t that what the executives are most concerned about? I invested a weekend afternoon, working alongside a local AI agent stack, and built a production-grade automation engine. It completely eliminated a major source of personal and operational friction for me. It operates with absolute mathematical precision, and for me, the important part is that it has an ongoing operational maintenance cost of exactly INR 0. Enterprise Architecture is not just a corporate discipline reserved for massive scaling clusters at big tech corporates. It is a systematic mindset — yes, mindset. By applying these exact design constraints — decoupling, temporal boundaries, and idempotencies - to our small personal workflows, we are protecting our most valuable non-renewable resource — our human creative bandwidth. Let me ask you: how do you handle administrative friction or manual processes in your own side integrations or small-scale workflows? Would you prefer to see this system migrated to an edge-compute model like Cloudflare Workers, or evolved to hook directly into Meta’s WhatsApp Cloud API for notifications? Let me know in the comments below.

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen. Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel. In this blueprint, we will build a secure multi-tenant execution environment. We will containerize the agent runtime using Docker, provision a GKE cluster with Axion nodes, isolate the network, and orchestrate the execution layer using a robust Java backend. Step 1: Containerizing the Agent Runtime The first step is establishing a baseline execution environment. We want this Docker image to be as lightweight as possible to reduce the attack surface, while containing the necessary runtimes for the LLM to execute its logic. Dockerfile # Use a minimal Alpine base image to reduce attack surface FROM python:3.11-alpine # Create a non-root user for execution RUN addgroup -S agentgroup && adduser -S agentuser -G agentgroup WORKDIR /sandbox # Copy the execution wrapper script COPY --chown=agentuser:agentgroup execute_payload.py /sandbox/ # Enforce non-root execution USER agentuser # Prevent Python from writing pyc files and buffering stdout ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 CMD ["python", "execute_payload.py"] To make this functional, we need an entrypoint script that safely reads the LLM-generated code from an injected environment variable or a mounted volume, executes it, and captures the output. Here is a simplified execute_payload.py implementation: Python import os import sys import traceback def main(): # In a production environment, this payload might be injected via # a Kubernetes Secret or a secure sidecar proxy. encoded_payload = os.environ.get("AGENT_PAYLOAD", "") if not encoded_payload: print("Error: No payload provided.") sys.exit(1) try: # Execute the untrusted code within this isolated process # Security constraints are handled by the container and gVisor layers exec(encoded_payload, {"__builtins__": __builtins__}, {}) except Exception as e: print(f"Execution Error: {str(e)}") traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() Even if a malicious script breaks out of the Python runtime, it will find itself as an unprivileged user inside a minimal Alpine container. Step 2: Provisioning GKE With Axion and Agent Sandbox Google Axion (N4A) processors provide excellent performance per watt, making them ideal for running hundreds of concurrent, lightweight agent tasks. We will create a cluster and explicitly enable the sandbox feature. Shell # Create the GKE cluster with Sandbox enabled gcloud container clusters create agent-sandbox-cluster \ --region us-east4 \ --enable-sandbox \ --sandbox type=gvisor \ --release-channel regular # Create a dedicated node pool using Axion N4A instances gcloud container node-pools create axion-agent-pool \ --cluster agent-sandbox-cluster \ --region us-east4 \ --machine-type n4a-standard-4 \ --num-nodes 3 \ --node-labels dedicated=untrusted-agents \ --tags untrusted-workload Applying node labels ensures that trusted core microservices do not accidentally end up on the same physical infrastructure as untrusted agent execution environments. Step 3: Enforcing Network Isolation Compute isolation is useless if the untrusted code can scan your internal network or exfiltrate data to the public internet. We must deploy a strict NetworkPolicy to default-deny all egress traffic from our sandboxed namespace. YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-agent-egress namespace: isolated-agents spec: podSelector: matchLabels: app: agent-executor policyTypes: - Egress egress: # Only allow DNS resolution - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP # Allow outbound only to a specific internal API gateway if needed # - to: # - ipBlock: # cidr: 10.0.0.50/32 Step 4: Deploying the Sandboxed Workload With the network secured, we define the Kubernetes deployment. By setting the runtimeClassName to gvisor, Kubernetes routes the container lifecycle through the GKE Agent Sandbox rather than the standard container runtime. YAML apiVersion: apps/v1 kind: Pod metadata: generateName: dynamic-agent-task- namespace: isolated-agents labels: app: agent-executor spec: # Instruct GKE to use the Agent Sandbox (gVisor) runtimeClassName: gvisor # Ensure these pods only land on our Axion node pool nodeSelector: dedicated: untrusted-agents restartPolicy: Never containers: - name: execution-environment image: your-registry/agent-runtime:v1.0.0 env: - name: AGENT_PAYLOAD valueFrom: secretKeyRef: name: task-payload-secret key: payload # Drop all unnecessary Linux capabilities securityContext: runAsUser: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" volumeMounts: - name: temp-storage mountPath: /tmp volumes: - name: temp-storage emptyDir: {} Step 5: Orchestrating the Execution via Java Spring Boot To bring this architecture together, the control plane must dynamically spin up these sandboxed pods whenever an AI agent decides it needs to run code. In a modern distributed system, this is typically handled by a core backend microservice. Using the Fabric8 Kubernetes Client in a Java Spring Boot application provides a highly resilient way to orchestrate these ephemeral workloads programmatically. Java import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.client.KubernetesClient; import org.springframework.stereotype.Service; @Service public class AgentOrchestratorService { private final KubernetesClient kubernetesClient; public AgentOrchestratorService(KubernetesClient kubernetesClient) { this.kubernetesClient = kubernetesClient; } public String executeUntrustedCode(String tenantId, String pythonCode) { // 1. Create a Kubernetes Secret containing the code payload String secretName = createPayloadSecret(tenantId, pythonCode); // 2. Load the sandbox Pod template and inject the specific payload secret Pod sandboxedPod = kubernetesClient.pods() .inNamespace("isolated-agents") .load(getClass().getResourceAsStream("/k8s/agent-pod-template.yaml")) .item(); // 3. Launch the pod dynamically via the API server Pod runningPod = kubernetesClient.pods() .inNamespace("isolated-agents") .create(sandboxedPod); // 4. Await completion and extract the logs safely kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .waitUntilCondition(pod -> pod.getStatus().getPhase().equals("Succeeded") || pod.getStatus().getPhase().equals("Failed"), 30, java.util.concurrent.TimeUnit.SECONDS); String executionLogs = kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .getLog(); // 5. Clean up the ephemeral resources kubernetesClient.pods().delete(runningPod); kubernetesClient.secrets().withName(secretName).delete(); return executionLogs; } } The Defense in Depth Strategy This architecture relies on a strict defense in depth model. If an LLM hallucinates a malicious payload or a user deliberately attempts prompt injection to compromise the platform, the attacker faces multiple independent barriers. The code executes as a non-root user in a minimal Alpine environment with a read-only filesystem. Network access is completely blocked by native Kubernetes policies. Finally, any attempt to exploit kernel vulnerabilities is intercepted by the gvisor runtime boundary running on dedicated Axion hardware. By combining these layers, engineering teams can build and scale trustworthy Agentic AI platforms without risking the integrity of their core cloud infrastructure.

As a data engineer, I’ve noticed business teams submitting intake forms, compliance documents, and project proposals that a tech team then manually validates against a set of predefined business rules stored in a database that gets updated quarterly. The time it takes to validate a single form is typically in the hours, and by the time you’ve validated the form, the submitter has moved on to other work. When I needed to validate project intake forms against 60+ business rules of financial, compliance, and other types of business rules and guidelines (some of them to be used in a deterministic way and others to be used in a more nuanced manner), I knew that a simple if-else logic-based manual review process would not scale. This article walks through how I developed an async, AI-powered validation API with AWS Bedrock Agents and Serverless Architecture to process and validate intake forms within 60 seconds without blocking the user. The architecture also manages cross-account authentication to get access to the AI-powered engine and shows failure recovery gracefully. Why Async? The Problem With Synchronous AI APIs Integrating AI into an API synchronously means users send a request, the server processes it, and returns results in one HTTP response, but many systems that use AI-powered validation take more than 30 seconds. The AI agent I built was taking anywhere from 30 seconds to 1 minute to evaluate all of the form fields for all the applicable rules and conditions. But the hard limit for the API Gateway is 29 seconds (HTTP timeout). One approach to make this API request work is to transform the synchronous request and response into an async request with a subsequent background processing step and poll the results from a separate endpoint. This can be implemented as follows: Client submits the form via POST, receives a request_id immediately (under 2 seconds)Validation runs asynchronously in the background (30–60 seconds)Client polls a GET endpoint with the request_id until results are ready By making the form submission step separate from the AI validation of that form in the background, users can continue working on other tasks instead of being stuck staring at a page waiting 30 to 60 seconds for the form to be validated. Architecture Overview As a data engineer, I was required to tackle three main challenges to create a production AI validation API: 1) the frontend application is deployed in a different AWS account, 2) AI agent-based form validation is extremely computationally expensive to run, and 3) business rules for this type of validation are likely to change from time to time without API code deployment. The architecture consists of five components: API Gateway (REST API): With Cognito Authorizer for cross-account JWT authenticationAsync Handler Lambda: It’s an entry point for the API. An Async Handler Lambda function is invoked by a POST request. It will store the form payload on S3, then trigger the Validation Lambda function and store an initial "processing" status in S3. The function immediately returns a request_id to the frontend client within 2 seconds.Validation Lambda: This function loads up all the rules for a given request from S3. It then builds up all the prompts for the Bedrock Agent and runs the Agent. The results of the Agent are then saved off in S3 for the Polling API.Polling Lambda: Handles GET requests and checks S3 for completed resultsRules Sync Lambda: Separate independent process to read validation rules from the data warehouse using EventBridge scheduler and sync to S3 for validation with AI model. Implementation: The Async Handler The async handler is the entry point. Its task is quite straightforward. It accepts the payload, stores it, triggers the Validation Lambda function, stores an initial "processing" status in S3, and returns the “processing” status with the request ID to the client. The function does all of this within a couple of seconds. Here is the core implementation: Python import json, boto3, uuid from datetime import datetime s3 = boto3. client(' s3') Lambda_client = boto3. client('Lambda') S3_BUCKET = 'my-validation-bucket' VALIDATION_LAMBDA = 'ai-validation-function' def lambda_handler(event, context): payload = json. loads (event. get ('body', "{}')) request_id = str(uuid.uuid4()) # Store initial processing status s3.put_object( Bucket=S3_BUCKET, Key=f'validation-output/(request_id)/status.json', Body=json.dumps({ 'request_id': request_id, 'status': 'processing', 'submitted_at': datetime. ttenew() .isoformat() }) ) # Fire-and-forget: invoke validation async pay Load ['_request_id'] = request_id lambda_client.invoke( FunctionName=VALIDATION_LAMBDA, InvocationType='Event', # Async invocation Payload=json. dumps (payload) ) return { 'statusCode': 202, 'body': json. dumps ({ 'request_id': request_id, 'status': 'processing' }) } In the above code snippet, I specifically invoke the validation lambda from the async handler by setting the InvocationType='Event'. This allows the async handler to return immediately to the frontend with the request_id for the submitted request. The Validation Lambda will then complete asynchronously and store the results in S3. Implementation: The Polling Handler The Polling Handler Lambda function manages the GET endpoint; it polls S3 for the updated status file and returns the current status of Validation Lambda processing: completed or failed. Here is the core implementation: Python def lambda_handler(event, context): request_id = event['pathParameters']['request_id'] try: status_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/status.json' ) status = json.loads(status_obj['Body'].read()) if status['status'] == 'processing': return {'statusCode': 200, 'body': json.dumps(status)} # Completed - return full results results_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/results.json' ) results = json.loads(results_obj['Body'].read()) return {'statusCode': 200, 'body': json.dumps(results)} except s3.exceptions.NoSuchKey: return {'statusCode': 404, 'body': 'Request not found'} S3 Decoupling: Using S3 as an intermediary between the validation Lambda and the polling handler allows for natural decoupling. The validation Lambda writes the results of the validation to S3, and the polling handler reads from S3 to return the latest status to the frontend. There is no shared state between the validation handler and the polling handler; there are no database connections, and there are no race conditions. Integrating the Bedrock Agent for Intelligent Validation An intelligent validation function would need more than just a set of rules to check for requirements and best practices. There are a lot of judgment calls that a human would make based on examples of how a policy or guideline would be applied in real life. To achieve that, the more effective way is to integrate with an existing AI function that is designed to handle a wide variety of scenarios and functions The Bedrock Agent architecture solved this by combining: Knowledge base: Containing policy documents, guidelines, and past examples of work for the intelligent validation to reference during the evaluation process.Dynamic prompts: The prompts for the AI model are built dynamically from the current validation rules. These are loaded from S3 as a JSON file and then injected with the current values for the specific field being evaluated.Structured output: Parse the assessment’s pass/fail status, confidence in the assessment, and a set of detailed recommendations made by the agent. The prompt for the AI agent is generated at runtime by the validation function. The rules are loaded from S3 earlier in the function's execution. Here is an example prompt: “Evaluate field [Project Justification] with value [user input] against rule: The justification must clearly describe the business problem being solved and include quantified impact. Reference the knowledge base for examples of approved justifications.” The AI returns a structured assessment of whether or not the field has passed validation, the confidence that the AI has in the assessment, and recommendations. Dynamic Rules Management: Keeping Rules in Sync Without Code Deploys Rules typically change on a monthly or quarterly basis by the business teams. To keep up with the current policy, the rules must be separate from the rest of the application code. To achieve that, I used Rules Sync Lambda, triggered daily by EventBridge: EventBridge fires at 6 AM daily.The Rules Sync Lambda queries the Data Warehouse (Redshift) for the current validation rules for the application.It also takes a copy of the most current version of the rules in S3 for purposes of rollback.It transforms and then uploads the new rules file to S3 as a new copy of the Validation_Rules.json file.Upon failure to update the rules in S3, a CloudWatch Alarm is triggered, which in turn triggers an SNS notification to the appropriate engineering team. The rules are managed as a database of rules (as opposed to being stored within the application code), which allows business analysts to easily update the rules on a quarterly basis without requiring any code changes or deployments. Cross-Account Authentication With Cognito In this case, the frontend application and the AI backend were set up in two different AWS accounts. When deployed within different accounts (as within an enterprise), cross-account authentication is required. Since the frontend application was already authenticated against a company’s SSO (Single Sign On) using Cognito, it was only a matter of how to reuse these tokens within another account without involving the Frontend team for changes. The solution was to create a Cognito Authorizer and attach it to a REST API created in the API Gateway. This API can then be set up to trust the User Pool from the frontend account. Below is a simplified representation of this configuration: API Gateway REST API with a Cognito Authorizer pointing to the frontend account’s Cognito User Pool ARN.CORS (Cross-Origin Resource Sharing) configuration for only that frontend domain.The frontend application is already authenticated with CognitoThe backend application accepts the tokens that the frontend application is using for authenticationThe frontend application simply sends the existing Cognito tokens that the frontend application already has created in the authentication process From the frontend team’s perspective, this was a simple implementation that required them to send the existing Cognito token with the request and to implement a polling loop for the GET endpoint. Results and Lessons Learned After deploying to production: Validation time: reduced from 2 -3 hours (manual) to less than a minute (automated)API response time for form submission: less than 2 seconds for GET API using an async pattern, meaning the frontend never has to wait for the backend60+ validation rules: per form, including both deterministic and AI-judgement rules Zero code deploys: for changes to the rules, which are stored in the database, sync daily Key lessons as a developer building this: Design for async from the start: Retrofitting a synchronous API to be async is very hard. If your AI inference takes more than 5 seconds, which is generally the case, then design your API to be async from day one.Use S3 as your state machine: S3 is the simplest, cheapest, and most reliable way to pass results between decoupled Lambdas. No databases, no queues, no DynamoDB for this pattern.Separate dynamic rules from code: Separate process for managing rules which are dynamic and change often to avoid deployment bottleneck Bedrock Agents are good for making judgment calls. If you have a deterministic check (is a field empty), then you can code that. But for a judgment call (does a justification make sense), then use an AI agent to make the call. Conclusion There is an entirely new way to approach the request lifecycle for APIs in this AI-powered validation API development. The asynchronous API with polling for validation is better than simply trying to work around the timeout limits of APIs. Bedrock Agents, along with S3 to manage the state of the workflow and EventBridge to synchronize rules on a daily basis from a database created by business users via a simple UI created by frontend team, while backend team does not need to write any code for new rules, all integrated together to form complex data validation system powered by AI-powered judgment calls while maintaining simple to deploy and scalable system. As a data engineer, there’s nothing quite like watching hours of manual work by a reviewer get compressed down into 60 seconds or less of automated work while maintaining the high level of evaluation that a business stakeholder expects.

Not long ago, I broke a backtest without changing a single line of code. I moved the script to a different machine—same OS, supposedly the same Python version — and the equity curve suddenly told a completely different story. Nothing in the logic had changed. The environment was the only obvious difference. That was the day I stopped treating the runtime environment as an afterthought and started treating reproducibility as part of the experiment itself. What I learned is this: a backtest isn’t truly reproducible just because it ran once on your laptop. A result you can’t regenerate reliably isn’t a research finding — it’s a coincidence. And when that coincidence eventually meets real money, it can get expensive fast. What follows is a minimal but complete pipeline for containerizing and automatically testing a Python backtesting system. No over-engineering — just Docker, GitHub Actions, and a few habits that make your results trustworthy. The Real Goal We aren’t building a live trading platform. We’re building a workflow that ensures every change is verified in a controlled environment: Plain Text Code change → automated tests → versioned Docker image build The system must do four things: Produce deterministic results, within an acceptable numerical tolerance, from the same versioned inputs in the same containerized environment.Automatically test every code change before it can be merged.Halt the pipeline when a test fails, with no exceptions.Tag every image build so we can trace it back to the exact source-code version. If a workflow can’t do that, it’s just a script living on someone’s laptop. Project Structure Before containerizing the application, let’s organize the repository clearly: Python backtesting-system/ ├── app/ │ ├── engine.py │ └── main.py ├── tests/ │ └── test_engine.py ├── data/ │ └── sample.csv ├── requirements.txt ├── requirements-dev.txt ├── Dockerfile └── .github/ └── workflows/ └── ci.yml The app/ directory holds the strategy logic, while tests/ remains separate. The data/ directory contains a small, frozen sample dataset that never changes — our gold standard. There are no absolute paths or machine-specific assumptions. Everything that might vary, including the data path, starting capital, and fee rate, comes from environment variables or a configuration file. Containerizing With Docker The classic “works on my machine” problem usually indicates an environment mismatch. Docker reduces this drift by packaging the application and its runtime dependencies into a versioned image. Here is the Dockerfile: Dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app ./app COPY data ./data CMD ["python", "-m", "app.main"] A few decisions matter here. We pin the Python 3.12 image series and can use an image digest when stricter reproducibility is required. Dependencies are pinned to exact versions in requirements.txt, since version ranges can silently introduce changes. We also use slim to keep the image small. Crucially, we never copy local keys, cached results, or temporary files into the image. The container doesn’t guarantee that the logic is correct. It helps ensure that sound logic runs in a controlled and substantially more consistent environment. Adding Tests That Matter Automation without tests simply automates mistakes. Our tests don’t attempt to prove that a strategy is profitable. They prove that the program behaves consistently. We check for: Clear errors when input files are empty or missing.Deterministic results when the same data and seed are used.Correct fee calculations.Rejection of malformed rows rather than silent processing.Required fields in every output. Here is one example, including the necessary imports: Python import pytest from app.engine import run_backtest def test_backtest_is_reproducible(): first = run_backtest("data/sample.csv", seed=42) second = run_backtest("data/sample.csv", seed=42) assert first["trades"] == second["trades"] assert first["final_equity"] == pytest.approx( second["final_equity"], rel=1e-9 ) This test establishes a simple contract: given the same starting conditions, the system will not drift beyond an acceptable margin. The direct comparison of trades works here because we assume that every trade entry uses standardized types such as integers and strings. If the trade records contain floating-point prices, those values should be checked individually with an appropriate tolerance. Building the GitHub Actions Pipeline Now we automate the workflow. It runs on pushes, pull requests, and manual triggers through workflow_dispatch: YAML name: Backtest CI on: push: pull_request: workflow_dispatch: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -r requirements-dev.txt - run: pytest - run: docker build -t backtest:${{ github.sha } . The steps are straightforward: check out the code, set up Python, install the development dependencies, run the tests, and build the Docker image. The requirements-dev.txt file includes both the production dependencies and a pinned version of pytest: Python -r requirements.txt pytest==8.3.4 If pytest detects a failure, the job stops immediately. The broken change never reaches the image-build step. The resulting image is tagged with the Git commit hash, creating a clear link between the source code and the image built during that workflow run. Managing Configuration and Secrets Environment-specific configuration and secrets should never be baked into the image. Environment variables can control data paths and run modes. If the system is later connected to a live data source, API credentials should be stored in GitHub Secrets or a cloud secrets manager—never in the source code or Dockerfile. Logs must not expose keys or sensitive headers. Even if today’s “production” environment is only a scheduled test run, development and production should use separate configuration sets. Treat every secret as sensitive, and keep environment-specific configuration outside the image. Lightweight Monitoring and a Path to Rollback Once the pipeline runs regularly, monitoring must go beyond asking whether the process is still alive. Useful questions include: Did the latest job complete, or did it hang?Has execution time increased dramatically?Is the input data intact?Were the output files generated, and are they non-empty?Which image version produced the latest results? If images are later pushed to a container registry, retaining the last few stable versions provides a straightforward rollback path. For scheduled backtest runs, we should also archive the data snapshot, parameters, and results. That allows us to return a month later and answer a very specific question: “What exactly did we test on July 24?” These practices aren’t unique to systems we build ourselves. Commercial grid-trading interfaces make automated execution accessible without revealing every part of their internal deployment pipelines. BYDFi is one example I encounter in my work. Because I work with the platform, this is a disclosed reference rather than an independent recommendation. The comparison is conceptual: understanding reproducibility, automated checks, and configuration management helps developers evaluate any automated tool more thoughtfully. The Experiment Isn’t Finished Until It’s Verified We started with a broken backtest and a frustrating realization. Now we have a different mindset. Docker reduces environment drift. Automated tests guard program behavior. GitHub Actions ensures every change passes through the same gate. Monitoring and versioning give us a clear path to detect problems and support rollback as the pipeline evolves. In a reliable backtesting system, reproducibility and verification are not tasks that come after the experiment. They are part of the experiment itself. The moment we treat them that way, our results stop being anecdotes and start becoming evidence. And when the decisions involved can carry real financial weight, evidence is the only thing worth building.