Microservices

Every Java developer who runs services on Kubernetes has watched this scene play out. Traffic spikes, the autoscaler adds a pod, and then everyone waits. The container is running in two seconds. The application is not ready for another twelve seconds. During those ten seconds, your existing pods absorb the extra load, latency climbs, and if things are bad enough, the autoscaler panics and adds even more pods that are also not ready. I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work Where Those Twelve Seconds Actually Go When a Spring Boot application starts, the JVM is not mostly running your code. It is loading classes. A plain REST service with Spring Web, Spring Data, and a driver or two loads somewhere between ten and twenty thousand classes before it serves its first request. For every single one of those classes, the JVM does the same ritual. Find the class file inside a jar, read the bytes, parse them, verify the bytecode is legal, and build the internal metadata structures it needs at runtime. Thousands of times. Every startup. In every pod. Here is the part that should bother you. Your container image never changes after you build it. The same jar, the same classes, the same parsing work, repeated identically in every pod that ever starts from that image. The JVM is solving the same puzzle again and again and throwing away the answer each time. CDS is the JVM saying: let me solve it once, write the answer to a file, and just memory map that file next time. What a CDS Archive Is A CDS archive is a file, usually ending in .jsa, that contains classes already parsed and verified, stored in the exact internal format the JVM uses in memory. On startup, the JVM maps this file straight into memory. No finding, no parsing, no verifying. The work was done ahead of time. You have been using CDS without knowing it. Modern JDKs ship with a default archive covering the core JDK classes, which is why java -version is fast. The step almost everyone skips is creating an archive for your application classes, all fifteen thousand of them. That is where the real win lives. The mechanism has one rule that matters for us. The archive must be created with the same JVM and the same classpath that will use it. That rule sounds annoying until you realize a Docker image is the one place in your entire infrastructure where JVM and classpath are frozen forever. Docker is not just compatible with CDS. It is the perfect home for it. The Training Run Creating the archive takes two steps. First you do a training run, where the JVM starts your application, watches which classes get loaded, and writes the list down. Then you exit, and the JVM turns that list into the archive. Since Java 13, this is pleasantly simple: Shell java -XX:ArchiveClassesAtExit=app.jsa -jar app.jar Run the app, let it come up, stop it, and app.jsa appears. From then on you start the app like this: Shell java -XX:SharedArchiveFile=app.jsa -jar app.jar There is an obvious question here. The training run wants to actually start the application, and inside docker build there is no database, no message broker, nothing to connect to. A Spring Boot app that cannot reach Postgres will crash during training. Spring Boot 3.3 solved this neatly. Setting one property makes the application run through its entire startup sequence, create all bean definitions, and then exit just before touching the outside world: Shell java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app.jsa -jar app.jar The application loads nearly everything it will ever load, writes the archive, and exits cleanly with no infrastructure needed. This is exactly what a Docker build stage can do. The Dockerfile Here is the complete picture: a multi-stage build where the image trains itself: Shell FROM eclipse-temurin:21-jdk-alpine AS build WORKDIR /build COPY . . RUN ./mvnw -B package -DskipTests # Explode the jar so the classpath is stable RUN java -Djarmode=tools -jar target/app.jar extract --destination /app FROM eclipse-temurin:21-jre-alpine AS runtime WORKDIR /app COPY --from=build /app /app # Training run: start the context, record classes, exit RUN java -Dspring.context.exit=onRefresh \ -XX:ArchiveClassesAtExit=/app/app.jsa \ -jar /app/app.jar ENV JAVA_TOOL_OPTIONS="-XX:SharedArchiveFile=/app/app.jsa" ENTRYPOINT ["java", "-jar", "/app/app.jar"] Two details in there deserve a closer look. The extract step unpacks the fat jar into a folder with the dependencies laid out as plain files. CDS is picky about the classpath being identical between training and real runs, and a fat jar with nested jars inside it makes that fragile. The exploded layout keeps the classpath boring and stable, which is exactly what CDS wants. On Spring Boot 3.2 and older, the same idea works through the layertools jarmode instead. The training run happens as a RUN instruction, which means it executes once at build time on your CI server. Every container that ever starts from this image inherits the archive for free. You did the class loading homework once, in the build, and ten thousand pod starts copy the answer. What You Get Numbers vary with how heavy your application is, but the pattern is consistent. A typical Spring Boot 3 web service that started in 10 to 12 seconds lands somewhere between 5 and 7. The JVM portion of startup shrinks dramatically, and as a bonus, the archive is memory-mapped and shared, so if you run several JVMs on one node, they share those pages and total memory drops too. You can verify the archive is actually being used, which I recommend, because CDS fails silently by design. If something mismatches, it just quietly falls back to normal class loading: Shell docker run --rm my-service -Xlog:class+load=info | head -5 Classes loaded from the archive say source: shared objects file. If you see jar paths instead, the archive is being ignored, and the log will usually tell you why. The usual culprit is a classpath that differs from training, even by one entry. One honest caveat. The training run exercises startup, not your traffic. Classes that only load when a specific endpoint gets hit for the first time are not in the archive, so those first requests still do normal loading. The archive covers the framework and wiring, which is most of the cost, but it is not a magic warm-up for everything. Why This Beats the Alternatives You Have Heard Of Whenever container startup time comes up, someone mentions GraalVM native images, and native images are impressive. Millisecond startup is real. But they come with a price list: long build times, a closed-world assumption that fights with reflection, some libraries that simply do not work, and a different runtime profile you have to learn to debug. CDS costs you five lines of Dockerfile. Your application is still a completely normal JVM application. Same debugging, same profilers, same libraries, same behavior, just faster out of the gate. For most teams, that trade-off is not even close. It also stacks with what is coming. Project Leyden's AOT cache in Java 24 and beyond is essentially this same idea grown up, caching not just parsed classes but resolved linkage and compiled code. The Dockerfile pattern you build today, a training run at build time producing a cache file shipped in the image, is exactly the shape Leyden uses. Learning it now means the future is a flag change. The Takeaway Your Docker image is immutable. Your JVM does expensive, perfectly repeatable work on every startup. Those two facts fit together like puzzle pieces, and a training run inside docker build is where they connect. One extra build step, and every pod your autoscaler ever creates comes up in half the time. The next time you watch a rollout crawl because pods take forever to go ready, remember that the answer was hiding inside the build all along.

What if you could multiplex roughly 250 stateful agent sessions across eight Kubernetes worker Pods, then reactivate any one without losing its in-memory or filesystem state? The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. Agent Substrate is interesting not because it makes Kubernetes faster, but because it challenges a common deployment pattern around Kubernetes: coupling a workload's logical lifecycle to the compute allocated to run it. For AI agents that spend most of their time idle while retaining valuable state, separating those lifecycles could become an important building block for operating agents at significantly higher density. Kubernetes gave us an exceptionally durable abstraction for running workloads: the Pod. But emerging agent workloads expose places where that abstraction may become inefficient. They can be stateful, sandboxed, and overwhelmingly idle. A one-agent-per-Pod deployment model, while simple, couples each session's logical lifecycle to a Pod's runtime lifecycle. When sessions spend much of their time idle, that coupling can leave compute capacity allocated to workloads that are not actively executing. This is not a claim that Kubernetes is obsolete. It is an exploration of where Kubernetes remains the right substrate--provisioning capacity, managing worker Pods, and enforcing infrastructure policy--and where an agent-specific control plane may need a faster path for high-frequency lifecycle operations. The Thesis: Make Running Optional The central idea is surprisingly simple: an agent does not need to occupy compute merely because it exists. Agent Substrate calls an instance of a managed workload an actor. The deliberately broader term matters: an actor does not have to be an AI agent; it can be any OCI workload that benefits from being bursty, checkpointable, and independently suspendable. The system provides an agent-oriented workload runtime and control plane; it is not an agent framework or SDK. An actor can be suspended into a snapshot containing process memory, filesystem state, or both. The worker Pod is then freed. When another request arrives, the system restores the actor to a ready worker and routes the request to it. In this model, the actor becomes the logical workload, while the Pod becomes temporary compute capacity. That gives the architecture three defining properties: Warm capacity replaces per-session capacity. A smaller pool of ready workers serves a much larger population of actors over time.State survives worker reassignment. The next activation need not use the worker that ran the actor previously.Requests can initiate activation. The router can hold a request while the control plane brings a suspended actor back. The project's architecture document defines north-star targets including 100 ms p95 activation, one billion active and idle actors per cluster, and 1,000 wakeup events per second. These are architectural targets, not production benchmarks or guarantees. That distinction is important because the repository itself is candid that large parts of the architecture are still evolving. Architecture at a Glance The control flow becomes easier to follow once the logical workload is separated from the physical capacity: Why the Pod Becomes an Awkward Unit for Agents A conventional one-Pod-per-session deployment model can become inefficient when sessions spend most of their lifetime idle but retain valuable state. Agent-like workloads have a different shape: They wait much more than they compute.They may execute untrusted code, so multi-tenancy often means one sandbox per session.They keep useful state in memory and local filesystem changes.Their active periods can be short enough that creating and initializing a dedicated Pod for each session becomes noticeable user-facing latency. In a one-Pod-per-session design, the logical unit a user cares about — a coding session, sandboxed tool, or stateful agent — does not align cleanly with the physical unit Kubernetes schedules: a Pod. One straightforward approach is to keep each session's Pod alive. Agent Substrate asks whether that physical allocation can be temporary instead. Its answer is a pool of pre-provisioned worker Pods plus a separate actor record that tracks identity, lifecycle state, placement, and snapshots. This is an important architectural departure from Kubernetes' conventional control-plane model. Kubernetes intentionally optimizes for declarative desired state and asynchronous reconciliation. Agent Substrate moves high-churn actor state out of the Kubernetes API machinery so that wakeup, placement, and snapshot transitions can occur without making each actor a Kubernetes object. The point is not that Valkey or Redis is universally "faster than etcd." The interesting boundary is low-frequency desired state versus high-frequency runtime state: the two have different frequency and latency profiles, so the system gives them different control paths. Three Planes of State The resource model separates declarative configuration from dynamic runtime records. Operationally, snapshot contents form a useful third plane: STATE TYPEWHERE IT LIVESWHYActorTemplate, WorkerPool, and SandboxConfigKubernetes CRDsLow-frequency infrastructure configuration benefits from Kubernetes RBAC, auditability, and reconciliation.Actors, workers, assignments, lifecycle state, and snapshot referencesControl-plane store (Redis/Valkey by default; experimental PostgreSQL support is also available)These records change on lifecycle transitions and need low-latency reads and writes.Snapshot contentsNode-local storage for Pause; object storage for snapshots committed during SuspendSnapshot scopes trade off locality, durability, and transfer cost. An ActorTemplate defines an actor class: its container image, snapshot behavior, and compatible worker selection. A WorkerPool declares warm Pods. An Actor is a specific instance that moves between workers through its lifetime. Here is a trimmed ActorTemplate from the repository's multi-template demo: YAML apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: counter spec: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter workerSelector: matchLabels: workload: multi-template-shared The important omission is a dedicated Pod. The template describes the workload and selects compatible reusable capacity; a separate WorkerPool provides the warm Pods, while the actor's identity and lifecycle remain independent of whichever worker hosts it. The deeper architectural pattern is a separation of three related lifecycles: infrastructure, workload, and execution. Kubernetes manages infrastructure capacity; the actor control plane manages logical workload identity, placement, and lifecycle; snapshot and sandbox machinery preserve and reconstitute execution state. Once those lifecycles are separated, a worker Pod becomes a reusable execution slot rather than the identity of the workload itself. An actor is addressed by (atespace, name), not by name alone. That is more than a naming detail: the glossary defines an atespace as a logical actor isolation boundary, not a replacement for Kubernetes namespaces or a sandbox security boundary. The same actor name can exist in different atespaces. The atespace also appears in the actor's routable DNS name: Plain Text <actor-name>.<atespace>.actors.resources.substrate.ate.dev This is the first place the project begins to look less like a set of Kubernetes objects and more like a runtime: stable logical identity remains while the physical worker assignment changes. The Request Path: Routing Becomes Placement The most consequential design choice is that ingress is part of activation. The networking architecture provides the actor DNS model and an Envoy-based router. The router's ext_proc handler reads the actor reference from the request authority, calls the control plane to ensure that actor is running, and then selects the assigned worker as the upstream. Plain Text Error: Parse error on line 22: ...atunnel Ateom->>Actor: Forward over ----------------------^ Expecting '+', '-', '()', 'ACTOR', got 'participant_actor' The ingress detail matters. The router does not forward directly to the actor's application endpoint. It opens an mTLS connection to the worker's atunnel listener. atunnel validates the router and forwards only to the actor currently assigned to that worker. That makes routing part of the security boundary, not merely service discovery. There is also a practical admission-control insight here. A saturated worker pool should not turn a burst of requests into an unbounded queue. The router can park a bounded number of requests while it retries transient capacity and control-plane conditions; once the parking limit is reached, it sheds new work. Activation latency is therefore not just a restore-time problem. It is also a backpressure problem. What Happens During Suspend and Resume The control plane coordinates a distributed workflow rather than pretending this is a single atomic operation. For a resume, it locks the actor, reads its state and template, selects an eligible idle worker, asks the node-level supervisor to restore a snapshot or cold boot, and marks the actor running only after the worker is ready. For a suspend, it checkpoints state, persists the requested snapshot scope, clears the worker assignment, and returns the worker to the pool. The details reveal deliberate distributed-systems trade-offs. In the default Redis backend, actor and worker records are separate keys that may occupy different cluster slots, so they cannot be updated in one cross-slot action. The implementation uses per-record version checks, actor locking, ordering, retries, and idempotent workflow steps to coordinate these transitions. The architectural implication is that lifecycle operations are treated as recoverable workflows rather than atomic infrastructure mutations. A repeated lifecycle call can discover completed steps and move forward instead of blindly redoing them. The Worker Is a Reusable Sandbox, Not the Actor Below the control plane, atelet runs as a DaemonSet and manages the node-side work: image preparation, OCI bundle assembly, snapshot transfer, and communication with the worker. ateom runs inside the worker Pod and drives the sandbox runtime. The repository currently defines gVisor and microVM sandbox classes. In the gVisor path, ateom drives runsc checkpoint and restore. Depending on the configured scope, snapshots can preserve process and filesystem state, allowing an actor to resume later on another worker. This is why the demo can show an in-memory counter continuing after a suspend/resume cycle: the application was restored, rather than restarted from scratch. The security model should be described with care. Sandboxing and mTLS are real implementation elements, and the project has a detailed threat model. But that document explicitly says security hardening remains early. A fair reading is that Agent Substrate is making the right boundaries visible--sandbox, worker reuse, actor identity, snapshot access, and router-to-worker authentication--rather than claiming those boundaries are already production complete. What the Demos Prove--and What They Do Not The most accessible proof is the counter demo. A tiny HTTP service increments an in-memory counter. Create an actor in an atespace, send requests through atenet-router, suspend it, and resume it. The counter continues. The demo makes the abstract claim concrete: memory and filesystem state can outlive a worker assignment. The README's published density demonstration goes further: about 250 stateful actors multiplexed across eight physical worker Pods. The repository also includes examples for Claude Code multiplexing, request parking, autoscaled worker pools, and different templates sharing a worker pool. These examples validate the model and its developer experience. They do not prove the project's one-billion-actor target, production reliability, or a universal cost model. Treating that distinction honestly makes the architecture more interesting, not less: the open questions are precisely where the difficult engineering begins. From Traffic Locality to Compute Locality My previous DZone article, "Zone-Aware Routing in Kubernetes", examined a related infrastructure question: how should a platform place traffic so requests stay local when that improves latency, resilience, or cost? That work led me to a broader question: if locality matters for packets, what happens when locality also matters for stateful compute? Zone-aware routing asks where traffic should go. Agent Substrate raises a harder question: where should the compute state itself live when a workload can disappear from one worker and reappear on another? This turns locality from a networking concern into a workload-lifecycle concern. That change has consequences: Scheduling cannot be evaluated only by where free CPU exists; snapshot location and resume cost matter too.Routing cannot be evaluated only by endpoint availability; it can trigger a state transition.Security cannot stop at the Pod boundary; worker reuse and snapshot access become first-class concerns.Autoscaling cannot only count replica demand; it must account for how long actors remain active, parked, or suspended.Storage cannot be treated as an afterthought; snapshot placement, transfer time, durability, and locality become part of the activation path. This suggests a broader infrastructure question for agent workloads. The challenge is not simply running more containers; it is hosting large populations of mostly-idle, stateful, potentially untrusted processes without allocating dedicated compute to each one. Where the Hard Work Remains The project is unusually direct about its unfinished work: control-plane performance and reliability, worker autoscaling, identity and policy, actor network isolation, storage design, observability, and support for different sandbox runtimes all remain active areas of development. Those concerns are not peripheral; they determine whether the architecture can operate reliably at the scale it targets. A system that makes activation fast must still decide how to shard state, restore safely, apply policy before execution, isolate one actor from the state left by another, and reason about locality without turning every wakeup into a storage bottleneck. Four questions are especially important: Snapshot locality. If an actor's state is remote, resume latency becomes partly a storage and network-transfer problem.Snapshot correctness. Checkpointing a live process is not equivalent to serializing application state. Open connections, timers, external leases, credentials, and dependencies can make a restored process semantically different from a freshly initialized one.Activation bursts. Multiplexing improves average utilization, but a correlated wake-up event can turn many inexpensive idle actors into a sudden demand spike. The system therefore needs admission control and worker autoscaling that respond to activation pressure, not only steady-state utilization.Fairness. A small number of highly active actors can monopolize workers unless scheduling and admission control account for competing demand. Agent Substrate is therefore more compelling as an emerging architectural pattern than as a product claim. Its value is in making these trade-offs explicit and providing a runnable implementation that exposes where the abstractions are strong and where they remain unfinished. Conclusion The Pod is unlikely to disappear. But it may stop being the only unit we think about when we build infrastructure for agents. Kubernetes remains a powerful system for provisioning and operating compute. Agent Substrate is exploring what happens when the logical lifecycle of an agent is separated from the lifecycle of the Pod that temporarily runs it. If agents become ubiquitous--long-lived, intermittently active, stateful, and capable of executing untrusted code--the infrastructure challenge will not simply be running more Pods. It will be deciding where an agent should exist when it is inactive, how quickly it can become active, and how efficiently thousands or millions of them can share the same underlying compute. That is the problem Agent Substrate is attempting to solve. At a billion actors, the central question is no longer how to run more agents. It is how to make running optional. Further Reading Agent Substrate repositoryArchitectureThreat modelRequest parkingCounter demo Agent Substrate is Apache-2.0 licensed, explicitly not an officially supported Google product, and in active early development. The architectural analysis and opinions in this article are my own.

For two decades, we focused on moving data to the intelligence. Now, we’re seeing a massive shift: we have to move the intelligence to the data. That flip changes everything. Your host platform isn’t just an API gateway anymore; it’s an operating system. The Meeting That Wasn’t About Models The meeting that changed how I think about AI infrastructure had almost nothing to do with models. We spent months obsessing over model quality. Then, over a few weeks, the agenda quietly reorganized itself. We were talking about onboarding third-party units. We debated what happens when two versions of the same model disagree under replay. We worried about whether one tenant’s inference could starve a neighbor’s on a shared accelerator. We fought over who pays for a millisecond. At some point, I wrote down what was actually on the whiteboard: routing, versioning, isolation, admission control, resource accounting, governance boundaries, and latency budgets. That is not a machine learning agenda. That started to look more like an operating systems agenda. We had stopped solving an AI problem and started designing a runtime. It happened the way architecture usually happens, as an accumulation of decisions that only later reveal their shape. This article names and outlines that shape. I call it the Portable Intelligence Architecture, or PIA. It is not a product, a vendor category, or a rebranding of edge ML. It is an architecture pattern that several teams appear to be converging on independently, and an argument that the runtime layer deserves to be treated as a first-class architectural concern rather than an implementation detail discovered later in production. Twenty Years of Moving Data to Intelligence We’ve been living in an API-first world because, for a long time, the math was simple: intelligence was expensive and centralized, payloads were small, and network costs were rounding errors. It made sense to ship the data to the model. That premise was right for its time. If you’re running a proprietary fraud model that needs nightly retraining and a custom feature store, you don't ship that code to the caller. You expose an endpoint. The caller sends a few kilobytes, and you send back a score. That made sense at the time. Every abstraction we’ve built lately has just been a refinement of where that intelligence lives. Libraries. APIs. Microservices. Containers. Portable Intelligence Units. I see this as a ladder. Libraries were linked. APIs were called. Microservices were deployed. Containers were scheduled. Each step made the unit more self-contained and independent. A Portable Intelligence Unit (PIU) is just the next rung. It’s a versioned, resource-declared unit that brings the inference straight to the host where the context already lives, and at scale. The old API-first rule isn't dead, but it’s become conditional. In my experience, three things changed the game. Accelerators became first-class citizens. GPUs and LPUs aren't "specialized hardware" anymore. They're just another resource class for the scheduler. Once you can allocate inference hardware like memory, deploying a model locally becomes a standard infra task, not a massive research project. Models got smaller. Thanks to distillation and quantization, a killer model can now be a few gigabytes. When the model is smaller than the context it needs to digest, moving the model is the only logical choice. Context is live and now at scale. The signal that matters for enterprise decisions isn't a static prompt. It's live inventory, session data, and real-time supply conditions. That state is too heavy and sensitive to export. It stays where it is. Just think about all the session data the new LLM chats are generating. I first assumed portable intelligence would mostly mean shipping simple decision logic outward: rules, gradient-boosted trees, small classifiers, the kind of thing you push to the edge because it is cheap to push. My assumption was wrong. Modern accelerator economics mean we can run heavy-duty models right at the point of decision. The old constraints that forced us to keep things simple at the edge have almost vanished. Where APIs Alone Become Insufficient Remote inference isn't going away, but I've found it’s structurally insufficient for a certain class of high-performance systems. Here are the four forces we keep running into. Context gravity. We’ve known for years that you move computation to the data when the data is expensive to ship. But now, it’s about fidelity. If you try to export the live state to a remote endpoint, you have to freeze and flatten it. You don't just lose bandwidth; you lose the predictive relationships that make the model work.The millisecond wall. When you have a 10ms budget, a remote round trip isn't just "slow." It's a non-starter. You can't optimize your way out of a budget the network ate before your code even touched the CPU.Governance and silos. Between data residency laws and partner contracts, the data often physically cannot move. Moving the model is frequently the only way to stay compliant. The model crosses the fence; the data doesn't. Think GDPR, etc.The cost of scale. Per-call pricing is great when you're small, but at high volume, it'll kill your margins. We usually discover this late, and it’s a painful, expensive lesson to learn. These forces don't kill the API, but they force us to build a second plane where we deploy intelligence instead of just calling it. Remote inference Portable intelligence Context moves. Model moves. Boundary crossed by data. Boundary crossed by artifact. Cost unit is per call. Cost unit is per allocated compute. Failure mode is tail latency. Failure mode is operational drift. Portable Intelligence Architecture In simple terms, PIA is an architecture where you package inference into a portable, versioned unit and drop it into a host runtime that already owns the data. It has three parts. A Portable Intelligence Unit, or PIU, is the deployable artifact: a model plus its preprocessing, its declared resource envelope, its data contract, its version identity, and its governance metadata. A PIU is not “a container with a model in it.” The container is packaging. The contract is the architecture.The host is the runtime that admits, schedules, routes to, isolates, meters, and observes PIUs. It owns the context that made deployment worthwhile in the first place.The decision plane is the hot path where PIUs execute against live context under a latency budget. It is deliberately separated from the control plane, which handles admission, versioning, policy, and rollout on a different time scale. Running a model near data isn't new; we've been doing that with embedded code for decades. The real innovation here is the multi-tenant, contract-governed host. We're talking about units from different teams, on different release cycles, all sharing one runtime to make one decision. That’s a runtime problem, plain and simple. And luckily, operating systems have already given us the blueprint. Figure 1. The control plane governs the host runtime, which routes requests across Portable Intelligence Units and reduces their outputs into a decision. The Five Principles Move the smaller thing. The architectural question is never “should intelligence be central or distributed.” It is “which artifact is cheaper to move in bytes, in fidelity, and in legal exposure? Is it the model or the context?” Answer that honestly per workload, and the topology falls out. It will not be the same answer for every workload in the same system.The host is an operating system, not a gateway. A gateway routes requests. An operating system admits programs, isolates them, schedules them against finite resources, accounts for what they consume, mediates access to privileged state, and defines the interface through which they ask for more. Once you have multiple third-party units sharing accelerators under a shared latency budget, you are building the second thing whether you intended to or not. Building it deliberately is cheaper.Optimize the composition, not the component. This is the principle I would have argued against a year ago. Marginal model quality is usually not the binding constraint. Routing, admission, fallback, and sequencing of specialized units produce more end-to-end improvement than another point of accuracy on any single unit. The system-level win comes from composition, and composition is an orchestration property.Operational properties are the contract. Latency, versioning, governance, and cost are not things you tune after the design. They are the declared interface of a unit. A PIU that does not declare a p99 budget, a data-class permission set, and a resource envelope cannot be safely admitted, because the host has no basis for scheduling it. Treat these as first-class contract fields, or you will enforce them later with incident reviews.Assume re-entry. Single-shot inference is the easy case and increasingly the minority case. Design the interface between a unit and the host assuming that intelligence will call back into the runtime mid-reasoning for a lookup, a tool, or another unit. This is the hardest principle to satisfy, and I will be honest about the fact that we have not fully solved it. Composition Is Where the Value Is The single most consequential thing we learned was not that one sophisticated model could be deployed at the decision point. It was that several specialized units working together were worth substantially more than one general unit. The reasons are structural, not empirical: Specialized units are small, which makes them cheap to schedule and fast to load.They can be owned by different teams, or different companies, and released independently.They can be versioned independently, which means a regression is contained.The routing decision between them is itself cheap, so composition costs less than you would guess. The router is where the architecture actually lives. It decides which units see a request, whether they run in parallel or in sequence, what happens when one exceeds its budget, and what the fallback path is. Every hard question in this pattern eventually becomes a router question. The Host Becomes an Operating System The analogy is not decorative. It is predictive: it tells you which problem you will hit next. Operating system concept PIA equivalent Process Portable Intelligence Unit Scheduler Router and admission controller Memory protection Tenant and data-class isolation Resource accounting Per-unit compute metering and attribution System calls Callback interface from unit into host Package management Partner onboarding and unit registry Permissions Governance policy on data classes Device drivers Accelerator abstraction If you take one thing away, let it be this: keep your control plane and decision plane strictly separate. It’s the highest-leverage move you can make. The control plane handles admission, registry, version promotion, policy, entitlements, cost models, rollout, and rollback. The decision plane handles request context, routing, inference, composition, fallback, and emitting the decision. When you let control-plane junk leak into the decision path, like a registry call during routing, you’re coupling a minutes-scale system to a milliseconds-scale one. Your tail latency will let you know exactly why that was a mistake. The Contract Look at the manifest below. These fields aren't just metadata; they're everything the host needs to schedule, isolate, and govern a unit without ever needing to call the owner. Figure 1: PIU Manifest JSON JSON { "piu": { "id": "risk-scorer", "version": "4.2.1", "owner": "partner:northwind", "artifact": { "image": "registry.internal/piu/risk-scorer@sha256:9f2c...", "signature": "cosign:...", "format": "onnx" }, "resources": { "accelerator": { "class": "gpu.small", "count": 1 }, "memory_mb": 6144, "max_concurrency": 32 }, "budget": { "p50_ms": 4, "p99_ms": 18, "timeout_ms": 25, "on_exceed": "fallback:[email protected]" }, "data_contract": { "input_schema": "schemas/[email protected]", "compat": "backward", "required_fields": ["entity_id", "signal_vector", "channel"], "null_policy": "reject" }, "governance": { "data_classes": ["pseudonymous", "aggregate"], "prohibited_classes": ["pii", "cross_tenant"], "residency": ["eu"], "audit": "sampled:0.01" }, "economics": { "billing_unit": "accelerator_ms", "attribution": "tenant" }, "capabilities": { "reentrant": false, "max_callbacks": 0 } } } Two fields in that manifest are doing far more work than the rest. data_contract.compat is where partner integrations actually succeed or fail. capabilities.reentrant is where the open problem lives. Getting this onto Kubernetes is the easy part once you have the contract. The substrate is just plumbing; the contract is the architecture. Figure 2: Kubernetes Deployment Manifest YAML apiVersion: apps/v1 kind: Deployment metadata: name: piu-risk-scorer labels: piu.host/id: risk-scorer piu.host/version: "4.2.1" piu.host/tenant-class: partner spec: replicas: 6 selector: matchLabels: piu.host/id: risk-scorer template: metadata: labels: piu.host/id: risk-scorer piu.host/version: "4.2.1" spec: nodeSelector: accelerator.class: gpu.small containers: - name: unit image: registry.internal/piu/risk-scorer@sha256:9f2c... env: - name: PIU_MANIFEST value: /etc/piu/manifest.json - name: HOST_CALLBACK_SOCKET value: /var/run/piu/host.sock resources: limits: nvidia.com/gpu: 1 memory: 6Gi requests: memory: 6Gi readinessProbe: httpGet: path: /healthz/warm port: 8080 initialDelaySeconds: 20 volumeMounts: - name: host-socket mountPath: /var/run/piu volumes: - name: host-socket hostPath: path: /var/run/piu type: Directory Pay attention to the readiness probe. If a unit says it's ready before the accelerator memory is loaded, it'll start taking traffic it can't handle. That’s how you would end up exceeding your p99. Operational Lessons, Ranked By Cost Partner onboarding is the hardest problem, and it is not a packaging problem. I expected packaging friction. What we actually hit was data compatibility. Two teams agree on a schema, ship against it, and still fail because one side’s “session” means something subtly different from the other’s, or a field is nullable in practice but not in contract, or the feature distribution the unit was trained on does not match the distribution the host produces. Schema compatibility is necessary and nowhere near sufficient. What moved the needle was a developer protocol: an SDK, a conformance test suite, and golden datasets that a partner could run before ever touching our environment.Versioning models is easy. Testing across versions is not. Tagging a version takes an afternoon. Knowing whether version 4.2.1 behaves acceptably across every tenant, every context distribution, and every composition path it participates in is a combinatorial problem. Shadow traffic and production replay are the only honest tests I know of. Unit tests on models are theater.Multi-tenancy on accelerators is harsher than on CPU. Noisy-neighbor effects that are annoying on CPU are structural on shared accelerators, particularly once batching enters the picture. Batching couples tenants’ latency profiles: one tenant’s traffic shape now determines another tenant’s tail. Either isolate hard and pay for it, or accept the coupling explicitly and model it.Design the cost model before deploying sophisticated inference. This should be a boring statement, but it isn't. If you deploy first, you learn your unit economics from a bill, in arrears, after the architecture has ossified. Decide early what the billing unit is, whether it's accelerator-milliseconds, admitted requests, or allocated capacity, because that choice propagates into routing policy and eventually into what you can sell.Average latency is a vanity metric. p99 is the product. Dean and Barroso made this point over a decade ago, and it applies with extra force here because composition fans out. If a decision touches four units and each has a well-behaved tail, the composed tail is worse than any individual one. Budget the composition, not the components. Enforce timeouts at the router with a defined fallback, and treat fallback as a normal outcome rather than an error. Tradeoffs, Stated Plainly PIA buys latency, governance topology, and cost predictability at high volume. It costs the following: Operational surface area. You now operate a runtime. That is a permanent staffing commitment, not a project.Debugging across trust boundaries. When a composed decision is wrong, and three of the units belong to other organizations, root cause becomes a negotiation.Supply chain risk. Admitting third-party inference into your runtime is admitting third-party code into your runtime. Signing, scanning, and resource limits are table stakes, not maturity.Freshness. Centralized models update on one cadence. Distributed units update on many. Some drift is now a design parameter rather than an accident.Capacity planning. Accelerator capacity is lumpy, and lumpy capacity plus strict latency budgets means paying for headroom you do not use. When Not to Use PIA Do not build this if: Your volume is low. Per-call pricing is a gift at low volume; take it.Your latency budget is loose. If 200 ms is fine, call the API.There is one model, one owner, one release cadence. You have a deployment, not a runtime.Your context is small and legally exportable. Then the context is the smaller thing to move, and principle 1 tells you to move it.Your models are iterating weekly. Central deployment has a much shorter feedback loop, and early-stage model velocity beats architectural elegance every time. The pattern earns its complexity at the intersection of high volume, tight budgets, multiple owners, and immovable context. Outside that intersection, it is overhead with a nice diagram. The Future Is Agents Orchestrating Units The direction this is heading is not one enormous model at the decision point. It is an agent orchestrating many specialized Portable Intelligence Units, selecting and sequencing them dynamically based on the decision at hand. That future is architecturally coherent right up until it hits the problem we have not solved. Iterative workflows break the budget model. Single-shot inference has a clean contract: the host gives a unit context, the unit returns a result inside a declared budget. Re-entrant reasoning does not work that way. A unit pauses mid-reasoning, calls back into the runtime for a lookup or another unit’s output, and resumes. Now the budget is not a duration; it is a session with an unknown number of stages, holding accelerator memory the whole time. Every mechanism the host relies on gets harder: admission control cannot know the cost of admitting a request, scheduling has to handle units that are resident but idle, fair-sharing has to prevent one long chain from starving short ones, and tracing has to reconstruct a call graph that did not exist at admission time. Operating systems solved the analogous problems with preemption, quotas, and priority scheduling over roughly thirty years. I do not think we get a shortcut. But I do think naming the problem correctly is most of the work, and the correct name is scheduling, not prompting. Conclusion Portable Intelligence Architecture does not replace APIs. It complements them. Most systems will run both planes, and the interesting design work is deciding which decisions belong on which plane. What has changed is where the difficulty sits. For most of the last decade, the limiting factor in enterprise AI was model quality, and the industry organized itself accordingly. That is no longer where the constraint binds. The models are good enough for a large and growing set of enterprise decisions. The new bottleneck is the runtime. We need a layer that lets portable intelligence run safely and at scale across different owners and tight budgets. We stumbled into our runtime one whiteboard at a time. Trust me, it’s much cheaper to build it on purpose. This isn't an implementation detail; it’s a first-class architectural concern. References Gray, J. Distributed Computing Economics. Microsoft Research, 2003.Dean, J. and Barroso, L. A. The Tail at Scale. Communications of the ACM, 2013.Barroso, L. A., Clidaras, J., Hölzle, U. The Datacenter as a Computer. Morgan & Claypool.Sculley, D. et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.Dehghani, Z. Data Mesh: Delivering Data-Driven Value at Scale. O’Reilly, 2022.Kubernetes documentation: Device Plugins and Dynamic Resource Allocation.ONNX: Open Neural Network Exchange specification.

Most teams don't decide to build microservices. They get pushed into it. One app grows for a couple of years. More people push into the same codebase. Then a change to something totally unrelated breaks checkout on a Tuesday. Nobody planned that. That's usually when someone says it, half-joking, half not: maybe we should just split this thing up. And Node.js is the name that comes up. Not because anyone ran a deep framework comparison. Honestly, half the time it's already running the API layer and chewing through small request/response calls all day, so nobody has to fight for it. It's already there. Easiest sell in the room. What people get wrong going in: the win isn't "we use Node.js now." It's narrower than that. Node.js microservices earn their keep when a service actually needs to scale on its own — checkout during a flash sale, say, while the blog section sits idle. Split things up without that need, and you haven't built microservices. You've built one tightly coupled app, just now with network calls between the pieces instead of function calls. Same mess. Slower. The real work is designing the microservices architecture in Node.js properly: keeping services loosely coupled, getting them to talk without one outage taking three other services down with it, and figuring out which pieces genuinely need their own database versus which ones are fine sharing. That's what this covers. Core Architecture Components A Node.js microservices architecture usually has the same handful of pieces, even if the specifics change from one company to the next. componentpurposecommon tools API Gateway Routes requests, handles auth, rate limiting Express Gateway, Kong, NGINX Service Framework Builds individual business services Express, Moleculer Synchronous Calls Request/response between services axios, fetch, gRPC Async Messaging Event-based communication RabbitMQ, Kafka Resiliency Prevents cascading failures Opossum (circuit breaker) Containerization Isolates services and dependencies Docker Orchestration Scaling, restarts, rollouts Kubernetes Logging Centralized, searchable logs Winston, Pino Monitoring Tracks performance and health Prometheus, Grafana 1. API Gateway Clients never talk to your services directly. They hit the gateway first, and it figures out where the request needs to go. This is usually also where auth checks happen and where rate limiting lives, so one client can't flood the system with requests. 2. Individual Services Behind the gateway are the actual services, each one handling a single piece of the business: orders, users, whatever it is. Express is still the default choice for building these. Some teams are moving to Moleculer instead, since it's built specifically for microservices rather than being a general framework stretched to fit. When choosing a Node.js microservices framework, the right option depends on how much infrastructure your team wants the framework to handle. 3. Database Per Service This is the corner teams cut, and it always shows up later, usually a few months in, once nobody remembers why the shortcut got taken. If the order service and the user service are both querying the same database, you don't actually have two services. You have one database wearing two name tags. Each service needs to own its data, full stop. Need something from another service? Ask through its API, or listen for the event it fires. Don't go around the back and query its tables directly; that's the shortcut that turns into a rewrite. 4. Message Broker Not every interaction needs an answer right away. When someone places an order, the order service shouldn't sit around waiting for a confirmation email to go out; it fires off an event and moves on to the next request. Something else, usually RabbitMQ or Kafka, is listening for that event and deals with it on its own time. How to Build Microservices With Node.js The honest answer to how to build microservices with Node.js is: don't start by spinning up five repos. Start by figuring out where the actual boundaries are. Each service needs to own one business capability, its data, its logic, everything it needs to run without leaning on another service to function. A practical Node.js microservices tutorial usually comes down to a sequence like this: Define service boundaries: Figure out the independent business functions: users, orders, payments, notifications, whatever they are for you.Create a Node.js project for each service: Deployable on its own, with its own dependencies and config. Not a shared node_modules folder pretending to be independent.Choose the right framework: Express is fine for lightweight services; reach for a dedicated Node.js microservices framework such as Moleculer when you need more built-in.Expose APIs: Give each service a clean REST or gRPC interface for anything synchronous. This approach keeps building microservices with Node.js focused on business boundaries rather than simply splitting a large codebase into smaller applications. Node.js Microservices Example A simple Node.js microservices example could be an e-commerce application divided into four services: User service: Manages customer accounts and authentication.Product service: Handles product information and inventory.Order service: Creates and tracks customer orders.Notification service: Sends email or other order-related notifications. For example, when a customer places an order, the Order Service can publish an order.created event. The Notification Service listens for that event and sends the confirmation without forcing the Order Service to wait for the email process to finish. This is also a practical example of how to create microservices in Node.js: start with independent business capabilities, expose only the interfaces other services need, and use events when a response isn't required immediately. Communication Strategies Some requests need an answer right away. Others just need to notify another service that something happened, and nobody's waiting on a response. Most Node.js microservices setups use a mix of both. Synchronous Calls One service asks, waits, gets an answer back. That's really it. Most of the time plain HTTP is enough: axios or fetch, nothing fancy. gRPC only earns its keep once two services are hammering each other with requests constantly and the JSON overhead starts showing up in your latency numbers. It runs over HTTP/2, uses Protocol Buffers, and has smaller payloads. Asynchronous Messaging Different situation. Order comes in; the order service doesn't need to hang around until the confirmation email actually sends; it just says done and picks up the next request. Somebody else deals with the email later. RabbitMQ if you care about routing, sending different messages down different paths. Kafka if you're dealing with volume, logs, activity streams, stuff that never really stops flowing. Design Patterns for Resiliency Distributed systems fail in ways a single app never does. One service going down shouldn't mean the whole system goes down with it, so a few patterns exist specifically to contain that damage. Circuit Breaker Something's failing, so the instinct is to retry, and retrying just adds load to a service that's already drowning. A circuit breaker cuts that off. After enough failures in a row, it stops sending requests to that service for a stretch and lets it recover instead of burying it further. Opossum is what most people reach for in Node.js when they're setting this up. Saga Pattern You can't roll back a transaction across three different databases the way you'd roll back one. So instead of a single transaction, you get a chain; each step commits on its own in its own service. If step four fails, you don't just stop; you run backward through one, two, and three, undoing what already happened. It's not clean. It's what you're left with once one database per service is no longer optional. Idempotent Consumers Networks resend messages sometimes; that's just how it goes. If your order service can't tell a retry apart from a brand new order, you end up double-charging someone eventually. A uniqueness check on the event solves most of this; before acting on a message, the service checks whether it's already seen it. Dead Letter Queues Some messages are never going to process no matter how many times you retry them: bad data, a broken payload, whatever the cause. Rather than let one bad message jam everything behind it, it gets pulled into its own queue and dealt with separately later, instead of stalling the rest of the line. Production Deployment and Observability Getting this running locally is one thing. Running it in production with actual traffic is where most of these decisions get tested for real. Containerization Each service, along with its database and anything else it depends on, gets wrapped in its own Docker container. This keeps one service's dependencies from clashing with another's, and it means what runs on your laptop is basically the same thing that runs in production, no more "works on my machine." Orchestration By the time you've got more than two or three containers, doing this manually just doesn't hold up. Kubernetes takes that off your plate; more traffic comes in, it spins up more instances on its own. Something crashes, it gets restarted without anyone needing to notice at 3 am. Rolling out a new version doesn't mean downtime either; it shifts traffic over gradually. And on the security side, secrets management means your API keys aren't just sitting in a config file somewhere waiting to get committed to git by accident. Centralized Logging Logging to a file on each individual server doesn't work once you've got a dozen services running across different machines. Nobody's going to SSH into ten boxes trying to piece together what happened. Tools like Winston or Pino send structured logs somewhere central instead, so you can actually search across everything at once when something breaks. Metrics and Monitoring The goal is finding out something's wrong before a user emails you about it. In a Node.js system specifically, event loop lag is the one to watch closely; a blocked event loop doesn't throw an error, it just quietly slows everything down until someone notices things feel off. Memory usage and response times matter too, obviously. Prometheus is usually what's pulling these numbers together, and Grafana is where you'd actually go look at them. Wrapping Up None of this is complicated on its own: gateway, services, a message broker, some way to keep failures from spreading. What makes it hard is doing all of it at once, correctly, while the system is already handling real traffic and you don't get a do-over if you get the database boundaries wrong on day one. Node.js fits well here mostly because it doesn't get in the way. It's lightweight, it handles the kind of request volume microservices tend to generate, and the ecosystem around it — Express, gRPC libraries, message broker clients — is mature enough that you're not building plumbing from scratch. Whether you're pulling a monolith apart piece by piece or starting fresh, the patterns covered here (separate databases, circuit breakers, idempotent consumers, proper observability) are the parts that actually determine whether the system holds up once it's under load, not just when it's running clean on your laptop.

When putting their model into production, every team or organization encounters the same issue. Failures go unnoticed for days at first because there is no monitoring. As teams begin to fix the issues, they identify areas where production results deviate from the training data, create dashboards for every metric, and set alerts for every threshold. This results in engineers being paged at two in the morning for a bug that fixes itself within an hour, and when an important alert arises, it goes unanswered due to alert fatigue, creating a pipeline that silently feeds garbage into the model. When a team learns to disregard 95% of the issues, they are very likely to disregard the remaining 5% that are actually important, and the solution to this isn’t less monitoring. The good solution to this problem is monitoring, which is tiered, routed, and pruned differently from the infrastructure monitoring that most teams already know. The Problem With Applying Old Monitoring Rules To AI Traditionally, application monitoring used to be binary, which is whether the application or service is up or down, latency is high or low, etc. But AI models don’t fail with these signs; they usually degrade over time. For instance, a recommendation model does not show exceptions when the user behavior shifts; it just silently gets worse at what it was supposed to do. A classifier model does not throw an error when its input distribution changes; it just returns answers confidently with increasingly wrong predictions. An AI application does not crash when it hallucinates; instead, it returns a normal HTTP 200 response with incorrect content. This creates two problems: When AI models fail, the reason for failure is invisible to classical infrastructure monitoring, which causes teams to bolt on multiple checks like data quality checks, drift detectors, and output scorers, each introducing a new source of noise. AI models are statistical in behavior and not deterministic, so setting threshold alerts on them leads to them firing constantly, and training teams have to tune the model. As a result, thorough AI monitoring does not make the application safer; beyond a certain point, it only makes things worse. What to Actually Monitor Monitoring issues that no one will ever take action on is often the first step towards alert fatigue. It is useful to consider it in four layers, each with its own owner and mode of failure. Infrastructure and service: Metrics like inference latency, throughput, Graphics Processing Unit (GPU)/Central Processing Unit (CPU) utilization, error rates, and cost per request and token consumption for anything calling a hosted large language model (LLM) API are classic operational metrics and can usually be monitored with the existing Application Performance Monitoring (APM) tools. Data quality: This is another important thing to keep an eye on because it can cause broken feature pipelines, upstream schema changes, input formats being changed without getting noticed, and null-rate spikes. These are usually the worst failures because you can't see them unless you're looking for them, and the model keeps making predictions based on bad data. Model quality: This can be tracked by looking at changes in the Confidence Score or how much the prediction distribution has changed from what was seen during training. This can be used instead of measuring accuracy because it's hard to tell right away how measures like accuracy are calibrating, because to measure accuracy, you would have to compare the predicted result to the actual correct answer, which doesn't always exist at the time of prediction. Generative artificial intelligence/large language model quality: Metrics like hallucination rate, coherence, factual grounding, toxicity, and susceptibility to prompt injection need different types of tooling to identify them because they are not like traditional metrics and would require human-in-the-loop sampling or an LLM as a judge for identifying them. The mistake many teams make is that they apply the same alerting techniques to all four layers, which is the infrastructure one, as that is the traditional way of setting up monitoring for applications, but issues related to data quality and model quality require a trend-based review. How to Alert Without the Noise Replace static thresholds with adaptive baselines. When systems learn a baseline from historical behavior and trigger alerts on deviations from it, like “alert if latency exceeds 200ms,” this ignores the daily and weekly traffic patterns, and the same is valid for data volume and null rates, which leads to a large number of false alarms being raised. So, teams that have made this switch from static thresholds to adaptive baselines have reportedly reduced noisy alerts by 60–90%. Introduce real severity tiers. When an alert is critical and poses an instant business risk, it is sent to an on-call engineer so that the problem can be fixed right away. Warnings about poor performance that are not critical are sent to a Teams chat channel during business hours, and signals about long-term trends land on the dashboard to be looked at from time to time. This helps to make sure that the notification's urgency matches its real urgency. Correlate and deduplicate before notifying. One change to the schema upstream can cause a dozen problems downstream. Sending a dozen alerts for one root cause either makes the team too busy or forces them to mentally group alerts together, which your tools should be doing for you. Route alerts to whoever can act on them. Misrouting is a common cause of tiredness. If the central platform team doesn't know about the business, they might ignore a spike they can't understand, and the domain team that would be able to understand it would never see the alert. Both problems are solved by linking alerts to the right person by domain, based on where the problem starts. Prioritize by business impact. A system that looks for unusual events handles all alerts the same way because it doesn't know which parts of your system are important to the business. When you think about how important each problem is before choosing how loud to alert, you get a lot fewer alerts overall, and a lot more of them are ones that you should actually act on. Conclusion It's important to understand that all of the ideas we've talked about work together; none of them can be used on their own. For example, adaptive thresholds only give out fewer alerts that aren't differentiated by severity. Without proper routing, severity tiers send the wrong messages about how important something is to the incorrect individuals. To avoid alert fatigue, teams need to take comprehensive actions, which include proper alert designs and organizational practices. They should also ensure that every alert can be acted on, which is better than monitoring everything, because AI monitoring only scales, and not having anyone see a model fail could have serious consequences. Good monitoring means building a system that sends alerts only when it matters, so when it does, people actually act on it.

There’s always more to our contributors than what you see in their author profiles. For our latest Member Spotlight, I sat down with Shamsher Khan to learn more about his newest project. What started as a frustrating Kubernetes troubleshooting problem has since grown into published research, a new way of thinking about operational evidence, and ongoing open-source work. What first got you interested in digging into complex infrastructure and systems problems? "I’ve always been interested in problems where the visible symptom is not necessarily the real cause. In infrastructure, especially distributed systems, a service can look healthy from one angle while something important is already failing underneath. Troubleshooting becomes less about finding one bad log line and more about understanding how the application, container, node, network, scheduler, and platform interacted over time. That is what made Kubernetes particularly interesting to me. It automates a lot of recovery, which is great operationally, but that also means the system can change very quickly while you are still trying to understand what happened. Over time, I found myself increasingly interested not just in fixing incidents, but in understanding what information engineers actually have available during and after those incidents, what disappears, and where existing tooling helps or still leaves gaps. That curiosity has shaped a lot of my writing and open-source work." Your DZone article, “When Kubernetes Forgets: The 90-Second Evidence Gap,” ended up becoming the starting point for Operational Memory Architecture (OMA). What were you seeing in Kubernetes that made you think, “There’s a bigger problem here”? It came from a very specific frustration during incidents. A pod would crash, Kubernetes would restart it, and by the time I got there to investigate, some of the information I wanted was already gone or had changed. One example is LastTerminationState. Kubernetes keeps information about a container’s most recent termination, but when that container fails again, the previous termination context is replaced. In a fast crash loop, that can happen repeatedly in a short period of time. You can arrive at a pod that has restarted thousands of times and still have only a very small window into how that sequence began. What made me think the problem was bigger was realizing that this was not really a Kubernetes bug. Kubernetes is primarily designed to maintain desired state and restore workloads. Preserving a complete forensic history is a different concern. Once I started looking more systematically, I saw similar boundaries elsewhere. Kubernetes Events have limited retention, short-lived workloads can exist entirely between monitoring samples, and some node- or runtime-level evidence can become difficult or impossible to reconstruct after the underlying state changes. There are already strong observability tools that help with logs, metrics, traces, and events, so the question was not, “Why doesn’t Kubernetes keep everything forever?” That would not be realistic or necessarily desirable. The question became more specific: are there predictable points after which certain diagnostic evidence can no longer be recovered, and can we reason about those points explicitly? I started calling those points evidence horizons. OMA grew from trying to characterize those horizons and explore what evidence may need to be captured before they are crossed." Now that the research is being published in IEEE Access, what do you hope people working with these systems take away from it? And where would you like to see OMA go from here? "The main thing I hope people take away is that recovery and diagnosis are related, but they are not the same problem. A platform can successfully restore an application while still losing some of the context that would have helped explain why it failed. I think many engineers have experienced this without necessarily having a name for it. If you have ever finished an incident review with, “We’re not completely sure what actually triggered this,” disappearing or short-lived evidence may be one reason. I also want to be careful not to suggest that OMA replaces existing observability platforms. Tools for logs, metrics, traces, events, and distributed tracing are already essential. OMA is better thought of as a way of reasoning about when different kinds of evidence remain available and when they may cross a point where recovery becomes difficult or impossible. There is also a practical side to this. Teams doing post-incident reviews, reliability analysis, or audit and compliance work may need to reconstruct what happened after the system has already recovered. Thinking explicitly about evidence retention and recovery boundaries can help teams decide what information is worth preserving. As for where OMA goes next, the research is still early. The work evolved in stages: I first published the foundational OMA idea on arXiv, then extended it with a broader evidence-horizon taxonomy and additional validation before developing it into the peer-reviewed IEEE Access paper. The implementation and experiments are public, and the most useful next step is independent validation in environments different from the ones I tested. There are also limitations in the current work. For example, some node-level evidence across kubelet or node restart boundaries requires deeper integration than the current architecture provides. I documented that rather than trying to claim the problem was solved. Some of these ideas have also influenced practical work I’m doing in OpsCart, an open-source Kubernetes operational triage project. OpsCart is not a replacement for OMA or for established observability tools. I use it more as an engineering testbed for exploring how incident context, workload history, and diagnostic evidence can be surfaced in a way that is useful during everyday Kubernetes troubleshooting. I would like to see other engineers test both the research assumptions and the practical tooling, challenge the model, and point out where it does not hold up. That kind of feedback is more valuable at this stage than claiming the architecture is complete." Research: https://ieeexplore.ieee.org/document/11656328OMA implementation: https://github.com/opscart/k8s-causal-memoryOpsCart: https://github.com/opscart/opscart-k8s-watcher After spending so much time thinking about Kubernetes, what’s your ideal way to completely unplug for a weekend? The first requirement is definitely no Kubernetes dashboards. I spend a lot of time during the week thinking about systems, debugging, writing, and experimenting, so on weekends I like doing almost the opposite: spending time with family, getting outside, going somewhere for the day, or just having time where I’m not trying to solve a technical problem. Infrastructure problems have a way of staying in your head even after you close the laptop, so sometimes the best reset is doing something that has absolutely nothing to do with technology. To see more of Shamsher's content, here's the link to his DZone profile.

In this blog, you will take a closer look at the different exchange types that can be used in RabbitMQ. All are demonstrated by means of examples in a Spring Boot application. Enjoy! Introduction In the previous blog, you learned the basic concepts of RabbitMQ and how to use it in a Spring Boot application. However, you only scratched the surface of it, so now it is time to dig a bit deeper into the different exchange types. If you are not yet familiar with the basic concepts, it is advised to read the previous blog. The official RabbitMQ documentation also provides detailed information that is worth reading. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose;Basic knowledge of RabbitMQ. Topics The code can be found in the topics module. In the previous blog, you created two consumers A and B. Consumer A was bound to Queue A with routing key event.general.*. Consumer B was bound to Queue B with routing keys event.general.* and event.specific.*. The asterisk (*) wildcard was used and is a substitute for exactly one word. In the examples, the routing keys event.general.message and event.specific.message were used. You can also use the hash (#) wildcard, and this is a substitute for zero or more words. This is visualized in the figure below. In the RabbitMqConfig, you declare queue C and bind it to the TopicExchange with routing key event.general.#. Java public static final String QUEUE_CONSUMER_C = "consumer-c.queue"; public static final String ROUTING_KEY_NESTED_GENERAL_MESSAGE = "event.general.#"; @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } @Bean public Queue queueConsumerC() { return new Queue(QUEUE_CONSUMER_C, false); } @Bean Binding bindingConsumerCNestedGeneral(Queue queueConsumerC, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerC).to(exchange).with(ROUTING_KEY_NESTED_GENERAL_MESSAGE); } In the MessageController, you create an endpoint for sending a message with routing key event.general.message.nested. This routing key will not match the bindings of consumers A and B. Java @RequestMapping( method = RequestMethod.POST, value = "send-nested-general" ) public ResponseEntity<Void> sendNestedGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message.nested", message); return new ResponseEntity<>(HttpStatus.CREATED); } The ReceiverC listens to messages received in queue C and prints a message. Java @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_C) public void receiveMessage(String message) { System.out.println("Queue Consumer C received <" + message + ">"); } } Start the application from within the topics module. Shell mvn spring-boot:run First, post a general message; this should be received by all consumers. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" In the application console log, you notice that all consumers receive the message. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Queue Consumer C received <This is a general message> Now, post a nested general message, which should be received only by consumer C. Shell curl -X POST http://localhost:8080/send-nested-general \ -H "Content-Type: text/plain" \ -d "This is a nested general message" In the application console log, you notice that the message is only received by consumer C. Plain Text Queue Consumer C received <This is a nested general message> Work Queues The code can be found in the work module. With work queues, you can publish a message and dispatch it to a pool of consumers. One of the consumers will pick up the message and start processing it. This is especially useful for dispatching long-running tasks. You use the default direct exchange in this case, and the queue name is used as the routing key. No need to use a custom exchange. This is visualized in the figure below. The RabbitMqConfig is quite small; you only define the queue. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_TASK = "task.queue"; @Bean public Queue queueTask() { return new Queue(QUEUE_TASK, false); } } When sending a message via an endpoint, you use the queue name as the routing key. Java @RequestMapping( method = RequestMethod.POST, value = "send-work" ) public ResponseEntity<Void> sendWorkMessage(@RequestBody String message) { messageService.sendMessage(RabbitMqConfig.QUEUE_TASK, message); return new ResponseEntity<>(HttpStatus.CREATED); } Every consumer listens to the queue. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer A <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer B <" + message + ">"); } } @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer C <" + message + ">"); } } Start the application from within the work module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-work \ -H "Content-Type: text/plain" \ -d "This is a work message" The message is processed by one consumer. Plain Text Task picked up by Consumer A <This is a work message> Fanout The code can be found in the fanout module. With fanout, you want to broadcast messages to all queues. You send messages to the exchange, but there is no need to specify a routing key. You can also ensure that temporary queues are used. When temporary queues are used, the queue name will be generated. In the RabbitMqConfig, you define a FanoutExchange. The queues are defined as an AnonymousQueue. This creates a non-durable, exclusive, auto-delete queue with a generated name. You bind the queues to the exchange. Java @Configuration public class RabbitMqConfig { public static final String FANOUT_EXCHANGE_NAME = "fanout.exchange"; @Bean FanoutExchange fanoutExchange() { return new FanoutExchange(FANOUT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new AnonymousQueue(); } @Bean Binding bindingConsumerA(Queue queueConsumerA, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange); } @Bean public Queue queueConsumerB() { return new AnonymousQueue(); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } } In order to send messages, you only need to send them to the exchange. This can be seen in the MessageService. Java public void sendMessage(String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.FANOUT_EXCHANGE_NAME, "", message); } On the receiving side, you listen to the generated queue name (thus not a specific one in this case). Java @Component public class ReceiverA { @RabbitListener(queues = "#{queueConsumerA.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = "#{queueConsumerB.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Start the application from within the fanout module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-to-all \ -H "Content-Type: text/plain" \ -d "This is a fanout message" In the application console log, you notice that the message is consumed by all queues. Plain Text Queue Consumer B received <This is a fanout message> Queue Consumer A received <This is a fanout message> RPC The code can be found in the RPC module. Remote Procedure Call (RPC) can be used when you need to execute a function on a remote application and wait for the result. The event is sent to the queue and is processed by Consumer A. The result is sent to a queue in the replyTo field of the request. The publisher waits for data to be returned on this callback queue. When the message appears, it checks the correlationId. If it matches the value of the request, the response is returned to the publisher. All of this is done automatically by the RabbitTemplate. In the RabbitMqConfig, a DirectExchange is used. With a DirectExchange, you match exactly on events; you cannot use wildcards here, just like a TopicExchange. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String DIRECT_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_RPC_MESSAGE = "event.rpc"; @Bean DirectExchange eventsExchange() { return new DirectExchange(DIRECT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, DirectExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_RPC_MESSAGE); } } The MessageController contains an endpoint for sending the event. Java @RequestMapping( method = RequestMethod.POST, value = "send-rpc" ) public ResponseEntity<Void> sendRpcMessage(@RequestBody String message) { messageService.sendMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you use convertSendAndReceive and process the response. Java public void sendMessage(String message) { Object response = rabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } } In the receiver, you receive the message and send a response. Do note that some additional processing is added in order to trigger a timeout. More on that in a moment. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public String receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); if (message.equals("This is an rpc message")) { return "success"; } else if (message.equals("This is a timeout message")) { try { Thread.sleep(10000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "success"; } else { return "failure"; } } } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is an rpc message" In the application console log, you notice that the message is consumed by consumer A, and that a successful response is received by the publisher. Plain Text Queue Consumer A received <This is an rpc message> Sender received response: success But what if it takes too long to process the message? In real life, the remote application can be unreachable for one reason or another. Send a timeout message. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the MessageService, the response will return null, and a timeout exception is raised. Plain Text Queue Consumer A received <This is a timeout message> No response received 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] o.s.amqp.rabbit.core.RabbitTemplate : Reply received after timeout for 2 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed. org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted 2026-04-25T14:50:16.790+02:00 ERROR 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] .l.DirectReplyToMessageListenerContainer : Failed to invoke listener org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted How to solve this? In this case, you are better off using the AsyncRabbitTemplate. This template is not automatically autowired, so you have to define it as a bean. Let's do so in the RabbitMqConfig. Java @Bean public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate) { return new AsyncRabbitTemplate(rabbitTemplate); } In the MessageController, you define an endpoint to trigger the async template. Java @RequestMapping( method = RequestMethod.POST, value = "send-async" ) public ResponseEntity<Void> sendAsyncMessage(@RequestBody String message) { messageService.sendAsyncMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you autowire the AsyncRabbitTemplate. And because it is an async call, you catch the response by means of a CompletableFuture. Java public void sendAsyncMessage(String message) { CompletableFuture<Object> future = asyncRabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); future.thenAccept(response -> { if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } }); } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-async \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the application log, you see the same result: the response is null, but no timeout exception anymore. Conclusion In this post, you learned different exchange types. Each serves its own use case. It is up to you to choose the right pattern for your use case.

By Gunter Rotsaert CORE

Most Docker content targets web developers shipping stateless services. However, data engineers, who represent a huge and growing population of Dockers users, are mostly left to figure things out alone, and it shows. The get pipelines that pass locally, but explode on clusters. They pit notebook-only development against expensive cloud workspaces, and more. This article applies six years of production data platform experience in financial services and healthcare to a question nobody answers well: How to you make a laptop behave like a lakehouse? A Familiar Routine If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set. Web developers solved "works on my machine" a decade ago with containers. Data engineers, somehow, are still developing against shared cloud workspaces, paying per-minute cluster costs to debug a GROUP BY, and discovering environment drift in production. This article is the workflow I wish someone had handed me years ago: a fully containerized lakehouse development environment — Spark, Delta Lake, object storage, a catalog, and orchestration — that runs on a laptop, mirrors production closely enough to trust, and plugs into CI without mocks. The Real Problem: Data Pipelines Have Four Environments, Not One A typical stateless web service has one environment to reproduce: the app runtime. A data pipeline has at least four, and they drift independently: The compute runtime — Spark version, Scala version, JVM, Python, native libs (Arrow, Parquet, libhdfs).The table format layer — Delta Lake / Iceberg versions and protocol versions, which are not the same thing.The storage layer — S3/ADLS semantics: multipart uploads, eventual consistency quirks, path-style vs virtual-hosted access.The orchestration layer — the scheduler's Python environment, which is famously not your job's environment. Mocking any one of these in tests means you aren't testing the thing that breaks. The goal of containerizing a lakehouse is to pin all four layers in code and version them together. Step 1: A Reproducible Spark Image You Actually Control Don't develop against latest. Build a base image that pins every layer of the compute runtime and treat it like an artifact: Dockerfile # syntax=docker/dockerfile:1.7 FROM eclipse-temurin:17-jre-jammy AS base ARG SPARK_VERSION=3.5.4 ARG DELTA_VERSION=3.3.0 ARG HADOOP_AWS_VERSION=3.3.6 RUN apt-get update && apt-get install -y --no-install-recommends \ python3.11 python3-pip tini && \ rm -rf /var/lib/apt/lists/* # Pin Spark itself, not just PySpark RUN curl -fsSL https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \ | tar -xz -C /opt && mv /opt/spark-${SPARK_VERSION}-bin-hadoop3 /opt/spark ENV SPARK_HOME=/opt/spark PATH=$PATH:/opt/spark/bin PYTHONHASHSEED=0 TZ=UTC # Delta + S3 connectors resolved at build time, never at job submit time RUN /opt/spark/bin/spark-shell --packages \ io.delta:delta-spark_2.12:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION} \ -e "println(\"deps cached\")" && \ cp /root/.ivy2/jars/*.jar /opt/spark/jars/ COPY requirements.lock /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.lock # Never run Spark as root RUN useradd -m -u 1001 spark USER 1001 ENTRYPOINT ["/usr/bin/tini", "--"] Three details that matter more than they look: --packages at build time, not submit time. Resolving connector JARs at spark-submit is the #1 source of "it worked yesterday" failures — Maven Central is a runtime dependency you didn't mean to have.PYTHONHASHSEED=0 and TZ=UTC kill two classes of "non-deterministic only in prod" bugs.A lockfile, not requirements.txt. Compile with pip-compile or uv pip compile so transitive dependencies (looking at you, pandas/pyarrow) can't drift. Step 2: The Lakehouse-In-A-Box With Docker Compose Here's the part most teams never build: the rest of the lakehouse, locally. MinIO stands in for S3 (it speaks the same API), and a real Spark master/worker pair stands in for the cluster, because local[*] mode hides every serialization and shuffle bug you'll meet in production. Dockerfile # compose.yaml services: spark-master: build: . command: /opt/spark/sbin/start-master.sh environment: [SPARK_NO_DAEMONIZE=true] ports: ["7077:7077", "8080:8080"] spark-worker: build: . command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077 environment: - SPARK_NO_DAEMONIZE=true - SPARK_WORKER_MEMORY=4g - SPARK_WORKER_CORES=2 depends_on: [spark-master] deploy: replicas: 2 # >1 worker = real shuffles, real serialization minio: image: minio/minio:RELEASE.2025-09-07T16-13-09Z command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: localdev MINIO_ROOT_PASSWORD: localdev-secret ports: ["9000:9000", "9001:9001"] volumes: [lake-data:/data] healthcheck: test: ["CMD", "mc", "ready", "local"] interval: 5s mc-init: # create the bronze/silver/gold buckets on boot image: minio/mc:latest depends_on: { minio: { condition: service_healthy } } entrypoint: > /bin/sh -c "mc alias set local http://minio:9000 localdev localdev-secret && mc mb -p local/lakehouse/bronze local/lakehouse/silver local/lakehouse/gold" volumes: lake-data: Point Spark at MinIO with three config lines and your medallion pipeline reads and writes s3a://lakehouse/... paths exactly like production: Python spark = (SparkSession.builder .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") .config("spark.hadoop.fs.s3a.path.style.access", "true") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate()) docker compose up and you have bronze → silver → gold on your laptop. Total cloud cost of a debugging session: $0. Step 3: Integration Tests That Run Real Spark — Testcontainers The payoff of all this is CI you can trust. With Testcontainers, your pipeline tests spin up the same images your developers use: Python import pytest from testcontainers.minio import MinioContainer from pyspark.sql import SparkSession @pytest.fixture(scope="session") def lake(request): with MinioContainer("minio/minio:RELEASE.2025-09-07T16-13-09Z") as minio: yield minio def test_silver_dedup_keeps_latest_record(lake, spark): # write duplicate customer events to bronze bronze_path = f"s3a://test/bronze/customers" write_fixture_events(spark, bronze_path, duplicates=True) run_silver_dedup(spark, bronze_path, "s3a://test/silver/customers") result = spark.read.format("delta").load("s3a://test/silver/customers") assert result.count() == EXPECTED_UNIQUE assert latest_record_wins(result) No mocked DataFrames. No unittest.mock.patch("boto3..."). The test exercises Delta's actual transaction log against actual object storage. When this suite is green, deployments stop being scary. A pattern I use in regulated environments: keep a fixtures/ directory of small, synthetic Parquet files that mirror production schemas (never production data), and version them with the code. Schema drift then fails a unit test instead of a 2 a.m. pipeline run. Step 4: One Image From Laptop → CI → Production The final principle: the image you test is the artifact you ship. Multi-stage builds let one Dockerfile serve dev (with Jupyter, debuggers) and prod (minimal, non-root): Dockerfile FROM base AS dev USER root RUN pip install --no-cache-dir jupyterlab pytest debugpy USER 1001 FROM base AS prod COPY --chown=1001:1001 src/ /app/src/ COPY --chown=1001:1001 jobs/ /app/jobs/ # nothing else — no notebooks, no test deps, no shell tools you don't need In CI: build once, tag with the git SHA, run the Testcontainers suite against prod, scan it (Docker Scout, or your registry's scanner), sign it, and promote that exact digest through staging to the scheduler. Whether the scheduler is Airflow's DockerOperator/KubernetesPodExecutor or a managed Spark platform pulling custom containers, the principle holds: environments are immutable, versioned, and identical by construction. Lessons Learned From Production Run ≥2 workers locally.local[*] mode never serializes between JVMs. The day you switch to a real cluster, every closure-capture and UDF-pickling bug appears at once. Two 2-core workers in Compose surfaces them on day one.Pin the table format protocol, not just the library. Delta and Iceberg both evolve table protocol versions. A newer writer can produce tables an older reader can't open. Encode the protocol version in your image build args and test reads with the oldest reader you support.MinIO is a stand-in, not a clone. It won't reproduce S3 request throttling or cross-region latency. Keep a small smoke-test suite that runs against real object storage nightly; do everything else locally.Resource-limit your local Spark. Without SPARK_WORKER_MEMORY caps, a skewed join will cheerfully eat your laptop. Limits also force you to think about partitioning early — which is the point.Treat the orchestrator's image as layer four. Airflow DAG-parse environments drift too. Containerize the scheduler with the same lockfile discipline as the jobs. Production Considerations Before you take this pattern to a real platform team, three things to plan for: secrets (local Compose uses throwaway creds; production should inject via your cloud's secret manager or Docker secrets — never baked into images), image provenance (sign images and generate SBOMs in CI; regulated industries will ask, and in 2026 the tooling is mature enough that "we didn't get to it" no longer flies), and base image hygiene (start from minimal, hardened bases and rebuild on a schedule, not just on code change — CVEs don't wait for your sprint). Conclusion Containers gave application developers reproducibility ten years ago. Data engineering is finally having the same moment — and the teams that containerize their lakehouse development loop ship faster, test honestly, and stop paying cloud bills to find typos. Try it: clone the Compose stack above, point your gnarliest pipeline at it, and see what breaks locally that used to break in prod. Then tell me about it — I'd genuinely like to hear which layer drifted on you. If this was useful, follow me here and on LinkedIn. Next up in this series: load-testing Delta merge performance locally, and contract testing between pipeline stages.

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

A live production integration case study. Introduction and Purpose of This Article This article is written for mid- and high-level managerial and technical decision-makers. I am the author of the open-source Java library MgntUtils. The article presents an analysis of a real integration of the stack trace-filtering feature from that library into a live commercial production environment. A few important clarifications up front: This is not a side-project pilot and not a lab demo. The feature was integrated into a production service of a company that serves a high volume of real customers. Due to legal constraints, I am not at liberty to name the company.This is not a how-to article for implementers. If you came looking for code samples or logging-framework wiring, please see the dedicated articles listed in the Disclaimer below.MgntUtils can be used in Java projects and in other JVM-based languages such as Kotlin. Before diving into the production numbers, it is worth stating briefly what the feature does and why those numbers matter. Server-side stack traces are usually full of framework and infrastructure noise — proxies, filter chains, containers, thread pools, and similar boilerplate — while the few lines that actually explain the failure are easy to lose in the pile. The MgntUtils filtering utility keeps the application frames and the exception / Caused by chain, and collapses that noise. The result is a much shorter stack trace without losing the information you actually need. When those stack traces are later consumed — sent to an LLM for analysis, or opened by an engineer — that reduction can mean: Substantial AI token savingsTypically more accurate AI root-cause answers, because the model has less framework noise to latch onto and hallucinate aboutA meaningful productivity boost for human triage The rest of this article focuses on what was observed after integrating this feature in production: the measured benefits, how to interpret them, and the integration experience itself — including gotchas that only surfaced in a real live environment, as opposed to a pilot project. Disclaimer This article deliberately does not discuss the technical design of stack trace filtering or the technical details of the integration. Each of those topics has its own dedicated article: Filtering Java Stack Traces With MgntUtils Library DZone: https://dzone.com/articles/filter-java-stacktrace-mgntutilsDEV Community: https://dev.to/mgantman/java-stacktrace-filtering-utility-1c1i Zero-Code-Change Stack Trace Filtering for Spring Boot: An Infrastructure-Level Integration DEV Community: https://dev.to/mgantman/zero-code-change-stacktrace-filtering-for-spring-boot-an-infrastructure-level-integration-3fk5 Production Results and Benefits Below are the observations and conclusions from monitoring the live production system after the feature integration. The feature had been running for about a month, and filtering was also temporarily turned off for comparison. What the Production Environment Looked Like Anonymized sketch of the deployment (enough to judge fit, without identifying the company): High-traffic JVM/Spring Boot service in a commercial production estateStructured JSON logging to a major observability platformObservability billing dominated by per-event (not per-byte) pricingIn a typical production day, that service emitted on the order of ~70,000+ log events carrying a stack trace That is a large stream of stack trace payloads — expensive if fed to an LLM, and tiring if engineers open them by hand. Stack Trace Volume Reduction Range in Production Filtering was measured across production stack traces with filtering on vs off. Observed size/token reductions typically fell in roughly the ~75%–95% range: Toward the high end (~90–95%): framework-heavy request-handling traces (long security/container/proxy tails)Toward the lower end (~75%+): more application-dense traces, where a larger share of frames is your own code The average reduction on a typical trace in this environment was about ~91%. The table below is a real before/after example — shown so you can see what that looks like in practice: MetricUnfilteredFilteredReductionLines19518~91%Bytes~22,200~1,900~91%Input tokens (approx.)~6,300~540~91%Application framesall (buried in noise)all (kept)no signal lost Every application frame in the business call path was retained; what disappeared was framework and infrastructure noise (proxies, filter chains, container/thread-pool frames, and similar boilerplate). Stack traces tokenize poorly for LLMs — package separators, generated class names, and (File:line) markers all split into extra tokens — so the token reduction tracks the size reduction closely. Root-cause readability was unchanged. In both versions, the failure was identifiable from the application frames and the exception message. Filtering did not remove diagnostic signal; it removed the large majority of the payload that never helped. What Improved AI analysis: cheaper and more accurate (when exceptions are analyzed). For every exception sent to an LLM, the stack trace input payload shrank by roughly ~75–95% depending on the trace shape (~5,800 tokens saved on a typical ~91% trace). That saving repeats for every analyzed event. In an environment where tens of thousands of stack traces are emitted per day, any AI triage, clustering, or “explain this error” pipeline pays that tax over and over unless the noise is stripped first. Cost is only half of the AI benefit. Filtering also improves answer quality. The removed frames are framework and infrastructure boilerplate — identical across many errors and unrelated to the application failure. When those frames remain in the prompt, models often latch onto them and hallucinate a root cause in the noise. With them collapsed, the model is steered toward the application frames and exception message that actually explain the failure — so analysis is not only cheaper, but typically more accurate. Sensitivity calculator (illustrative — not this company’s AI spend). If your org analyzes exceptions with an LLM, you can size token cost roughly as: Plain Text annual token saving ≈ (exceptions analyzed per year) × (tokens saved per exception) × (model input price per token) Using ~5,800 tokens saved per exception (average on a typical ~91% trace) and an illustrative model input price of $3 per 1 million input tokens: Analyzed exceptions / dayApprox. tokens saved / dayApprox. saving / year5,000~29M~$32K50,000~290M~$318K250,000~1.45B~$1.6M Plug in your own analysis volume, your place in the ~75–95% reduction range, and your model pricing. The production measurement that is firm is the observed per-exception reduction range, with application frames preserved. Secondary AI upside: More errors per context window. Because a typical filtered stack trace is so much smaller (~540 tokens vs ~6,300 in the example above), many more distinct exceptions fit into a single model call. That is a capability change, not just a cost saving: cross-error analysis — clustering failures, or asking “what went wrong in the last N hours?” — becomes practical instead of blowing the context window on framework noise. It is secondary to the per-exception token and accuracy benefits, but it matters for any AI workflow that looks at more than one error at a time. Human triage productivity. Engineers reading a filtered typical trace see the full application call path at the top (~18 lines in the example above) instead of scrolling through ~195 lines to confirm there is no hidden nested cause and to piece the business path together. For on-call and incident review, that is a direct readability win. What Changed in Log Volume — and What Did Not It helps to separate event count from bytes per event. Event count did not change. A stack trace is still one log event whether it is 195 lines or 18. If your observability vendor bills per event (or per indexed log line item), filtering does not reduce that charge. In this production environment, that was the dominant billing model — so there were no savings on a per-event bill. Bytes per stack trace event did change. Each filtered stack trace was roughly ~75–95% smaller than its unfiltered counterpart (commonly ~90% for framework-heavy traces). There is a real reduction in stack trace payload size. How much that shows up in total log volume is not deterministic. Overall space / ingested-byte savings depend on what share of all logs are stack traces: Plain Text overall byte reduction ≈ (stacktrace share of total log volume) × (~75–95% reduction on those stacktraces) In this company’s environment, stack traces were only about ~1% of total log volume — which is unusually low (an anomaly for many systems, but what we observed here). Cutting ~90% of that 1% yields only a fraction of a percent of total logs, which is easy to lose inside normal day-to-day traffic variance. That is why aggregate ingested-byte charts did not show a clear step when filtering was toggled. In another organization where stack traces are a much larger share of log volume, the same per-trace cut would produce a more visible space saving. Those savings are real in principle, but variable by workload and not the main point of this case study. The main point here is consumption cost. The firm, repeatable benefit we are highlighting is what happens when a stack trace is analyzed by an LLM or read by an engineer: large payload reduction, same diagnostic signal. Treat log-space savings as a possible secondary effect, sized by your own stack trace-to-total-logs ratio — not as the success criterion for this feature. How to Read These Results as a Decision Maker QuestionAnswer from this production caseDid filtering remove useful diagnostic information?No — application frames and exception chain structure remained.How large is the per-exception reduction?Roughly ~75–95% across production traces (often ~90%+ on framework-heavy request traces).Does that reduce per-event log billing?No — event count is unchanged.Is there space / byte saving?Yes per stack trace (~75–95%); overall only if stack traces are a meaningful share of total logs (here ~1%, so barely visible).Where is the upside for AI analysis?Far fewer tokens and less hallucination on framework noise — cheaper and typically more accurate.AI context-window upside?More exceptions fit in a single context window — useful for clustering or “what failed in the last N hours?” analysis.Other upside?Time saved when humans read errors.Who should adopt it?Teams that already (or soon will) send production exceptions to LLMs at volume, and/or teams whose engineers routinely open noisy stack traces. The production evidence supports a clear, bounded claim: when stack traces are consumed, filtering delivers a large, repeatable reduction in payload size with no loss of application signal. Per-event log bills do not drop. Overall log-space savings may exist but depend on stack traces’ share of total volume — and are not the primary reason to adopt the feature. Integration Experience I started from an implementation I already had in the MgntUtilsUsage side-project repository — a runnable Spring Boot demo of MgntUtils features, meant to emulate real-life apps as closely as possible. It was a very good starting point. Still, as I worked through the live commercial integration, a few gotchas surfaced that a single-JVM demo simply does not force you to confront. Gotchas That Showed Up in a Real Production Environment 1. Feature Toggle Storage Across Multiple Containers My demo app runs in a single JVM. A real production service typically runs on several containers that scale in and out. In the demo, the on/off flag for stack trace filtering lived in memory — which is fine for one process, and useless once you have more than one. In a multi-container environment, you need an external, shared flag holder that every instance can read. Redis (or an equivalent shared store available to all containers) is a good candidate. 2. JSON Logging Adapters, Not Only the Classic Logback Pattern When I first modified the Logback configuration, my demo mainly used conventional Logback pattern-based adapters. A real production app will most likely also use a JSON encoder for external logging systems such as Datadog (and similar platforms). That special adapter has its own throwable-handling path, so wiring the filter there is a must — otherwise you can end up with filtered console output locally and unfiltered stack traces in the system that actually matters. 3. Hardening the Fail-Safe Path A fall-back option already existed for the case where anything goes wrong inside the filtering path. For production, that fail-safe had to be hardened a bit further to make it as bullet-proof as possible: if filtering ever fails, the system must still emit a full standard stack trace and must never drop the log event. 4. Logback Is Not the Only Popular Logging Framework This company uses Logback, so that is what the production integration targeted. But Logback is not the only widely used option — my own favorite, for example, is Log4J. For the dedicated integration article (linked in the Disclaimer), I also had to provide Log4J instructions, even though Log4J was not used in this particular environment. Anyone planning an org-wide rollout should assume more than one logging stack may need to be covered. Effort, Timeline, and Outcome All in all, the integration was smooth, and the side-project was close enough to the final result in the real app. About 4–5 hours to get an integrated version up and running in the staging environmentAbout one day of observing staging to make sure there were no unexpected behaviorsThen deployment to production, with about another day of close monitoring before declaring the feature live So roughly half a day of integration work, and about 1.5 working days of testing / staging observation / production monitoring. Not a single bug was found. There are two contributing factors for that: The stack trace-filtering feature itself is mature and battle-tested — I am tempted to say it has no bugs, but let’s just say it is highly stable and reliable.The integration itself is simple enough. The next integration should be even faster, since this one is now well documented (including the dedicated Spring Boot integration article linked in the Disclaimer). If you are interested in integrating this feature into your project, the detailed integration instructions are in the article Zero-Code-Change Stack Trace Filtering for Spring Boot: An Infrastructure-Level Integration. If you are interested in support for the integration, feel free to contact me at or through my LinkedIn profile. Conclusion This case study supports a simple decision: Adopt stack trace filtering if your organization already analyzes production exceptions with LLMs at a meaningful volume, or if engineers routinely open noisy stack traces during triage and on-call. In those cases, the live evidence is clear: typically about ~75–95% less stack trace payload (around ~91% on a typical trace), with application frames preserved — cheaper AI analysis, typically more accurate answers, and easier human reading. Do not adopt it expecting your per-event observability bill to drop, or expecting a large automatic cut in total log volume. Event count does not change. Overall byte savings depend on how large a share stack traces are of all logs — and that varies by organization. Consumption cost is the main point; log-space savings are secondary and workload-dependent. On effort and risk: in this live commercial integration, getting to staging took about half a day of work, followed by roughly a day and a half of staging observation and production monitoring. No bugs were found. The feature is mature, the integration is simple, and the demo-to-production gaps (shared toggle, JSON logging adapters, fail-safe hardening, and covering more than one logging framework) are now documented. If that profile matches your environment — high exception volume that is actually consumed by AI or by people — this is one of the cheaper, lower-risk improvements available. If exceptions are mostly logged and rarely looked at, the benefit will be thin, and that is an honest reason to pass.