Containers

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

The first time I containerized a fine-tuned Llama model for a client's internal search tool, the build finished at 38 gigabytes. I remember staring at the terminal thinking there was no way that was right. It was right. The image included a CUDA base, PyTorch with every backend compiled in, model weights baked directly into the layer, and a pip cache that had not been cleaned. Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all. Why This Problem Actually Matters Most Docker advice out there is written for stateless web services, small images, fast cold starts, and horizontal scaling on demand. LLM workloads break almost every assumption baked into that advice. The artifact is huge, the runtime is GPU-bound, startup involves loading gigabytes into VRAM, and half your "application code" is actually a C++/CUDA binary blob you didn't write and can't easily trim. If you treat an inference container like a Flask app with a bigger base image, you end up with slow deploys, wasted GPU spend, and autoscaling that technically works but arrives too late to matter. The First Wrong Turn: One Image to Rule Them All Our early approach was a single monolithic image model with weights, tokenizer, inference server, and dependencies all baked together, rebuilt on every model version bump. It felt simple. It wasn't. Every retrain meant rebuilding a 30+ GB image even when the code hadn't changed a single line. Registry storage costs gradually increased until someone in finance questioned why our container registry bill resembled that of a second AWS account. Worse, rollbacks were painful because reverting to a previous model meant pulling an entire previous image rather than swapping a much smaller artifact. The solution that actually worked was separating the model weights from the serving image entirely. The image contains the runtime, the inference server (we used vLLM for most of our transformer workloads), and pinned dependencies. Weights live in object storage and are pulled at container start via an init container or a lazy loading entry point. The approach felt counterintuitive at first. Are we effectively transitioning the slower process to startup instead of build time? — but it turned out to be the right trade. Startup pulls are parallelizable, cacheable on the node, and don't bloat the registry. Build time dropped from twenty-plus minutes to under four. A Smaller Base Image Than You'd Expect This is where the challenges began. Everyone defaults to using nvidia/cuda:*-devel images because the framework documentation recommends them, but these devel images include the entire CUDA toolkit, which contains compilers that you will never use at runtime. Switching to the runtime variant and only installing the exact CUDA and cuDNN versions your framework's wheel actually needs cuts roughly 4GB off the base alone. A minimal multi-stage build looks something like this: Dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder RUN pip install --no-cache-dir vllm==0.4.2 FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 COPY --from=builder /usr/local/bin/python3.10 /usr/local/bin/ ENV MODEL_PATH=/mnt/models ENTRYPOINT ["python3", "-m", "vllm.entrypoints.api_server"] The build stage compiles anything that needs the full toolkit; the runtime stage only carries what's needed to execute. It's a basic Docker pattern, but I've seen it skipped constantly on ML teams because the assumption is always, "the model is the heavy part; the image doesn't matter." The model is heavy, sure, but a bloated base image adds real minutes to every autoscale event, and in production that's the difference between absorbing a traffic spike and dropping requests. The OOM Kill: Nobody Explained Well This is the war story I bring up most often. We had a container that ran fine locally and in staging, then got silently killed in production under load — no crash log, no stack trace, just a pod restart and a confused on-call engineer at 2 AM. It turned out to be the kernel OOM killer, not an application-level exception, because our memory limit accounted for the model weights in VRAM but excluded the growing KV cache for long-context requests plus the CPU-side tokenizer buffers. GPU memory and container memory limits are two completely separate accounting systems, and Kubernetes will happily kill your pod over host RAM even if your GPU has headroom to spare. The fix was unglamorous: we set explicit memory requests and limits with a real margin above peak KV cache usage, moved batch size and max sequence length into environment-configurable values instead of hardcoding them, and added a lightweight health assessment that reported GPU memory utilization alongside the standard liveness probe. None of that is exotic. All of it was missing because we'd copy-pasted a manifest template built for a stateless API and never revisited the resource math for a model that holds state in memory for the duration of a request. Where I'd Push Back on Common Advice A lot of guidance recommends one model per container for isolation, and for many teams that's right. But if you're serving several small fine-tunes of the same base model, that pattern wastes GPU memory by duplicating base weights across containers. We transitioned to a multi-adapter setup, where one base model is loaded once, and LoRA adapters are swapped for each request; this approach is more complex operationally but reduces the GPU footprint by nearly half. I wouldn't consider it a default; it represents a level of complexity that is justified only after demonstrating that plain per-model containers are indeed the bottleneck. I'd also push back on containerizing every workload the same way. Batch inference and real-time serving have almost opposite goals: one wants throughput and tolerates slow cold starts; the other needs rapid readiness and predictable latency. We split these into separate images with separate resource profiles, even though it meant more Dockerfiles. Fewer surprises beat fewer files. Key Takeaways Separate model weights from the serving image; bake them in the runtime and pull weights at startup from object storage.Use CUDA runtime images, not devel images, unless you genuinely compile something at container start.Account for GPU memory and host memory as two separate budgets; KV cache growth is the usual silent killer.Split batch and real-time serving into different images; their optimization goals are conflicting.Don't reach for multi-adapter serving or other density tricks until you've measured that plain per-model containers are actually the bottleneck. Closing Thought None of this required exotic tooling, no custom orchestrator, and no proprietary platform. It required treating the container as part of the model's runtime behavior rather than a packaging afterthought bolted on after the research work was done. The teams that struggle most with this approach usually aren't lacking Docker knowledge; they're applying web-service intuition to a workload that behaves nothing like a web service. If you're mid-migration on something similar, I'd genuinely ask: are you optimizing your image for build convenience or for what actually happens the moment traffic hits a cold node? Those answers are rarely the same, and figuring out which one you've been solving for is usually the first real fix.

If different Docker Engine versions are running simultaneously in a Docker Swarm cluster, this may lead not to an obvious service outage but to a more subtle scenario: partial traffic degradation on individual nodes. In this case, the issue appeared on one of the manager nodes, Traefik started reporting an unavailable status (health=0) for the router-app service, and the cause, according to the working hypothesis, was related to differences in iptables rules and overlay networking between Docker 28.1.1 and 28.2.2. On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time. Context At the time of the incident, the Docker Swarm cluster consisted of 5 manager nodes, several dozen worker nodes, and approximately 40–50 services. External HTTP traffic passed through Traefik, deployed on each of the five manager nodes, and was then routed by Traefik to the backend application containers. One of the key services was router-app, responsible for building public transportation routes on the frontend. It was one of the critical entry-point services with the highest SLA, and any disruption to its availability could have led to severe penalties from the customer, so any deviation in its availability required an immediate response. Grafana dashboards showed application availability through Traefik health-check statuses. Those statuses were generated based on HTTP health-check endpoints implemented by the developers for most services, primarily the most critical ones. This was enough to quickly localize the problem at the ingress traffic level. The cluster had one important characteristic. Some nodes were running Ubuntu 20.04 (focal), while others were running Ubuntu 22.04 (jammy), and different APT repositories were pulling different Docker Engine versions. As a result, after another scheduled Docker update on the nodes, the production environment ended up with a mix of nodes running 28.1.1 and 28.2.2 at the same time. The docker node ls screenshot additionally confirmed that mixed versions were present not only on worker nodes but also on manager nodes, including swarm4 and swarm5. How the Incident Manifested The incident was detected not through user complaints and not through a general service outage, but through Traefik monitoring. The triggered alerts showed partial unavailability of one of the router-app containers, after which the health-check dashboard confirmed that the issue affected not the entire service but one of the manager nodes. This is important because the red blocks on the dashboard did not indicate a complete outage of router-app. It meant that Traefik on one of the manager nodes started receiving health=0 when checking that service’s backend endpoint, while the other manager nodes continued to see the backend as healthy. In practice, it looked like this: traffic through one of the manager nodes stopped reaching the router-app containers correctly, but Traefik, running on all five manager nodes, automatically excluded requests to the unhealthy entry point. As a result, from the outside the incident appeared as partial degradation rather than full unavailability. That is exactly what made the situation tricky. Fault tolerance limited the impact of the incident, but the underlying cause remained inside the cluster and continued to affect one of the entry points. What Docker Showed After localizing the issue to one of the manager nodes, it became clear that the cause should be sought not in router-app itself but in the network path between Traefik and the backend containers. At the same time, docker service ps did not show a widespread service failure, and the containers still appeared as running. The next useful signal came from the dockerd logs on swarm5. Repeated messages appeared there, including Peer delete operation failed, neighbor entry not found, and errors related to deleting FDB and neighbor entries for the VXLAN interface vx-001001-5lk08. For example: Plain Text Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.366807202Z" level=warning msg="Peer delete operation failed" error="could not delete fdb entry for nid:5lk08r7jjvtq5idqggzeygmlv eid:4e7d63d00fffaa6be7ce6362f47acd7912f7c11e5ac6e018393722decc16c210 into the sandbox:neighbor entry not found for IP 10.170.0.37, mac 02:42:0a:1b:14:2c, link vx-001001-5lk08" Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.765092537Z" level=warning msg="error deleting neighbor entry" error="no such file or directory" ifc=vx-001001-5lk08 ip=10.170.0.138 mac="02:42:0a:1b:14:65" Such messages were highly consistent with problems in Docker Swarm’s overlay network. In essence, Docker was trying to delete network records that were no longer present in the tables, which usually points to desynchronization of network state at the VXLAN, FDB, or neighbor-table level. By themselves, these messages still did not provide a complete explanation, but they pushed the investigation in the right direction. It became clear that the problem was not in the application’s business logic but in the network layer on one of the nodes. Additionally, docker node inspect self --pretty on swarm5 showed that from Swarm’s point of view the node looked normal: State: Ready, Availability: Active, Raft Status: Reachable, Leader: No, while Engine Version was already 28.2.2. This was an important point: the control plane still considered the node healthy, even though at the traffic-flow and network-state level it was already behaving differently. Diagnostics The investigation was carried out at the node level. The tools used included docker node ls, docker version, apt-cache policy docker-ce, as well as ip link show, bridge fdb show, ip neigh show, and comparisons of iptables chains across different nodes. The key fact became visible after docker node ls. The cluster was not homogeneous in terms of Docker Engine version: some manager nodes and some worker nodes were already running 28.2.2, while the others remained on 28.1.1. This led to the assumption that the issue might be at the iptables rules level. After that, iptables had to be compared separately on healthy and problematic nodes. On swarm5, a full rules dump was collected using the combination of iptables -S, iptables -t nat -S, and iptables -t mangle -S. Those rules showed the DOCKER, DOCKER-FORWARD, DOCKER-INGRESS, and DOCKER-USER chains, as well as ACCEPT, DROP, and DNAT rules for traffic through docker_gwbridge, published ports, and ingress routing.... To test the hypothesis, not only the problematic swarm5 but also the first manager node, swarm1, was compared, where a stable stack with Docker 28.1.1 had long been running. On swarm1, the output of iptables -S and iptables -t nat -S showed the expected picture: the DOCKER and DOCKER-INGRESS chains contained a full set of ACCEPT and DNAT rules for all published ports (80, 8080–8082, and dozens of internal service ports) with symmetric dport/sport pairs, while DOCKER-USER effectively boiled down to a clean RETURN. Taken together with the dump from swarm5, this reinforced the conclusion that on nodes running 28.1.1, the iptables configuration for ingress and routing was consistent, and the differences seen on 28.2.2 were related not to manual changes but to the behavior of Docker Engine itself. After that, iptables had to be compared separately on other nodes running different Docker versions. On nodes with 28.1.1, the DOCKER and DOCKER-USER chains and the associated rules were in the expected state, whereas on nodes with 28.2.2 some of the required rules were missing or the chains were reduced to a minimal RETURN. This explained the observed behavior well. The services remained running, Swarm did not appear broken, but external traffic and part of the overlay routing through a specific manager node were working incorrectly, causing Traefik on that node to report health=0 for router-app. It is worth noting separately that journalctl -u docker and docker service ps did not provide a simple direct cause for the incident. They did not show a picture of a general failure, so the conclusion had to be assembled from several sources: Traefik monitoring, dockerd logs, Docker versions, and the state of iptables on different nodes. Fix Once the main hypothesis had narrowed down to mismatched Docker Engine versions, the solution was fairly straightforward: return the cluster to a homogeneous configuration by rolling back to version 28.1.1 as the fastest solution. The rollback was performed for swarm4, swarm5, and all worker nodes where version 28.2.2 had already been installed. To do this, a specific package version was pinned via apt, then Docker was restarted, and the installed version was verified. One version of the commands looked like this: Shell apt-cache madison docker-ce | grep 28.1.1 apt-get install docker-ce=5:28.1.1-1~ubuntu.22.04~jammy \ docker-ce-cli=5:28.1.1-1~ubuntu.22.04~jammy \ containerd.io && systemctl restart docker && docker --version Additionally, it made sense to check the package sources and, if necessary, remove conflicting APT entries so that the nodes would no longer receive an unsuitable Docker version from another repository. In practice, it looked like this: Shell sudo rm /etc/apt/sources.list.d/download_docker_com_linux_ubuntu.list sudo apt update After the rollback, docker version was checked again, as well as the DOCKER and DOCKER-USER chains. After Docker Engine had been unified to 28.1.1 on both manager and worker nodes, the issue disappeared. From the perspective of external behavior, this was confirmed immediately. Health checks in Traefik returned to the green zone, and the partial unavailability of router-app on one of the manager nodes could no longer be reproduced. Root Cause Based on the available data, the most well-founded working version is this: in this environment, Docker Engine 28.2.2 formed or applied iptables rules related to DOCKER, DOCKER-USER, FORWARD, ingress, and overlay networking differently. In a mixed cluster, this led to one of the manager nodes no longer forwarding traffic correctly to the router-app backend containers, even though from the perspective of the control plane and service state this did not look like a direct failure. It is important here not to overstate what the data allows. This case does not prove a universal upstream bug in Docker 28.2.2 for all Swarm installations, but it does show that even closely related Docker Engine versions can affect the cluster’s network plane differently, especially when different Ubuntu distributions and different package sources are present in production at the same time. What Follows From This The first conclusion is simple: Docker Swarm is sensitive to Docker Engine version mismatches. If some manager or worker nodes have been updated while others have not, this can lead not only to version drift as an organizational problem, but also to practical issues with traffic, published ports, and overlay routing. The second conclusion is that after updating Docker, it is necessary to check not only docker version but also the node’s network behavior. The minimum set includes iptables -L DOCKER -v -n, iptables -L DOCKER-USER -v -n, checking published ports, ingress/overlay state, and health checks from the edge proxy. The third conclusion is that it is useful to maintain a single baseline stack across all nodes. One Ubuntu LTS distribution, unified repositories, and the same update order reduce the chance that cluster state will remain formally healthy while part of the network traffic is already being handled incorrectly. The fourth conclusion concerns update order. In our case, that was exactly how it happened, but it is worth noting separately. When Docker is updated in a Swarm cluster, it is better to update worker nodes first, then manager nodes, and the leader last, while after each stage separately checking the node’s behavior under real traffic conditions (service availability, correct routing, and published ports). In our case, enhanced monitoring was in place, so no additional manual checks of node behavior in traffic were required: if any part of the infrastructure became unavailable, we would promptly receive an alert. The final conclusion relates to monitoring. In this case, Traefik not only helped limit the impact of the incident by routing around the unhealthy node, but also provided the first precise signal that the problem was localized to a specific entry point rather than existing at the level of the entire service or the entire cluster.

Introduction The meeting that changed the platform team's direction was not a technical one. It was a conversation with a product engineer who had been at the company for eight months and had never successfully deployed to production without help from someone on the platform team. Not because she lacked skill. She was smart, experienced, and had successfully launched production systems at two previous jobs, but getting a working service into production meant dealing with fifteen different configuration files across four repositories, figuring out how Helm values files and Kustomize overlays worked together, and knowing which of the three CI pipeline templates to use based on whether the service needed a sidecar, a job scheduler, or neither. She had read the documentation. It was accurate. It just didn't tell her what to do when the documented path didn't match the state of her specific service in her environment. The platform team had built powerful infrastructure. They had not built a usable platform. That distinction between infrastructure and platform is where most Kubernetes-based internal developer platform efforts go wrong, and it's worth being precise about what it means. Infrastructure vs Platform: A Practical Distinction Infrastructure is the machinery: the Kubernetes clusters, the networking layer, the CI pipelines, the secrets management system, and the monitoring stack. A platform is the interface that makes that machinery accessible to developers who aren't Kubernetes experts without requiring them to become ones. The confusion between the two produces a situation that's extremely common in engineering organizations: a technically sophisticated infrastructure that's effectively only usable by the people who built it. The test for whether you have a platform or just infrastructure is simple: can a developer who joined three months ago deploy a new service to production without asking anyone for help? Not by following a tutorial someone wrote last year that may or may not still be accurate, but through tooling that guides them through a current, correct process. If the answer is no, you have infrastructure. The platform is the missing layer. This statement is not an argument against complexity in the underlying system. Kubernetes is complex, and that complexity exists for beneficial reasons: flexibility, programmability, and a rich ecosystem. The platform layer should absorb the complexity, rather than exposing it to every developer who needs to ship a service. What the First Attempt Got Wrong The infrastructure team built the first version of the internal platform in their spare time, juggling it with other priorities. It consisted of a set of Helm chart templates, a GitHub Actions workflow library, and a wiki with deployment instructions. This approach is how most internal platforms start, and it has a predictable failure mode: the templates encode the assumptions of the people who wrote them, the wiki goes stale within weeks, and the gap between the documented process and the actual state of the infrastructure grows invisibly until it becomes a significant tax on every developer who hits it. The fundamental mistake was treating platform work as documentation work rather than product work. A wiki is not a platform. A set of templates that require understanding to use correctly is not a platform. A platform is software that makes the correct path the easy path, that validates inputs before they cause problems downstream, and that fails loudly and helpfully rather than silently and mysteriously. The second attempt started from a different premise: the platform is a product, developers are its users, and the measure of success is whether they can do their jobs without needing the platform team. The Abstraction Layer: Custom Resources and Admission Webhooks The technical decision that made the most difference was introducing a custom resource definition (CRD) that represented a service in the platform's domain model, not a Kubernetes Deployment or Service, but a higher-level construct that encoded the platform's opinionated defaults and generated the underlying Kubernetes objects from a much simpler specification. YAML # Platform-level CRD: what developers actually write apiVersion: platform.company.com/v1 kind: AppService metadata: name: payment-api namespace: production spec: image: payment-api:v1.4.2 tier: backend # drives resource limits, network policy replicas: 3 port: 8080 healthCheck: /healthz env: DATABASE_URL: secretRef: payment-db-credentials This twelve-line manifest replaced the hundred-plus lines of Kubernetes YAML that developers had previously been required to write and maintain. The controller running in the cluster, a standard Kubernetes operator built with controller-runtime, read the AppService resource and generated the Deployment, Service, HorizontalPodAutoscaler, PodDisruptionBudget, and NetworkPolicy that the platform's standards required, with defaults applied consistently across every service. The key design decision was what to expose in the CRD and what to hide. The tier field is a prime example: rather than exposing resource requests and limits directly, which requires understanding what values are appropriate for the cluster, the CRD accepts a tier label (frontend, backend, worker, batch) that maps to a predefined resource profile. A backend tier service receives a specific CPU and memory allocation appropriate for the cluster's node types. A batch tier service receives a different profile with different eviction priorities. The developer specifies intent; the platform enforces the appropriate configuration. Go # Controller logic: tier maps to resource profile (Go pseudocode) func resourceProfileForTier(tier string) corev1.ResourceRequirements { profiles := map[string]corev1.ResourceRequirements{ "frontend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("128Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("500m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, }, "backend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("250m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("1000m"), corev1.ResourceMemory: resource.MustParse("512Mi"), }, }, // batch, worker profiles follow same pattern } return profiles[tier] } Admission webhooks complemented the CRD by catching misconfiguration before it reached the cluster. A validating webhook checked every AppService manifest against a set of rules: the image tag must not be 'latest,' the health verification path must respond within the cluster, and secret references must exist in the target namespace and return a clear error message describing exactly what was wrong and how to resolve it. This shifted error detection from 'runtime, after deployment' to 'submission time, before anything breaks,' which dramatically reduced the debugging load on both developers and the platform team. The Golden Path and Its Limits The CRD and controller approach works well when services fit the platform's model. Here's where things became challenging: not all services fit the platform's model. A service that needed custom init containers, a service that required a specific affinity rule because of a hardware dependency, and a batch job with a complex retry policy that didn't map cleanly to the tier abstraction. Each of these required either extending the CRD or breaking the abstraction and falling back to raw Kubernetes YAML. The temptation is to keep extending the CRD to cover every case. Resist it. A CRD that tries to expose every Kubernetes feature is just a more complicated way to write Kubernetes YAML, and it loses the simplicity that made the abstraction valuable. The better model is a golden path, the CRD for the 80% of services that fit the standard model, and a documented escape hatch for the 20% that don't. The escape hatch is raw Kubernetes resources, maintained by the teams that need them, with the platform team providing support rather than ownership. The key is being honest with developers about which path they're on. A service using the AppService CRD gets platform-managed defaults, automatic updates when the platform evolves, and first-class support. A service using raw Kubernetes resources owns its own configuration and gets best-effort support. That distinction in the support model is what makes the trade-off legible rather than arbitrary. What I'd Do Differently In hindsight, the most important investment was the admission webhook, and I'd build it earlier. The CRD and controller took significant time to design and implement. The webhook could have been built in a few days and would have immediately improved the developer experience by catching misconfiguration at submission time rather than deployment time. Validation before generation is higher-leverage than generation that might produce something invalid. I'd also measure platform adoption from day one. Which teams are using the AppService CRD? Which teams are on raw Kubernetes? What's the conversion rate of new services to the platform abstraction? Without that data, the platform team relies on intuition instead of evidence to guide their investment. The teams that adopt slowly are often the ones with the most valuable feedback about where the abstraction doesn't fit, and they're also the teams most likely to be quietly maintaining fragile custom configurations that will become incidents later. When should you not build a CRD-based platform abstraction? If you have fewer than fifteen to twenty engineers deploying services, the overhead of designing, building, and maintaining a CRD-based platform abstraction almost certainly exceeds the value. Helm charts and excellent templates get you most of the way there with a fraction of the complexity. The operator pattern earns its cost when you have enough services that inconsistency becomes a real operational problem when the differences between how services are configured start causing incidents and nobody can tell you why a particular service is configured the way it is. Key Takeaways Infrastructure and platform are different things. Infrastructure is the machinery; a platform is the interface that makes machinery accessible without requiring expertise in its internals. Most Kubernetes-based IDPs stop at infrastructure. Custom resource definitions let you define a domain model that encodes your platform's opinions. Developers specify intent (tier, replicas, port); the controller generates the correct Kubernetes objects with consistent defaults applied. Admission webhooks shift error detection from runtime to submission time. A clear error message at kubectl apply is worth more than a mysterious pod crash two minutes later. Maintain a golden path for the majority of services and a documented escape hatch for the rest. A CRD that tries to cover every Kubernetes feature loses the simplicity that justified building it. Conclusion The platform engineer's job is to make complexity disappear, not by eliminating it, but by absorbing it into tooling so that the people building products don't have to carry it. That's a harder problem than building the infrastructure itself, and it requires a fundamentally different mindset: less systems engineering, more product thinking. Who are the users? What tasks do they need to accomplish? Where does the current experience fail them? The teams building internal developer platforms who get the process right tend to look, from the outside, like they have unusually productive engineering organizations. Individual contributions ship faster, incidents caused by misconfiguration drop, and the platform team spends less time on support and more time on improvements. The causal chain runs directly from platform quality to engineering output, even though it's usually measured differently. The open question is whether the CRD-based abstraction model scales to genuinely heterogeneous service fleets, the kinds of organizations where services span multiple languages, multiple deployment patterns, and multiple infrastructure dependencies. The golden path works when most services look similar enough that a shared abstraction is useful. What occurs to the platform model when 40% of services utilize the escape hatch? At that point, is the abstraction still earning its cost, or is it adding complexity without delivering the simplicity it promised?

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen. Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel. In this blueprint, we will build a secure multi-tenant execution environment. We will containerize the agent runtime using Docker, provision a GKE cluster with Axion nodes, isolate the network, and orchestrate the execution layer using a robust Java backend. Step 1: Containerizing the Agent Runtime The first step is establishing a baseline execution environment. We want this Docker image to be as lightweight as possible to reduce the attack surface, while containing the necessary runtimes for the LLM to execute its logic. Dockerfile # Use a minimal Alpine base image to reduce attack surface FROM python:3.11-alpine # Create a non-root user for execution RUN addgroup -S agentgroup && adduser -S agentuser -G agentgroup WORKDIR /sandbox # Copy the execution wrapper script COPY --chown=agentuser:agentgroup execute_payload.py /sandbox/ # Enforce non-root execution USER agentuser # Prevent Python from writing pyc files and buffering stdout ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 CMD ["python", "execute_payload.py"] To make this functional, we need an entrypoint script that safely reads the LLM-generated code from an injected environment variable or a mounted volume, executes it, and captures the output. Here is a simplified execute_payload.py implementation: Python import os import sys import traceback def main(): # In a production environment, this payload might be injected via # a Kubernetes Secret or a secure sidecar proxy. encoded_payload = os.environ.get("AGENT_PAYLOAD", "") if not encoded_payload: print("Error: No payload provided.") sys.exit(1) try: # Execute the untrusted code within this isolated process # Security constraints are handled by the container and gVisor layers exec(encoded_payload, {"__builtins__": __builtins__}, {}) except Exception as e: print(f"Execution Error: {str(e)}") traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() Even if a malicious script breaks out of the Python runtime, it will find itself as an unprivileged user inside a minimal Alpine container. Step 2: Provisioning GKE With Axion and Agent Sandbox Google Axion (N4A) processors provide excellent performance per watt, making them ideal for running hundreds of concurrent, lightweight agent tasks. We will create a cluster and explicitly enable the sandbox feature. Shell # Create the GKE cluster with Sandbox enabled gcloud container clusters create agent-sandbox-cluster \ --region us-east4 \ --enable-sandbox \ --sandbox type=gvisor \ --release-channel regular # Create a dedicated node pool using Axion N4A instances gcloud container node-pools create axion-agent-pool \ --cluster agent-sandbox-cluster \ --region us-east4 \ --machine-type n4a-standard-4 \ --num-nodes 3 \ --node-labels dedicated=untrusted-agents \ --tags untrusted-workload Applying node labels ensures that trusted core microservices do not accidentally end up on the same physical infrastructure as untrusted agent execution environments. Step 3: Enforcing Network Isolation Compute isolation is useless if the untrusted code can scan your internal network or exfiltrate data to the public internet. We must deploy a strict NetworkPolicy to default-deny all egress traffic from our sandboxed namespace. YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-agent-egress namespace: isolated-agents spec: podSelector: matchLabels: app: agent-executor policyTypes: - Egress egress: # Only allow DNS resolution - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP # Allow outbound only to a specific internal API gateway if needed # - to: # - ipBlock: # cidr: 10.0.0.50/32 Step 4: Deploying the Sandboxed Workload With the network secured, we define the Kubernetes deployment. By setting the runtimeClassName to gvisor, Kubernetes routes the container lifecycle through the GKE Agent Sandbox rather than the standard container runtime. YAML apiVersion: apps/v1 kind: Pod metadata: generateName: dynamic-agent-task- namespace: isolated-agents labels: app: agent-executor spec: # Instruct GKE to use the Agent Sandbox (gVisor) runtimeClassName: gvisor # Ensure these pods only land on our Axion node pool nodeSelector: dedicated: untrusted-agents restartPolicy: Never containers: - name: execution-environment image: your-registry/agent-runtime:v1.0.0 env: - name: AGENT_PAYLOAD valueFrom: secretKeyRef: name: task-payload-secret key: payload # Drop all unnecessary Linux capabilities securityContext: runAsUser: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" volumeMounts: - name: temp-storage mountPath: /tmp volumes: - name: temp-storage emptyDir: {} Step 5: Orchestrating the Execution via Java Spring Boot To bring this architecture together, the control plane must dynamically spin up these sandboxed pods whenever an AI agent decides it needs to run code. In a modern distributed system, this is typically handled by a core backend microservice. Using the Fabric8 Kubernetes Client in a Java Spring Boot application provides a highly resilient way to orchestrate these ephemeral workloads programmatically. Java import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.client.KubernetesClient; import org.springframework.stereotype.Service; @Service public class AgentOrchestratorService { private final KubernetesClient kubernetesClient; public AgentOrchestratorService(KubernetesClient kubernetesClient) { this.kubernetesClient = kubernetesClient; } public String executeUntrustedCode(String tenantId, String pythonCode) { // 1. Create a Kubernetes Secret containing the code payload String secretName = createPayloadSecret(tenantId, pythonCode); // 2. Load the sandbox Pod template and inject the specific payload secret Pod sandboxedPod = kubernetesClient.pods() .inNamespace("isolated-agents") .load(getClass().getResourceAsStream("/k8s/agent-pod-template.yaml")) .item(); // 3. Launch the pod dynamically via the API server Pod runningPod = kubernetesClient.pods() .inNamespace("isolated-agents") .create(sandboxedPod); // 4. Await completion and extract the logs safely kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .waitUntilCondition(pod -> pod.getStatus().getPhase().equals("Succeeded") || pod.getStatus().getPhase().equals("Failed"), 30, java.util.concurrent.TimeUnit.SECONDS); String executionLogs = kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .getLog(); // 5. Clean up the ephemeral resources kubernetesClient.pods().delete(runningPod); kubernetesClient.secrets().withName(secretName).delete(); return executionLogs; } } The Defense in Depth Strategy This architecture relies on a strict defense in depth model. If an LLM hallucinates a malicious payload or a user deliberately attempts prompt injection to compromise the platform, the attacker faces multiple independent barriers. The code executes as a non-root user in a minimal Alpine environment with a read-only filesystem. Network access is completely blocked by native Kubernetes policies. Finally, any attempt to exploit kernel vulnerabilities is intercepted by the gvisor runtime boundary running on dedicated Axion hardware. By combining these layers, engineering teams can build and scale trustworthy Agentic AI platforms without risking the integrity of their core cloud infrastructure.

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we'd agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas. What it didn't account for is that each replica needed almost three minutes just to pull a 14GB checkpoint and warm up CUDA kernels before it could answer a single request. The load balancer, seeing new pods report as running, immediately started routing traffic to them. For three minutes, a chunk of our users got 504s while perfectly healthy-looking pods sat there loading a model into memory. Nobody on the infra side had touched Docker that day. The incident wasn't a Docker bug. We assumed that container orchestration designed for web services would function the same way for processes that take minutes to become useful, rather than those that operate in milliseconds. Why LLM Containers Break the Usual Assumptions Packaging an LLM serving stack in Docker still makes sense for the same reason it always has; CUDA versions, driver compatibility, and Python ABI mismatches are miserable to manage across a fleet without a frozen artifact. But an LLM container carries baggage that a typical inference service doesn't. The weights are tens of gigabytes, not a few hundred megabytes. GPU memory is a single shared pool that one greedy container can quietly exhaust for everyone else on the box. And “ready” doesn't mean “process started”; it means the model is resident in VRAM and the CUDA graph is warmed, which can take minutes on a cold node pulling weights from object storage over the network. The Mistakes, in Order Our first version baked the model weights directly into the image, because it felt simpler: one artifact, one pull, done. In practice, it meant a 16GB image, painfully slow CI pushes, and a registry bill nobody wanted to look at. Worse, every time we bumped into a new fine-tuned checkpoint, we rebuilt and repushed the entire layer regardless of caching, because the COPY step touching gigabytes of weight files invalidates everything below it. Unlike a typical ML inference image, there's no meaningful caching win here at all; the layer is simply too big to ever be a cache hit across versions. We moved weights out to a mounted volume, fetched at container start from object storage, and never looked back. Second mistake, and this one actually cost us a production incident: we ran the container with Docker's default shared memory size. vLLM, which we used for serving, spins up worker processes that talk to each other over shared memory even on a single GPU. With the default 64MB /dev/shm, those workers would crash with cryptic bus errors under any real concurrency. The fix was almost embarrassingly small: Shell docker run --gpus all \ --shm-size=2g \ -e MODEL=mistralai/Mistral-7B-Instruct-v0.2 \ -e GPU_MEMORY_UTILIZATION=0.85 \ -e MAX_MODEL_LEN=8192 \ -p 8000:8000 \ llm-serve:latest The third mistake was more subtle and took longer to diagnose. vLLM's continuous batching reserves a large slice of GPU memory upfront for the KV cache, controlled by gpu_memory_utilization. We'd set that fraction high to maximize throughput, then bin-packed two replicas onto the same GPU to save cost. Under normal traffic, fine. During a burst of unusually long-context requests, such as someone summarizing a 6,000-word document instead of a tweet, the KV cache for that single batch ballooned, causing the container to run out of memory (OOM) mid-generation and taking down every other in-flight request in the same batch. This failure mode is more severe than a typical web service OOM because it not only drops the new request but also terminates queries that were already halfway through generating answers for paying customers. What We Actually Changed The readiness adjustment turned out to matter more than any Docker flag. We split liveness from readiness: liveness just checks that the process hasn't died; readiness fires a real, tiny generation request through the local API and only flips to healthy once that round trip succeeds. That alone killed the cold-start routing problem because the load balancer stopped trusting a merely alive process. We also gave up on bin-packing two replicas per GPU. In hindsight, treating GPU memory like it's as elastic as CPU or RAM was the actual root cause, not any single Docker setting. We implemented a model that uses one GPU, sets a conservative memory utilization ceiling, and enforces a request-level token limit at the proxy in front of the container, rather than inside it, because it is too late to make adjustments once the batch is already running. On the orchestration side, we stopped trying to scale-to-zero or scale aggressively off CPU-style metrics. Scale-to-zero is effective for web apps but doesn’t fit GPU-bound LLM serving, where cold starts can outlast traffic spikes. We kept a warm floor of replicas sized to baseline traffic and let a request queue absorb bursts instead of expecting new pods to materialize in time. It's less elegant than the autoscaling story everyone likes to tell, and it costs more in idle GPU time, but it's honest about what the hardware can actually do. What We Rejected, and Why We seriously considered dropping self-hosting altogether and routing through a managed inference API. For a side project, that's probably the right call — less to own, no GPU bin-packing headaches. We rejected it due to data residency requirements that prohibited sending raw text to a third party, and at our volume, managed pricing would quickly exceed our GPU costs. We also looked at Ray Serve and Triton early on, and they solve some of the issues more natively, but the team's Docker and Kubernetes muscle memory was strong enough that rebuilding on a new serving framework felt like trading one set of unknowns for another, at least for the first version. Key Takeaways Never bake multi-gigabyte model weights into the image — there's no caching benefit at that size, only slower pushes and bigger registry bills.Set shared memory explicitly; vLLM and similar multiprocess servers will fail under load with Docker's tiny default.Treat GPU memory utilization conservatively and avoid bin-packing replicas onto a single GPU unless you can guarantee a strict ceiling per container.Build a readiness assessment that performs a real generation, not just a process check; cold model loading will otherwise receive routed live traffic.Don't expect autoscaling to save you on cold-start timescales measured in minutes; a warm floor plus a queue is more honest than reactive scaling. Closing Thought None of these issues was really a Docker failure; the container did exactly what we told it to do. The failure was treating a multi-gigabyte, GPU-bound, slow-to-warm process like it was just another stateless web container that happens to need a GPU flag. I suspect that many teams will learn this lesson in the same way we did, during an incident on a Friday afternoon. Is it the right move to keep stretching Docker and Kubernetes to fit LLM serving, or is this the workload that finally pushes most teams toward purpose-built serving layers?

Not long ago, I broke a backtest without changing a single line of code. I moved the script to a different machine—same OS, supposedly the same Python version — and the equity curve suddenly told a completely different story. Nothing in the logic had changed. The environment was the only obvious difference. That was the day I stopped treating the runtime environment as an afterthought and started treating reproducibility as part of the experiment itself. What I learned is this: a backtest isn’t truly reproducible just because it ran once on your laptop. A result you can’t regenerate reliably isn’t a research finding — it’s a coincidence. And when that coincidence eventually meets real money, it can get expensive fast. What follows is a minimal but complete pipeline for containerizing and automatically testing a Python backtesting system. No over-engineering — just Docker, GitHub Actions, and a few habits that make your results trustworthy. The Real Goal We aren’t building a live trading platform. We’re building a workflow that ensures every change is verified in a controlled environment: Plain Text Code change → automated tests → versioned Docker image build The system must do four things: Produce deterministic results, within an acceptable numerical tolerance, from the same versioned inputs in the same containerized environment.Automatically test every code change before it can be merged.Halt the pipeline when a test fails, with no exceptions.Tag every image build so we can trace it back to the exact source-code version. If a workflow can’t do that, it’s just a script living on someone’s laptop. Project Structure Before containerizing the application, let’s organize the repository clearly: Python backtesting-system/ ├── app/ │ ├── engine.py │ └── main.py ├── tests/ │ └── test_engine.py ├── data/ │ └── sample.csv ├── requirements.txt ├── requirements-dev.txt ├── Dockerfile └── .github/ └── workflows/ └── ci.yml The app/ directory holds the strategy logic, while tests/ remains separate. The data/ directory contains a small, frozen sample dataset that never changes — our gold standard. There are no absolute paths or machine-specific assumptions. Everything that might vary, including the data path, starting capital, and fee rate, comes from environment variables or a configuration file. Containerizing With Docker The classic “works on my machine” problem usually indicates an environment mismatch. Docker reduces this drift by packaging the application and its runtime dependencies into a versioned image. Here is the Dockerfile: Dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app ./app COPY data ./data CMD ["python", "-m", "app.main"] A few decisions matter here. We pin the Python 3.12 image series and can use an image digest when stricter reproducibility is required. Dependencies are pinned to exact versions in requirements.txt, since version ranges can silently introduce changes. We also use slim to keep the image small. Crucially, we never copy local keys, cached results, or temporary files into the image. The container doesn’t guarantee that the logic is correct. It helps ensure that sound logic runs in a controlled and substantially more consistent environment. Adding Tests That Matter Automation without tests simply automates mistakes. Our tests don’t attempt to prove that a strategy is profitable. They prove that the program behaves consistently. We check for: Clear errors when input files are empty or missing.Deterministic results when the same data and seed are used.Correct fee calculations.Rejection of malformed rows rather than silent processing.Required fields in every output. Here is one example, including the necessary imports: Python import pytest from app.engine import run_backtest def test_backtest_is_reproducible(): first = run_backtest("data/sample.csv", seed=42) second = run_backtest("data/sample.csv", seed=42) assert first["trades"] == second["trades"] assert first["final_equity"] == pytest.approx( second["final_equity"], rel=1e-9 ) This test establishes a simple contract: given the same starting conditions, the system will not drift beyond an acceptable margin. The direct comparison of trades works here because we assume that every trade entry uses standardized types such as integers and strings. If the trade records contain floating-point prices, those values should be checked individually with an appropriate tolerance. Building the GitHub Actions Pipeline Now we automate the workflow. It runs on pushes, pull requests, and manual triggers through workflow_dispatch: YAML name: Backtest CI on: push: pull_request: workflow_dispatch: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -r requirements-dev.txt - run: pytest - run: docker build -t backtest:${{ github.sha } . The steps are straightforward: check out the code, set up Python, install the development dependencies, run the tests, and build the Docker image. The requirements-dev.txt file includes both the production dependencies and a pinned version of pytest: Python -r requirements.txt pytest==8.3.4 If pytest detects a failure, the job stops immediately. The broken change never reaches the image-build step. The resulting image is tagged with the Git commit hash, creating a clear link between the source code and the image built during that workflow run. Managing Configuration and Secrets Environment-specific configuration and secrets should never be baked into the image. Environment variables can control data paths and run modes. If the system is later connected to a live data source, API credentials should be stored in GitHub Secrets or a cloud secrets manager—never in the source code or Dockerfile. Logs must not expose keys or sensitive headers. Even if today’s “production” environment is only a scheduled test run, development and production should use separate configuration sets. Treat every secret as sensitive, and keep environment-specific configuration outside the image. Lightweight Monitoring and a Path to Rollback Once the pipeline runs regularly, monitoring must go beyond asking whether the process is still alive. Useful questions include: Did the latest job complete, or did it hang?Has execution time increased dramatically?Is the input data intact?Were the output files generated, and are they non-empty?Which image version produced the latest results? If images are later pushed to a container registry, retaining the last few stable versions provides a straightforward rollback path. For scheduled backtest runs, we should also archive the data snapshot, parameters, and results. That allows us to return a month later and answer a very specific question: “What exactly did we test on July 24?” These practices aren’t unique to systems we build ourselves. Commercial grid-trading interfaces make automated execution accessible without revealing every part of their internal deployment pipelines. BYDFi is one example I encounter in my work. Because I work with the platform, this is a disclosed reference rather than an independent recommendation. The comparison is conceptual: understanding reproducibility, automated checks, and configuration management helps developers evaluate any automated tool more thoughtfully. The Experiment Isn’t Finished Until It’s Verified We started with a broken backtest and a frustrating realization. Now we have a different mindset. Docker reduces environment drift. Automated tests guard program behavior. GitHub Actions ensures every change passes through the same gate. Monitoring and versioning give us a clear path to detect problems and support rollback as the pipeline evolves. In a reliable backtesting system, reproducibility and verification are not tasks that come after the experiment. They are part of the experiment itself. The moment we treat them that way, our results stop being anecdotes and start becoming evidence. And when the decisions involved can carry real financial weight, evidence is the only thing worth building.

Artificial intelligence is rapidly transforming software testing by enabling QA engineers to generate test cases and test plans, automate browser interactions, analyze and debug failures, and execute complex testing workflows using simple natural-language prompts. While cloud-based AI assistants offer impressive capabilities, they often require subscriptions and sharing potentially sensitive application data with third-party services. Running an AI-powered testing assistant locally addresses these concerns by providing better privacy, lower operating costs, and complete control over the testing environment. In this tutorial, we’ll learn how to build our own local AI QA engineer using Docker, Ollama, Qwen3:8b, LibreChat, and Playwright MCP. It will allow us to perform browser automation and interact with web applications using natural language, all without relying on cloud-based AI services. Understanding the Architecture Every interaction begins with the user. For example, a user enters a prompt in LibreChat, such as “Open the Playwright website and click the ‘Get Started’ button.” LibreChat serves as the conversational interface through which users interact with the AI assistant. Rather than processing the request itself, it forwards the prompt to a locally hosted large language model, Qwen3:8b, running via Ollama. After receiving the prompt, Qwen3:8b interprets the user’s intent and generates a step-by-step execution plan. Instead of interacting with the browser directly, the model determines which tools are required and communicates those instructions using the Model Context Protocol (MCP). These MCP requests are handled by the Playwright MCP Server, which acts as the bridge between the language model and the browser. It translates the AI-generated instructions into executable Playwright commands. The Playwright MCP Server then launches a Chrome browser and performs the requested actions. Depending on the prompt, it can navigate to websites, click buttons, complete forms, extract text from web pages, capture screenshots, and execute a wide range of browser automation tasks. Once the browser completes the requested operations, the execution results are returned to Qwen3:8b. The language model analyzes the browser output and transforms the technical details into a clear, human-readable response. LibreChat then presents this response to the user. Instead of displaying raw Playwright logs, it provides a concise summary such as: “Navigation completed successfully. The Playwright website was opened, and the Get Started button was clicked successfully.” This architecture enables browser automation through natural language while ensuring that every component runs locally. As a result, we benefit from enhanced privacy, greater security, and complete control over the entire AI-powered automation workflow. Prerequisites Before getting started, ensure that the following software is installed on your machine: DockerNode.js 20 or higher versionGitOllama We’ll use Docker Desktop to run LibreChat, Node.js to install and run the Playwright MCP Server, Git to clone the required repositories, and Ollama to download and serve the local large language model. Having these tools installed beforehand will make the setup process smooth and straightforward. System Requirements Running a local AI-powered browser automation stack requires a reasonably capable machine. A system with 16 GB of RAM or more is recommended to run Docker containers and the language model efficiently. We’ll also need 20–25 GB of available disk space, preferably on an SSD, to accommodate Docker images and downloaded models. While a dedicated GPU can significantly improve model inference speed, it is entirely optional, and the setup works well on modern CPUs. For this tutorial, I’m using the following configuration: Operating system: macOS (M2 Pro)Memory: 16 GB RAM We can have the same setup on Windows and Linux, with only minor platform-specific differences in the installation steps. Setting Up the Environment for the Local AI QA Engineer Docker, Node.js, and Git are widely used development tools, and detailed installation guides for each are readily available online. Installing Ollama To install Ollama, either download the installer from the official website or use the installation command provided for your operating system. For macOS, it can also be installed using the following Homebrew command: Plain Text brew install ollama Once the installation is complete, it can be verified by running the following command in the terminal: Plain Text ollama --version Installing Qwen3:8b Qwen3:8b is chosen for this setup because it offers a strong balance of reasoning, code generation, and performance, making it ideal for Playwright TypeScript test generation, AI agents, MCP integration, and modern QA automation workflows while running efficiently on a local machine. However, other higher models can also be chosen if you know a better one. Another factor in choosing this model was the available system memory. Since my machine has 16 GB of RAM, some memory also needs to be reserved for other tools used in this setup, such as Docker, LibreChat, and Playwright. We need to start Ollama first by running the following command from the terminal. (It should be kept running in the background): Plain Text ollama serve Open a new terminal and run the following command to pull the Qwen3:8b model: Plain Text ollama pull qwen3:8b It should take some time to complete the pull, as the model is around 5.2GB. Once the download completes, we can check the model by running the command: Plain Text ollama list It should list the model downloaded. Next, we can quickly verify by running the model using the command: Plain Text ollama run qwen3:8b Once the model starts, it will prompt you to enter a query. To verify that everything is working correctly, try a simple prompt such as “What is 2 + 2?”. Observe how the model processes the request and generates its response. If the setup is successful, it should return the correct answer, 4, confirming that the model has been downloaded, installed, and is functioning properly. To stop the model, type “/bye” in the prompt, and it should exit. Qwen3:8b provides a good balance between performance and resource usage, making it a suitable choice for this hardware configuration. If more RAM is available, you can opt for larger LLMs that offer stronger reasoning and coding capabilities. Installing LibreChat With Docker LibreChat is an open-source AI platform that provides a unified and customizable interface for interacting with multiple AI models. It enables us to manage all our AI conversations from a single application while supporting features such as AI agents, Model Context Protocol (MCP) servers, custom tools, and integrations with both local and cloud-based LLMs. LibreChat acts as the front-end chat interface that communicates with the locally running Qwen3:8b model through Ollama. It allows us to execute AI-powered browser automation workflows entirely on our local machine. Follow the steps below to install LibreChat: Step 1: Clone the LibreChat GitHub Repository The repository can be cloned by running the following command: Plain Text git clone https://github.com/danny-avila/LibreChat After cloning the repository, navigate to the LibreChat folder, copy the .env.example file, and create a new .env file from it. Plain Text cd LibreChat cp .env.example .env Let's keep the .env file as it is, using the default values. Step 2: Connect Ollama to LibreChat Ollama can be connected to LibreChat by updating its configuration in the “librechat.yaml” file. The example file is already available in the cloned repo. Run the following command to copy librechat.example.yaml and create librechat.yaml. Plain Text cp librechat.example.yaml librechat.yaml Update the following configuration in the file to connect Ollama to LibreChat: YAML endpoints: custom: - name: "Ollama" apiKey: "ollama" baseURL: "http://host.docker.internal:11434/v1" models: default: - "qwen3:8b" fetch: true titleConvo: true titleModel: "current_model" summarize: false summaryModel: "current_model" modelDisplayLabel: "Ollama" Make sure that this configuration is added to the “custom” block, which falls under the “endpoints” block. This configuration adds Ollama as a custom AI endpoint in LibreChat. The baseURL tells LibreChat where to connect to the Ollama API, while the default model specifies that Qwen3:8b should be used by default. Since LibreChat is running inside a Docker container while Ollama is running directly on the host machine, we use http://host.docker.internal:11434/v1 instead of localhost. The special hostname host.docker.internal allows the Docker container to access services running on the host system, enabling LibreChat to connect to the locally running Qwen3:8b model through Ollama. Setting fetch: true allows LibreChat to automatically detect and display all models available in Ollama. The remaining options configure the user interface by generating conversation titles using the current model, disabling conversation summarization, and displaying the endpoint with the label Ollama in the LibreChat interface. Step 3: Mount the Configuration in the docker-compose-override.yml The docker-compose-override.yml can be copied and created in the same way as we did “librechat.example.yaml”. Plain Text cp docker-compose.override.yml.example docker-compose.override.yml The following block should be updated in the docker-compose.override.yml file. YAML services: api: volumes: - ./librechat.yaml:/app/librechat.yaml This file mounts the custom “librechat.yaml” configuration file into the LibreChat container. By mapping ./librechat.yaml to /app/librechat.yaml, Docker ensures that LibreChat uses the custom configuration each time the container starts. This approach allows us to modify settings, such as custom endpoints and AI models, without rebuilding the Docker image. Step 4: Start the LibreChat Application Using Docker Compose The LibreChat application can be started using the following command: Plain Text docker compose up -d It will take some time for the Docker images to download, and containers will start. Run the following command from the terminal to check the Container status: Plain Text docker ps -a This command displays the status of all Docker containers. If any container is unhealthy or encounters an issue, its status will be clearly indicated in the output. In case any container is unhealthy or encounters an issue, the following command can be run to check its logs: Plain Text docker logs <container name> Once all the containers are started successfully, open a new browser and navigate to http://localhost:3080 to start LibreChat. Since we are accessing LibreChat for the first time, we will be prompted to register and create a new user account. After completing the registration process, we can sign in and start using the application. Step 5: Selecting Ollama > Qwen3:8b Model By default, the gpt-5.5 model is selected. To select the Qwen3:8b model: Click on the gpt-5.5 modelSelect Ollama > Qwen3:8b Once the Qwen3:8b model is selected, we can verify if it is working by sending a simple prompt such as “What is 2+2?” Make sure the command “ollama serve” is already running in the terminal in the background, else the model Qwen3:8b won't work on LibreChat. Once we receive a successful response from the model, we can confirm that the Qwen3:8b model has been configured and integrated successfully with LibreChat. Install Playwright MCP Server Playwright MCP can be installed by running the following command in the terminal: Plain Text npx @playwright/mcp@latest \ --host 0.0.0.0 \ --allowed-hosts "*" \ --port 8931 \ By default, Playwright MCP listens only on localhost, which means applications running inside Docker (like LibreChat) cannot connect to it. Using --host 0.0.0.0 makes the server accessible from Docker containers, while --allowed-hosys "*" allows requests from host.docker.internal instead of restricting access to localhost. Once the Playwright MCP server is started, we can leave it running in the terminal. After the Playwright MCP server starts, it shows the following message at the bottom: “For legacy SSE transport support, you can use the /sse endpoint instead”. We will configure the Playwright MCP server using the SSE (Server-Sent Events) transport. Although Playwright MCP also supports the Streamable HTTP transport, LibreChat currently does not support connecting to it via the /mcp endpoint. Therefore, the SSE transport is used to establish a reliable connection between LibreChat and the Playwright MCP server. Configure Playwright MCP Server in LibreChat Playwright MCP server can be added to LibreChat by updating the following configuration in the “librechat.yaml” file. YAML mcpServers: playwright: type: sse url: http://host.docker.internal:8931/sse timeout: 120000 This configuration registers the Playwright MCP server with LibreChat. The type: sse setting specifies that the connection uses the Server-Sent Events (SSE) transport, while the url points to the Playwright MCP server running on the host machine. The hostname host.docker.internal allows the LibreChat Docker container to communicate with services running outside the container. The timeout: 120000 sets the request timeout to 120 seconds, giving the AI agent sufficient time to complete browser automation tasks before the connection expires. However, the timeout can be extended to 15–20 minutes or more, as there is no harm in doing that. YAML mcpSettings: allowedDomains: - 'host.docker.internal:8931' - 'localhost:8931' The mcpSettings configuration also needs to be added under the ‘actions’ block in the “librechat.yaml” file. The mcpSettings.allowedDomains section defines the list of trusted MCP server endpoints that LibreChat is allowed to connect to. By including both host.docker.internal:8931 and localhost:8931, LibreChat can establish a secure connection to the Playwright MCP server, whether it is accessed from within the Docker container (host.docker.internal) or directly from the host machine (localhost). Any MCP server not included in this list will be blocked, providing an additional layer of security. Restart the LibreChat app so it reads the newly configured Playwright MCP server: Plain Text docker compose restart That, or we can also shut down the already running LibreChat and start it again by using the commands below: 1. To shut down LibreChat: Plain Text docker compose down 2. To start it again: Plain Text docker compose up -d After restarting LibreChat, log in and navigate to the home page, and follow the steps below: Click on the MCP Settings menu on the left-hand menu panel.In the MCP Settings window, click on the “+” button to add MCP. Fill in the details for adding the Playwright MCP server; make sure to add the following settings: MCP server URL: http://host.docker.internal:8931/sseTransport: SSEAuthentication: NoneTick the “I trust this application” checkbox. Click on the “Create” button to save the details. Make sure that the Playwright MCP server is started and running on the terminal as discussed in the earlier section Click Connect for the newly created MCP server to establish the connection and begin using it. If everything is fine, a message should be displayed on successful connection. Understanding Model Context Protocol (MCP) By itself, a large language model (LLM) is limited to generating text. It can answer questions, explain concepts, write code, or summarize information, but it cannot directly interact with external systems or perform real-world actions. Model Context Protocol (MCP) changes this by enabling AI models to communicate with external tools and services through a standardized interface. Instead of simply providing suggestions, an AI model can execute tasks such as interacting with browsers, reading files, querying databases, or creating pull requests. Think of MCP as USB for AI A simple way to understand MCP is by comparing it to the USB standard. Before USB became the universal standard, every hardware manufacturer used its own proprietary connector. Printers, keyboards, cameras, and other peripherals all required different cables and custom software integrations. This made connecting devices unnecessarily complicated. USB solved this problem by introducing a common communication standard. Once both the computer and the device supported USB, they could communicate regardless of the device type. Whether you connected a keyboard, webcam, microphone, or external hard drive, the same protocol handled the communication. MCP brings the same level of standardization to AI systems. Without MCP, every AI application requires building and maintaining custom integrations for every external tool it wants to use. If we switch to a different AI application, those integrations often need to be recreated from scratch, resulting in duplicated effort and increased maintenance. A collection of awesome servers for the Model Context Protocol can be found at mcpservers.org. With MCP, tools expose a common interface that any MCP-compatible AI application can use. The AI model only needs to understand the MCP protocol, while the implementation details are handled by the individual MCP servers. Why MCP Matters for QA Automation For QA Automation Engineers, MCP unlocks the ability to automate complete testing workflows rather than isolated tasks. Consider the following request: “Read the Jira story, generate Playwright tests, execute them, analyze any failures, and create a GitHub pull request.” With MCP, the AI agent can coordinate multiple tools to complete the entire workflow. For example, it can: Read the user story from JiraAccess the application’s source code from GitHubGenerate Playwright TypeScript testsExecute the tests in a real browserCapture screenshots, logs, and execution reportsCommit the generated tests to GitHubUpdate the Jira ticket with the test results Each of these actions may be handled by a different MCP server, such as a Jira MCP server, GitHub MCP server, and Playwright MCP server. From the AI model’s perspective, however, every server is accessed using the same standardized MCP protocol. This standardization is what makes MCP so powerful. Rather than building custom integrations for every tool, AI systems communicate through a single, consistent protocol. As a result, MCP servers for Playwright, GitHub, databases, and many other services can be integrated and used in a uniform, scalable manner, significantly simplifying the development of AI-powered automation workflows. Creating an AI Agent With Playwright MCP Server in LibreChat for Automation Testing Let’s create a new AI Agent for browser automation testing with Playwright MCP using the steps below: Step 1: Click on the Agent Builder menu on the left-hand menu panel. Step 2: Enter the following mandatory details to create a new agent: Name: Provide a meaningful name to the agent.Category: Provide a category to the agent.Model: Select Qwen3:8bMCP Servers: Click on the Add MCP Server Tools button > Select the Playwright MCP Server that we created in the earlier section.Click on the Save button. Step 3: Update the model parameters. Clicking on the Model field, which has Qwen3:8b selected, should open the Model Parameters page. The following parameters can be set using this page: Provider: OllamaModel: Qwen3:8bTemperature: 0.2Top P: 0.85Frequency Penalty: 0.00Presence Penalty: 0.00Reasoning Effort: MediumReasoning Summary: Auto Click on the Save button to set the parameters. Step 4: Setting the instructions for the AI agent. The Following instructions can be pasted into the Instructions field in the Agent Builder window, or a “SKILL.MD” file can be created and uploaded using the Skills section of this agent. Markdown # Skills for the Local AI Agent for automation testing You are an expert QA Automation Engineer controlling a browser through Playwright MCP. Your goal is to execute browser actions safely and reliably. ## Tool Usage Rules - Do not run all MCP tools at the same time - Use only one Playwright MCP tool at a time. - Wait for the result of each tool before deciding the next action. - Never assume the page state. - Inspect the current page before interacting. - Do not start the next MCP tool unless the first one is complete ## Navigation Rules Treat the following actions as navigation-triggering actions: - Clicking Login, Submit, Continue, Save, Next, Checkout, etc. - Clicking any hyperlink. - Form submission. - Any action that changes the URL or reloads the page. - Wait until the page is fully loaded before making another tool call. After any navigation-triggering action: 1. Do not call any DOM inspection tool immediately. 2. Wait until the page has completely loaded. 3. Wait for the URL to stabilize if it changes. 5. Continue only after the new page is available. 6. Never inspect the previous page after navigation. ## Rules for locating web elements - Take a fresh snapshot to inspect the current page - Do not use XPath locator strategy - Use the same field name to locate elements, do not hallucinate and add prefix or suffix to field names - Use Semantic locator strategy: getByRole, getByText, getByLabel, getByPlaceHolder, getByAltText, getByTitle, getByTestId - Never use brittle CSS selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. - Avoid nth() unless there is no unique locator. ## Interaction Rules - Verify and confirm that an element exists before interacting. ## Error Recovery If any Playwright tool fails: - Stop issuing new actions. - Inspect the current page. - Check Interaction Rules - Determine whether navigation has occurred. - Retry only if the page state confirms it is safe. - Do not repeat the same action more than once without confirming that the page state has not changed. Never repeat the same click more than once without checking the current page. ## Important If a click causes navigation, always assume the previous execution context has been destroyed. Do not read the DOM until the new page has fully loaded and a fresh snapshot has been obtained. Show a summary of test execution with the step count and pass or fail status - Run only the steps that are provided; do not hallucinate - Any deviation from these rules is not acceptable - Do not generate any additional steps - Always prioritize stability over speed. Providing instructions to an AI agent helps define its behavior, responsibilities, and the boundaries within which it should operate. These instructions act as persistent guidance, ensuring the agent follows consistent practices every time it performs a task instead of relying solely on the user’s prompt. For detailed setup instructions and troubleshooting guidance, refer to the GitHub repository. With these steps, the local AI agent is now ready to take commands. Running the AI Agent for Browser Automation To start using the AI Agent, click on New Chat.Click on the model name dropdown and select My Agents > The name of the agent that you created. Let’s use the following simple prompt and see how it works. Plain Text open http://playwright.dev verify the page title Once the prompt is submitted, we can observe the browser as the AI agent begins executing the task. The agent invokes the Playwright MCP server, which automatically launches a browser and performs the requested actions to navigate to the website and interact with the page. After the task is completed, Qwen3:8b analyzes the outcome and returns the results directly in the LibreChat conversation, demonstrating browser automation powered by Playwright MCP and Qwen3:8b. Let’s run another prompt for a login test scenario: Plain Text Navigate to https://parabank.parasoft.com/parabank/index.htm Locate "Username" field using "name=username" Enter "john" into the "Username" field. Locate "Password" field using "name=password" Enter "demo" into the "Password" field. Locator "Log In" button using "input[type="submit"] Click on the "Log In" button Verify that the "Accounts Overview" page is displayed This prompt also takes some time to understand the request before execution begins. It is important to note that the clearer and more specific the prompt, the more efficiently the AI agent can interpret and execute it. Well-structured prompts reduce ambiguity, minimize the chances of hallucinations, and typically result in faster execution and more accurate outcomes. As a best practice, break complex tasks into clear, sequential instructions whenever possible to improve the agent’s reliability and overall performance. As shown in the screenshot above, the AI agent invoked five tools from the Playwright MCP server to interact with the application and complete the requested workflow. It navigated to the website, located the username and password fields, entered the provided credentials, and submitted the login form. Finally, it verified that the login was successful by confirming that the “Accounts Overview” page was displayed. Since this setup runs entirely on a local machine, the AI agent takes approximately one minute to begin execution and around 4–5 minutes to complete a simple scenario. For more complex scenarios involving multiple steps, validations, or integrations, the AI agent is expected to take longer to analyze the request and complete the execution. But Execution time can be significantly reduced by running the setup on a machine with more powerful hardware, such as additional RAM, a faster CPU, or a dedicated GPU. Watch the step-by-step YouTube tutorial for Building your Local AI QA Engineer. Final Words Building a local AI QA engineer with Docker, Ollama, LibreChat, and Playwright MCP is an excellent way to explore the future of AI-powered software testing while keeping complete control over the data and infrastructure. By running everything locally, we eliminate recurring API costs, improve data privacy, and create a flexible environment for experimenting with AI-assisted browser automation using natural language. This setup is only the beginning of what’s possible. As we become more familiar with MCP and AI agents, the local QA assistant can be extended by integrating tools such as GitHub, Jira, databases, or custom MCP servers to automate even more of the testing workflow. Happy AI-powered testing!!

By Faisal Khatri CORE

It was 2:14 in the morning when the pager went off. Our recommendation model's inference service had started returning 503s under a traffic spike that, frankly, wasn't even that big. Maybe three times the normal load. By the time I'd opened my laptop, the container had been OOM-killed four times in ten minutes, and Kubernetes was cheerfully restarting it into the same wall every ninety seconds. The image was 14GB. Cold start took eighty seconds. Nobody on the team had looked closely at any of that until it started costing us actual money in lost requests. That night is the reason I now have strong opinions about Docker and AI infrastructure. Why This Keeps Happening Containers became the default way to ship machine learning models because they solve a real problem: a model trained with CUDA 11.8, PyTorch 2.1, and a very specific glibc version doesn't reliably run on a colleague's machine, let alone a fleet of GPU nodes spread across three cloud regions. “It works on my machine” isn't a joke in ML infra; it's a recurring incident report. Docker gives you a way to freeze that dependency tree and ship it as one artifact, and that part genuinely works. What doesn't get discussed enough is the many issues that Docker does not resolve, along with the subtle ways teams exacerbate problems by forcing AI workloads into a packaging model that was originally designed for stateless web services. What We Tried First (and Why It Blew Up) Our first version of the inference image was, in hindsight, a small crime. We started from nvidia/cuda:12.2.0-devel-ubuntu22.04 because someone had seen it in a tutorial, installed the full CUDA toolkit, pip-installed every dependency without pinning, and copied in not just the model weights but three checkpoint versions “just in case.” Fourteen gigabytes. Every deployment pulled the entire image onto a fresh node, and during autoscale events, we experienced over a minute of waiting just for the image to be pulled before the container started loading the model into GPU memory. The first fix everyone reaches for is a multi-stage build, and yes, it helps — but it's not the silver bullet people pitch it as. Splitting a devel build stage from a runtime stage cut us from 14GB to roughly 6GB: Dockerfile FROM nvidia/cuda:12.2.0-devel-ubuntu22.04 AS builder WORKDIR /build COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt --target=/deps FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 COPY --from=builder /deps /usr/local/lib/python3.10/site-packages COPY model/ /app/model/ COPY serve.py /app/ WORKDIR /app CMD ["python3", "serve.py"] That's better, but the real lie in that Dockerfile is the COPY model/ line. We baked multi-gigabyte weights into an image layer, causing a full re-push of the weights with every code change, even minor ones, since Docker re-hashes the entire build context. We moved weights to an external volume pulled from object storage at container start, with a checksum cache to skip redundant downloads. That alone cut most of our deployment time. The trade-off is a slightly more complex startup script and a new dependency on storage being reachable at boot, which is its failure mode. There's no free lunch here; you're just choosing which problem is going to page you at 2 am. The Detour: Skipping Containers Entirely We also tried, briefly, running everything on bare metal with conda environments and skipping containers altogether, mostly because one engineer was convinced Docker added overhead with no real benefit for GPU workloads. It's not a crazy position; Docker's GPU story is genuinely leaky. <nvidia-container-toolkit exposes the host driver directly to the container, leading to potential mismatches that application-level packaging cannot resolve, so there is no real isolation>. However, we reverted that experiment within a sprint because as soon as more than two people interact with the training pipeline, environment drift reoccurs immediately. “Works on my conda env” is just “works on my machine” wearing a hat. What Actually Held Up in Production The architecture that worked was less about clever Docker tricks and more about admitting that an inference container and a training container have almost nothing in common and shouldn't share a Dockerfile, a registry strategy, or a deployment pattern. For serving, we kept images lean, stateless weights externalized, and a health assessment that actually runs a tiny dummy inference instead of just pinging an HTTP port. A server can report itself as “up” while a model failed to load correctly, and that gap has burned us more than once. Locally, a docker-compose GPU reservation block mirrored how production scheduled GPUs, so dev environments stopped lying about resource contention: YAML services: inference: image: registry.internal/rec-model:latest deploy: resources: reservations: devices: - capabilities: [gpu] count: 1 healthcheck: test: ["CMD", "python3", "healthcheck.py"] interval: 15s timeout: 5s retries: 3 For training, we accepted slower builds because training jobs run for hours, and a ninety-second image pull is negligible compared to that, even though the images are larger. Spending engineering time shrinking training images was effort we'd burned for no real payoff. A contrarian point I'll happily defend: not every container needs to be small, only the ones sitting in your hot path. The other decision that mattered more than any Dockerfile tweak was plain layer-caching discipline, putting "pip install before COPY." It sounds obvious when written down, but I've reviewed more than one ML team's Dockerfile that copies the whole repo first “for simplicity” and invalidates every cached layer on a README change. Key Takeaways Externalize model weights from the image; baking them in wrecks your caching and your deploy speed.Multi-stage builds help, but they don't resolve a runtime image that's still hauling around a full CUDA devel toolkit.GPU isolation via containers is partial; complete driver and toolkit mismatches between host and container remain entirely your problem.Training and serving images deserve different optimization priorities; don't apply the same size obsession to both.A health check that only confirms the process is running, without verifying that the model has loaded correctly, will mislead you at the most critical moments. Closing Thought Docker didn't fail us that night; rather, it was our incorrect assumptions about its free services that let us down. It's easy to treat containers as a solved problem because the tooling is so mature for web services and then be surprised when AI workloads expose every shortcut you took. I still think GPU container isolation lacks a clean solution, and I’m curious if upcoming tools will address it or if we'll continue improving our health-check scripts. What's the worst 2 am lesson your infrastructure has taught you?