Cloud Architecture Resources

DZone's Featured Cloud Architecture Resources

Kubernetes Says Ready. Your LLM Still Isn’t.

By Shamsher Khan

CORE

A pod can look healthy in Kubernetes while the model behind it is still not ready to answer a request. That is the gap I wanted to measure. Kubernetes Ready means the pod passed the readiness condition you configured. It does not automatically mean the model is loaded, resident in memory, or able to complete inference. For a normal web service, an HTTP check is often good enough. With an LLM serving pod, it can be too shallow. The process may be running. The API may respond. The model file may even be on disk. The first real request can still spend several seconds loading the model before it completes. I ran controlled Ollama recovery experiments on Kubernetes to see how big that window was. Ready Is Only One Point in the Recovery Path I measured five timestamps: Plain Text T0 - pod replacement requested T1 - Kubernetes reports Ready T2 - inference runtime responds to HTTP T3 - first post-recovery inference request begins T4 - inference request completes successfully Figure 1: Kubernetes Ready vs. Functional Recovery That gave me four useful timings: Plain Text Kubernetes recovery = T1 - T0 Runtime recovery = T2 - T0 Functional recovery = T4 - T0 Ready -> inference gap = T4 - T1 The last one is where the problem becomes visible. The Results I ran 10 pod-replacement tests for each configuration: local Minikube on Mac, CPU-onlyAzure Standard_D16s_v5 Linux VM running Minikube, CPU-onlyOllamallama3.2:1bllama3.2:3bsame 2 CPU / 4 GiB container limit for the 1B and 3B comparison Figure 2: LLM Recovery Experiment Architecture Mean results: MetricLocal 1BLocal 3BAzure 1BAzure 3BKubernetes Ready1.66 s1.96 s1.61 s1.69 sRuntime reachable2.43 s2.44 s2.19 s2.17 sFunctional recovery11.11 s16.27 s5.43 s7.73 sReady -> inference9.45 s14.31 s3.83 s6.05 sModel load5.51 s8.60 s2.16 s3.96 s Kubernetes reported the pod Ready in about two seconds or less in all four configurations. Successful inference came later. The mean Ready-to-inference gap ranged from about 3.8 seconds to 14.3 seconds. The Azure environment was faster than the local environment for the inference-dependent part of recovery, but the gap was still there. I did not try to explain the cross-platform difference with one cause. CPU, storage, virtualization, architecture, and cache behavior can all affect the result. The point was simpler: Kubernetes recovery and inference recovery were not the same event. There Is More Than One Kind of "Ready" The experiments also exposed a few other states that are easy to mix together. The Runtime Can Be Up While the Model Is Gone One early version used emptyDir for Ollama model storage. After pod replacement, Ollama started normally. But: Shell ollama list returned no model. The runtime had recovered. The model artifact had not. Moving the model data to a PVC fixed the persistence problem. The Model Can Be on Disk Without Being Loaded A larger llama3.1:8b test made this very clear. Before inference, ollama list showed the model artifact, but ollama ps showed nothing resident. Cgroup memory usage was only around 14 MiB. After the first request, the model became resident and memory rose to roughly 5.27 GiB. So "model exists" and "model is ready to serve" are different checks. A Warm Node Can Make Recovery Look Better I also ran 10 warm-cache and 10 cold-cache tests for the 3B model on the same Azure node. For the cold condition: Shell sync echo 3 > /proc/sys/vm/drop_caches This clears the Linux page cache, dentries, and inode caches. It is a host filesystem/page-cache test, not an Ollama-specific model cache. metricwarmcoldFunctional recovery7.58 s8.09 sReady -> inference5.70 s6.26 sModel load3.95 s4.61 sRequest wall time5.11 s5.70 s Model load increased by about 16.6% under the cold condition. Kubernetes recovery barely moved. That is a useful warning for repeated recovery tests on the same node: the host may be helping more than you realize. Model Residency Can Overlap During memory testing, loading the 3B model under a 4 GiB limit once failed with: Shell signal: killed It looked like the 3B model did not fit. That was not the actual problem. A 1B model from an earlier request was still resident. When I tested the 3B model alone under the same limit, it worked, and the cgroup showed no OOM kill. The failure came from overlapping residency, not the 3B model by itself. A simple runtime health check would not have told me that. So What Should Readiness Check? A normal readiness probe usually asks something like: Plain Text Is the HTTP endpoint responding? That proves the runtime is reachable. For an LLM workload, I care about a stronger question: Plain Text Can this pod actually complete inference with the model it is supposed to serve? One way to test that is with a minimal inference request: YAML readinessProbe: exec: command: - sh - -c - | curl -sf -X POST http://localhost:11434/api/generate \ -H 'Content-Type: application/json' \ -d '{"model":"llama3.2:1b","prompt":"ping","stream":false}' \ | grep -q '"done":true' periodSeconds: 2 failureThreshold: 1 The exact command will depend on the serving image. The point is not curl. The point is that readiness now checks the model-serving path, not just the process. What Happened During Rollouts? For the 3B readiness test, I sampled Kubernetes EndpointSlice state at roughly 0.5-second intervals during 10 local rollouts and 10 Azure rollouts. metriclocal 3Bazure 3BMean new-endpoint non-serving duration47.6 s11.0 sSampled intervals with zero ready + serving endpoints00Rollouts observed1010 Across those 20 rollouts, I did not observe a sampled interval with zero ready-and-serving endpoints. That is not the same as proving packet-level availability between every sample. What it does show is that the replacement endpoint stayed out of Service eligibility until the inference-aware readiness condition succeeded. That is much closer to what I wanted Ready to mean. Readiness Is a Contract This was the main lesson for me. Readiness is not a universal definition of application health. It is a contract between the workload and Kubernetes. For a normal API, the contract might be: Plain Text My process is initialized and can accept requests. For an LLM workload, it may need to be closer to: Plain Text The runtime is running. The model exists. The model can be loaded. Inference can complete. If the probe only checks the first line but the team reads Ready as all four, the problem is not Kubernetes. The signal is just weaker than the expectation. What This Does Not Prove These tests were CPU-only. They used Ollama. They measured same-node pod replacement. And they used 10 repetitions per condition. So the numbers here should not be treated as universal timings or production SLAs. Cold-node relocation is also a separate problem. Moving an LLM workload to another node brings node-local cache state and possibly image or model acquisition into the recovery path. I am measuring that separately rather than mixing it into these same-node results. Takeaway In these experiments, Kubernetes readiness came back quickly. Inference recovery followed a different timeline. The mean Ready-to-inference gap ranged from about 3.8 seconds to 14.3 seconds, depending on the model and environment. The fix is not to distrust Kubernetes. It is to make the readiness condition represent the state you actually care about. A pod can be healthy. The runtime can answer HTTP. The model can exist on disk. And inference can still not be ready. Those are different states. For the full experiment setup, raw results, methodology, environment captures, and ongoing cold-node work, see the project write-up and repository: https://github.com/opscart/k8s-llm-recovery-lab. For the full experiment setup, methodology, raw results, environment captures, and ongoing cold-node work, see the complete OpsCart write-up and project repository. More

The Startup Time Trick Hiding Inside Your Docker Build

By Garima Agarwal

Every Java developer who runs services on Kubernetes has watched this scene play out. Traffic spikes, the autoscaler adds a pod, and then everyone waits. The container is running in two seconds. The application is not ready for another twelve seconds. During those ten seconds, your existing pods absorb the extra load, latency climbs, and if things are bad enough, the autoscaler panics and adds even more pods that are also not ready. I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work Where Those Twelve Seconds Actually Go When a Spring Boot application starts, the JVM is not mostly running your code. It is loading classes. A plain REST service with Spring Web, Spring Data, and a driver or two loads somewhere between ten and twenty thousand classes before it serves its first request. For every single one of those classes, the JVM does the same ritual. Find the class file inside a jar, read the bytes, parse them, verify the bytecode is legal, and build the internal metadata structures it needs at runtime. Thousands of times. Every startup. In every pod. Here is the part that should bother you. Your container image never changes after you build it. The same jar, the same classes, the same parsing work, repeated identically in every pod that ever starts from that image. The JVM is solving the same puzzle again and again and throwing away the answer each time. CDS is the JVM saying: let me solve it once, write the answer to a file, and just memory map that file next time. What a CDS Archive Is A CDS archive is a file, usually ending in .jsa, that contains classes already parsed and verified, stored in the exact internal format the JVM uses in memory. On startup, the JVM maps this file straight into memory. No finding, no parsing, no verifying. The work was done ahead of time. You have been using CDS without knowing it. Modern JDKs ship with a default archive covering the core JDK classes, which is why java -version is fast. The step almost everyone skips is creating an archive for your application classes, all fifteen thousand of them. That is where the real win lives. The mechanism has one rule that matters for us. The archive must be created with the same JVM and the same classpath that will use it. That rule sounds annoying until you realize a Docker image is the one place in your entire infrastructure where JVM and classpath are frozen forever. Docker is not just compatible with CDS. It is the perfect home for it. The Training Run Creating the archive takes two steps. First you do a training run, where the JVM starts your application, watches which classes get loaded, and writes the list down. Then you exit, and the JVM turns that list into the archive. Since Java 13, this is pleasantly simple: Shell java -XX:ArchiveClassesAtExit=app.jsa -jar app.jar Run the app, let it come up, stop it, and app.jsa appears. From then on you start the app like this: Shell java -XX:SharedArchiveFile=app.jsa -jar app.jar There is an obvious question here. The training run wants to actually start the application, and inside docker build there is no database, no message broker, nothing to connect to. A Spring Boot app that cannot reach Postgres will crash during training. Spring Boot 3.3 solved this neatly. Setting one property makes the application run through its entire startup sequence, create all bean definitions, and then exit just before touching the outside world: Shell java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app.jsa -jar app.jar The application loads nearly everything it will ever load, writes the archive, and exits cleanly with no infrastructure needed. This is exactly what a Docker build stage can do. The Dockerfile Here is the complete picture: a multi-stage build where the image trains itself: Shell FROM eclipse-temurin:21-jdk-alpine AS build WORKDIR /build COPY . . RUN ./mvnw -B package -DskipTests # Explode the jar so the classpath is stable RUN java -Djarmode=tools -jar target/app.jar extract --destination /app FROM eclipse-temurin:21-jre-alpine AS runtime WORKDIR /app COPY --from=build /app /app # Training run: start the context, record classes, exit RUN java -Dspring.context.exit=onRefresh \ -XX:ArchiveClassesAtExit=/app/app.jsa \ -jar /app/app.jar ENV JAVA_TOOL_OPTIONS="-XX:SharedArchiveFile=/app/app.jsa" ENTRYPOINT ["java", "-jar", "/app/app.jar"] Two details in there deserve a closer look. The extract step unpacks the fat jar into a folder with the dependencies laid out as plain files. CDS is picky about the classpath being identical between training and real runs, and a fat jar with nested jars inside it makes that fragile. The exploded layout keeps the classpath boring and stable, which is exactly what CDS wants. On Spring Boot 3.2 and older, the same idea works through the layertools jarmode instead. The training run happens as a RUN instruction, which means it executes once at build time on your CI server. Every container that ever starts from this image inherits the archive for free. You did the class loading homework once, in the build, and ten thousand pod starts copy the answer. What You Get Numbers vary with how heavy your application is, but the pattern is consistent. A typical Spring Boot 3 web service that started in 10 to 12 seconds lands somewhere between 5 and 7. The JVM portion of startup shrinks dramatically, and as a bonus, the archive is memory-mapped and shared, so if you run several JVMs on one node, they share those pages and total memory drops too. You can verify the archive is actually being used, which I recommend, because CDS fails silently by design. If something mismatches, it just quietly falls back to normal class loading: Shell docker run --rm my-service -Xlog:class+load=info | head -5 Classes loaded from the archive say source: shared objects file. If you see jar paths instead, the archive is being ignored, and the log will usually tell you why. The usual culprit is a classpath that differs from training, even by one entry. One honest caveat. The training run exercises startup, not your traffic. Classes that only load when a specific endpoint gets hit for the first time are not in the archive, so those first requests still do normal loading. The archive covers the framework and wiring, which is most of the cost, but it is not a magic warm-up for everything. Why This Beats the Alternatives You Have Heard Of Whenever container startup time comes up, someone mentions GraalVM native images, and native images are impressive. Millisecond startup is real. But they come with a price list: long build times, a closed-world assumption that fights with reflection, some libraries that simply do not work, and a different runtime profile you have to learn to debug. CDS costs you five lines of Dockerfile. Your application is still a completely normal JVM application. Same debugging, same profilers, same libraries, same behavior, just faster out of the gate. For most teams, that trade-off is not even close. It also stacks with what is coming. Project Leyden's AOT cache in Java 24 and beyond is essentially this same idea grown up, caching not just parsed classes but resolved linkage and compiled code. The Dockerfile pattern you build today, a training run at build time producing a cache file shipped in the image, is exactly the shape Leyden uses. Learning it now means the future is a flag change. The Takeaway Your Docker image is immutable. Your JVM does expensive, perfectly repeatable work on every startup. Those two facts fit together like puzzle pieces, and a training run inside docker build is where they connect. One extra build step, and every pod your autoscaler ever creates comes up in half the time. The next time you watch a rollout crawl because pods take forever to go ready, remember that the answer was hiding inside the build all along. More

Making Running Optional: Scaling AI Agents on Kubernetes With Agent Substrate

By Mayowa Fajobi

Deliberate Decoupling: 6 Architectural Patterns From a Regulated WAS-to-AWS Migration

By Alka Nimje

Inside terraform-provider-archive: A Memory Pattern From 2016 That Scales With Your Lambdas

By Oleg Mamiev

Containerizing Spark and Lakehouse Development with Docker

Most Docker content targets web developers shipping stateless services. However, data engineers, who represent a huge and growing population of Dockers users, are mostly left to figure things out alone, and it shows. The get pipelines that pass locally, but explode on clusters. They pit notebook-only development against expensive cloud workspaces, and more. This article applies six years of production data platform experience in financial services and healthcare to a question nobody answers well: How to you make a laptop behave like a lakehouse? A Familiar Routine If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set. Web developers solved "works on my machine" a decade ago with containers. Data engineers, somehow, are still developing against shared cloud workspaces, paying per-minute cluster costs to debug a GROUP BY, and discovering environment drift in production. This article is the workflow I wish someone had handed me years ago: a fully containerized lakehouse development environment — Spark, Delta Lake, object storage, a catalog, and orchestration — that runs on a laptop, mirrors production closely enough to trust, and plugs into CI without mocks. The Real Problem: Data Pipelines Have Four Environments, Not One A typical stateless web service has one environment to reproduce: the app runtime. A data pipeline has at least four, and they drift independently: The compute runtime — Spark version, Scala version, JVM, Python, native libs (Arrow, Parquet, libhdfs).The table format layer — Delta Lake / Iceberg versions and protocol versions, which are not the same thing.The storage layer — S3/ADLS semantics: multipart uploads, eventual consistency quirks, path-style vs virtual-hosted access.The orchestration layer — the scheduler's Python environment, which is famously not your job's environment. Mocking any one of these in tests means you aren't testing the thing that breaks. The goal of containerizing a lakehouse is to pin all four layers in code and version them together. Step 1: A Reproducible Spark Image You Actually Control Don't develop against latest. Build a base image that pins every layer of the compute runtime and treat it like an artifact: Dockerfile # syntax=docker/dockerfile:1.7 FROM eclipse-temurin:17-jre-jammy AS base ARG SPARK_VERSION=3.5.4 ARG DELTA_VERSION=3.3.0 ARG HADOOP_AWS_VERSION=3.3.6 RUN apt-get update && apt-get install -y --no-install-recommends \ python3.11 python3-pip tini && \ rm -rf /var/lib/apt/lists/* # Pin Spark itself, not just PySpark RUN curl -fsSL https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \ | tar -xz -C /opt && mv /opt/spark-${SPARK_VERSION}-bin-hadoop3 /opt/spark ENV SPARK_HOME=/opt/spark PATH=$PATH:/opt/spark/bin PYTHONHASHSEED=0 TZ=UTC # Delta + S3 connectors resolved at build time, never at job submit time RUN /opt/spark/bin/spark-shell --packages \ io.delta:delta-spark_2.12:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION} \ -e "println(\"deps cached\")" && \ cp /root/.ivy2/jars/*.jar /opt/spark/jars/ COPY requirements.lock /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.lock # Never run Spark as root RUN useradd -m -u 1001 spark USER 1001 ENTRYPOINT ["/usr/bin/tini", "--"] Three details that matter more than they look: --packages at build time, not submit time. Resolving connector JARs at spark-submit is the #1 source of "it worked yesterday" failures — Maven Central is a runtime dependency you didn't mean to have.PYTHONHASHSEED=0 and TZ=UTC kill two classes of "non-deterministic only in prod" bugs.A lockfile, not requirements.txt. Compile with pip-compile or uv pip compile so transitive dependencies (looking at you, pandas/pyarrow) can't drift. Step 2: The Lakehouse-In-A-Box With Docker Compose Here's the part most teams never build: the rest of the lakehouse, locally. MinIO stands in for S3 (it speaks the same API), and a real Spark master/worker pair stands in for the cluster, because local[*] mode hides every serialization and shuffle bug you'll meet in production. Dockerfile # compose.yaml services: spark-master: build: . command: /opt/spark/sbin/start-master.sh environment: [SPARK_NO_DAEMONIZE=true] ports: ["7077:7077", "8080:8080"] spark-worker: build: . command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077 environment: - SPARK_NO_DAEMONIZE=true - SPARK_WORKER_MEMORY=4g - SPARK_WORKER_CORES=2 depends_on: [spark-master] deploy: replicas: 2 # >1 worker = real shuffles, real serialization minio: image: minio/minio:RELEASE.2025-09-07T16-13-09Z command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: localdev MINIO_ROOT_PASSWORD: localdev-secret ports: ["9000:9000", "9001:9001"] volumes: [lake-data:/data] healthcheck: test: ["CMD", "mc", "ready", "local"] interval: 5s mc-init: # create the bronze/silver/gold buckets on boot image: minio/mc:latest depends_on: { minio: { condition: service_healthy } } entrypoint: > /bin/sh -c "mc alias set local http://minio:9000 localdev localdev-secret && mc mb -p local/lakehouse/bronze local/lakehouse/silver local/lakehouse/gold" volumes: lake-data: Point Spark at MinIO with three config lines and your medallion pipeline reads and writes s3a://lakehouse/... paths exactly like production: Python spark = (SparkSession.builder .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") .config("spark.hadoop.fs.s3a.path.style.access", "true") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate()) docker compose up and you have bronze → silver → gold on your laptop. Total cloud cost of a debugging session: $0. Step 3: Integration Tests That Run Real Spark — Testcontainers The payoff of all this is CI you can trust. With Testcontainers, your pipeline tests spin up the same images your developers use: Python import pytest from testcontainers.minio import MinioContainer from pyspark.sql import SparkSession @pytest.fixture(scope="session") def lake(request): with MinioContainer("minio/minio:RELEASE.2025-09-07T16-13-09Z") as minio: yield minio def test_silver_dedup_keeps_latest_record(lake, spark): # write duplicate customer events to bronze bronze_path = f"s3a://test/bronze/customers" write_fixture_events(spark, bronze_path, duplicates=True) run_silver_dedup(spark, bronze_path, "s3a://test/silver/customers") result = spark.read.format("delta").load("s3a://test/silver/customers") assert result.count() == EXPECTED_UNIQUE assert latest_record_wins(result) No mocked DataFrames. No unittest.mock.patch("boto3..."). The test exercises Delta's actual transaction log against actual object storage. When this suite is green, deployments stop being scary. A pattern I use in regulated environments: keep a fixtures/ directory of small, synthetic Parquet files that mirror production schemas (never production data), and version them with the code. Schema drift then fails a unit test instead of a 2 a.m. pipeline run. Step 4: One Image From Laptop → CI → Production The final principle: the image you test is the artifact you ship. Multi-stage builds let one Dockerfile serve dev (with Jupyter, debuggers) and prod (minimal, non-root): Dockerfile FROM base AS dev USER root RUN pip install --no-cache-dir jupyterlab pytest debugpy USER 1001 FROM base AS prod COPY --chown=1001:1001 src/ /app/src/ COPY --chown=1001:1001 jobs/ /app/jobs/ # nothing else — no notebooks, no test deps, no shell tools you don't need In CI: build once, tag with the git SHA, run the Testcontainers suite against prod, scan it (Docker Scout, or your registry's scanner), sign it, and promote that exact digest through staging to the scheduler. Whether the scheduler is Airflow's DockerOperator/KubernetesPodExecutor or a managed Spark platform pulling custom containers, the principle holds: environments are immutable, versioned, and identical by construction. Lessons Learned From Production Run ≥2 workers locally.local[*] mode never serializes between JVMs. The day you switch to a real cluster, every closure-capture and UDF-pickling bug appears at once. Two 2-core workers in Compose surfaces them on day one.Pin the table format protocol, not just the library. Delta and Iceberg both evolve table protocol versions. A newer writer can produce tables an older reader can't open. Encode the protocol version in your image build args and test reads with the oldest reader you support.MinIO is a stand-in, not a clone. It won't reproduce S3 request throttling or cross-region latency. Keep a small smoke-test suite that runs against real object storage nightly; do everything else locally.Resource-limit your local Spark. Without SPARK_WORKER_MEMORY caps, a skewed join will cheerfully eat your laptop. Limits also force you to think about partitioning early — which is the point.Treat the orchestrator's image as layer four. Airflow DAG-parse environments drift too. Containerize the scheduler with the same lockfile discipline as the jobs. Production Considerations Before you take this pattern to a real platform team, three things to plan for: secrets (local Compose uses throwaway creds; production should inject via your cloud's secret manager or Docker secrets — never baked into images), image provenance (sign images and generate SBOMs in CI; regulated industries will ask, and in 2026 the tooling is mature enough that "we didn't get to it" no longer flies), and base image hygiene (start from minimal, hardened bases and rebuild on a schedule, not just on code change — CVEs don't wait for your sprint). Conclusion Containers gave application developers reproducibility ten years ago. Data engineering is finally having the same moment — and the teams that containerize their lakehouse development loop ship faster, test honestly, and stop paying cloud bills to find typos. Try it: clone the Compose stack above, point your gnarliest pipeline at it, and see what breaks locally that used to break in prod. Then tell me about it — I'd genuinely like to hear which layer drifted on you. If this was useful, follow me here and on LinkedIn. Next up in this series: load-testing Delta merge performance locally, and contract testing between pipeline stages.

By Aniket Abhishek Soni

Multi-Account AWS Architecture: Isolating PHI Workloads Without Slowing Down Engineering Teams

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

By Garik H

From Bottlenecks to Reliability: A Practical Guide to Scaling Temporal in Production

Temporal is designed to preserve Workflow state through process crashes and infrastructure failures, but durable state does not remove ordinary capacity limits. In production, the control plane can remain healthy while throughput collapses because Worker slots are saturated, Task Queues mix incompatible workloads, or a failover activates a region without enough Worker capacity. Temporal Workers run outside the Temporal Service and execute Workflow and Activity code, so production scalability depends as much on Worker and routing design as on the service itself. The Worker Fleet Is Usually the First Capacity Boundary Schedule-to-Start latency is best treated as queueing delay rather than application execution time. It measures the interval between a Task being enqueued and a Worker starting it. Rising Schedule-to-Start latency, growing approximate backlog, and exhausted Worker task slots indicate that Tasks are arriving faster than the fleet can consume them. Temporal Cloud exposes temporal_cloud_v1_approximate_backlog_count, while SDK metrics expose Workflow and Activity Schedule-to-Start latency and available task slots. Temporal guidance recommends watching these signals together because backlog depth alone does not identify whether the limit is Worker count, Worker configuration, or polling behavior. Worker scaling has two layers. Horizontal scaling adds Worker processes, while concurrency tuning changes how many Tasks each process can execute simultaneously. For well-benchmarked workloads, fixed slot limits place a predictable ceiling on local resource consumption. The Java SDK exposes separate concurrency controls for Workflow Tasks and Activities, and a server-side Activity rate limit can cap dispatch across all Workers polling the same Task Queue. Java WorkerOptions options = WorkerOptions.newBuilder() .setMaxConcurrentWorkflowTaskExecutionSize(120) .setMaxConcurrentActivityExecutionSize(80) .setMaxTaskQueueActivitiesPerSecond(250) .build(); The values in this example are capacity-test outputs, not universal defaults. A CPU-heavy Activity fleet may need a lower Activity slot count than an I/O-heavy fleet. Newer Worker tuners can allocate slots dynamically from CPU and memory signals, while fixed-size suppliers remain more predictable when task resource cost is well understood. Temporal also recommends poller autoscaling for most workloads because too few pollers constrain ingestion and too many waste connections and reduce efficiency. Task Queue Topology Determines Isolation and Backpressure Adding replicas cannot repair a Task Queue topology that couples unrelated bottlenecks. A shared Task Queue is reasonable when Workflows and Activities have similar latency and resource characteristics, but it becomes risky when fast orchestration work shares capacity with slow database calls, GPU jobs, tenant bursts, or Activities constrained by a downstream API. Temporal supports specialized routing through separate Task Queues, and Activity-level server-side throttling applies to the entire queue. A throttled Activity therefore should not share a queue with work that must remain unrestricted. A Workflow can route a costly Activity to a dedicated fleet without changing the Workflow’s own Task Queue. The separation creates an independent scaling and backpressure boundary. Java ActivityOptions options = ActivityOptions.newBuilder() .setTaskQueue("payments-io") .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); PaymentActivities payments = Workflow.newActivityStub(PaymentActivities.class, options); With payments-io isolated, replicas, concurrency, credentials, network placement, and queue-wide rate limits can be tuned for payment traffic without changing the Worker pool that advances Workflow Tasks. The same principle applies to multi-tenant systems. Temporal documents per-tenant Task Queues as a strong isolation pattern and also supports fairness keys when many tenants share one queue. Priority and fairness operate within Task Queue partitions, so they manage contention inside a queue rather than replacing isolation when resource requirements differ fundamentally. Task Queue partitioning should also be distinguished from application-level queue proliferation. Temporal Task Queues are lightweight and scale internally through partitions; current documentation states that Task Queues use four partitions by default. Multiple partitions increase throughput but relax strict FIFO behavior because Tasks are distributed among partitions. Separate named queues should therefore be created for routing, isolation, or rate-control reasons, not merely to manufacture throughput that Temporal’s matching layer can already scale internally. Autoscaling Should Follow Queue Pressure, Not CPU Alone CPU-based autoscaling is insufficient for many Temporal workloads. An I/O-bound Activity can leave CPU utilization low while all Activity slots are occupied and backlog grows. Conversely, high CPU with near-zero Schedule-to-Start latency may mean that the fleet is efficiently utilized. A stronger autoscaling policy combines queue delay, backlog trend, slot availability, and host resource saturation. Temporal’s Worker health guidance treats Schedule-to-Start latency as a primary symptom of insufficient processing capacity and recommends correlating it with sync-match behavior and available slots before changing fleet size. On Kubernetes, Temporal’s Worker Controller can attach HPA or KEDA resources to versioned Worker deployments and scale from CPU, memory, Task Queue backlog, slot utilization, or custom metrics. Current guidance recommends HPA with a Prometheus adapter as the general default, while KEDA is positioned for scale-to-zero, long idle periods, or faster event-driven reactions. This matters because old and new Worker versions can coexist during safe rollout, so autoscaling should follow each active Worker Deployment Version rather than treating the fleet as a single anonymous pool. Scale-down deserves the same attention as scale-up. Backlog can reach zero while Activities are still running, and terminating aggressively can create retries or latency spikes. Worker shutdown should therefore be graceful, minimum replica counts should reflect availability requirements, and cooldowns should account for Activity duration and startup time. Pre-production tests should include Worker termination, burst recovery, and partial failure because Temporal durability preserves state but does not guarantee that an undersized replacement fleet will meet latency objectives. Regional Failover Has to Include Workers and Dependencies Regional failover is often mis-scoped as a Temporal Service feature. Temporal Cloud High Availability replicates a Namespace to a secondary region and can automatically promote the replica during an outage, but application Workers remain separately operated compute. Temporal documents a 20-minute RTO and sub-one-minute RPO for its HA service, yet application recovery can still be slower when the secondary region lacks ready Worker capacity, network access to the active Namespace, or available downstream systems. For latency-sensitive systems, Active/Hot-Passive is the most deterministic failover model: a full Worker fleet runs in both regions, the secondary fleet stays warm, and only the fleet local to the active replica processes Tasks. On failover, the warm fleet begins processing without a Worker cold start. Active/Passive costs less but requires starting or scaling Workers after failover, while Active/Active runs Workers in multiple regions even though the HA Namespace still has one active replica underneath. Connectivity must be tested as part of the failover path. For HA Namespaces, the Namespace Endpoint follows the active region through DNS; Temporal documents a 15-second TTL and roughly 30 seconds for clients to converge when resolvers honor that TTL. Private connectivity requires routes and DNS design that allow Workers to reach the promoted region. A test that switches only the Namespace but omits Worker connectivity, database promotion, queue access, secrets, codec servers, or proxies validates only part of the production path. Self-hosted multi-cluster deployments require explicit planning as well. Temporal’s Global Namespace model uses asynchronous cross-cluster replication and eventual conflict resolution, and successful failover requires Worker Processes to poll the Namespace in clusters that may become active. Replication versions determine which cluster can mutate Workflow history after failover, but they do not provision Worker compute or external dependencies. Conclusion Temporal becomes a production bottleneck when durable orchestration is treated as a substitute for capacity engineering. Stable performance comes from measuring queue delay and slot saturation, scaling Worker fleets from demand signals rather than CPU alone, separating Task Queues where workloads need independent isolation or rate control, and designing regional failover around ready Workers and reachable dependencies. With those boundaries in place, Temporal remains the durable coordination layer rather than the slowest component in the execution path.

By Akhil Madineni

CORE

AWS Bedrock vs Vertex AI vs Azure Foundry: Stop Comparing Benchmarks, Start Asking This Instead

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

By Balaji Venkatasubramaniyar

CORE

How Docker Is Becoming an AI Development Platform

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

By Pruthvi Raj Seknametla

Containerizing LLMs: Best Practices for Docker-Based AI Workloads

The first time I containerized a fine-tuned Llama model for a client's internal search tool, the build finished at 38 gigabytes. I remember staring at the terminal thinking there was no way that was right. It was right. The image included a CUDA base, PyTorch with every backend compiled in, model weights baked directly into the layer, and a pip cache that had not been cleaned. Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all. Why This Problem Actually Matters Most Docker advice out there is written for stateless web services, small images, fast cold starts, and horizontal scaling on demand. LLM workloads break almost every assumption baked into that advice. The artifact is huge, the runtime is GPU-bound, startup involves loading gigabytes into VRAM, and half your "application code" is actually a C++/CUDA binary blob you didn't write and can't easily trim. If you treat an inference container like a Flask app with a bigger base image, you end up with slow deploys, wasted GPU spend, and autoscaling that technically works but arrives too late to matter. The First Wrong Turn: One Image to Rule Them All Our early approach was a single monolithic image model with weights, tokenizer, inference server, and dependencies all baked together, rebuilt on every model version bump. It felt simple. It wasn't. Every retrain meant rebuilding a 30+ GB image even when the code hadn't changed a single line. Registry storage costs gradually increased until someone in finance questioned why our container registry bill resembled that of a second AWS account. Worse, rollbacks were painful because reverting to a previous model meant pulling an entire previous image rather than swapping a much smaller artifact. The solution that actually worked was separating the model weights from the serving image entirely. The image contains the runtime, the inference server (we used vLLM for most of our transformer workloads), and pinned dependencies. Weights live in object storage and are pulled at container start via an init container or a lazy loading entry point. The approach felt counterintuitive at first. Are we effectively transitioning the slower process to startup instead of build time? — but it turned out to be the right trade. Startup pulls are parallelizable, cacheable on the node, and don't bloat the registry. Build time dropped from twenty-plus minutes to under four. A Smaller Base Image Than You'd Expect This is where the challenges began. Everyone defaults to using nvidia/cuda:*-devel images because the framework documentation recommends them, but these devel images include the entire CUDA toolkit, which contains compilers that you will never use at runtime. Switching to the runtime variant and only installing the exact CUDA and cuDNN versions your framework's wheel actually needs cuts roughly 4GB off the base alone. A minimal multi-stage build looks something like this: Dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder RUN pip install --no-cache-dir vllm==0.4.2 FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 COPY --from=builder /usr/local/bin/python3.10 /usr/local/bin/ ENV MODEL_PATH=/mnt/models ENTRYPOINT ["python3", "-m", "vllm.entrypoints.api_server"] The build stage compiles anything that needs the full toolkit; the runtime stage only carries what's needed to execute. It's a basic Docker pattern, but I've seen it skipped constantly on ML teams because the assumption is always, "the model is the heavy part; the image doesn't matter." The model is heavy, sure, but a bloated base image adds real minutes to every autoscale event, and in production that's the difference between absorbing a traffic spike and dropping requests. The OOM Kill: Nobody Explained Well This is the war story I bring up most often. We had a container that ran fine locally and in staging, then got silently killed in production under load — no crash log, no stack trace, just a pod restart and a confused on-call engineer at 2 AM. It turned out to be the kernel OOM killer, not an application-level exception, because our memory limit accounted for the model weights in VRAM but excluded the growing KV cache for long-context requests plus the CPU-side tokenizer buffers. GPU memory and container memory limits are two completely separate accounting systems, and Kubernetes will happily kill your pod over host RAM even if your GPU has headroom to spare. The fix was unglamorous: we set explicit memory requests and limits with a real margin above peak KV cache usage, moved batch size and max sequence length into environment-configurable values instead of hardcoding them, and added a lightweight health assessment that reported GPU memory utilization alongside the standard liveness probe. None of that is exotic. All of it was missing because we'd copy-pasted a manifest template built for a stateless API and never revisited the resource math for a model that holds state in memory for the duration of a request. Where I'd Push Back on Common Advice A lot of guidance recommends one model per container for isolation, and for many teams that's right. But if you're serving several small fine-tunes of the same base model, that pattern wastes GPU memory by duplicating base weights across containers. We transitioned to a multi-adapter setup, where one base model is loaded once, and LoRA adapters are swapped for each request; this approach is more complex operationally but reduces the GPU footprint by nearly half. I wouldn't consider it a default; it represents a level of complexity that is justified only after demonstrating that plain per-model containers are indeed the bottleneck. I'd also push back on containerizing every workload the same way. Batch inference and real-time serving have almost opposite goals: one wants throughput and tolerates slow cold starts; the other needs rapid readiness and predictable latency. We split these into separate images with separate resource profiles, even though it meant more Dockerfiles. Fewer surprises beat fewer files. Key Takeaways Separate model weights from the serving image; bake them in the runtime and pull weights at startup from object storage.Use CUDA runtime images, not devel images, unless you genuinely compile something at container start.Account for GPU memory and host memory as two separate budgets; KV cache growth is the usual silent killer.Split batch and real-time serving into different images; their optimization goals are conflicting.Don't reach for multi-adapter serving or other density tricks until you've measured that plain per-model containers are actually the bottleneck. Closing Thought None of this required exotic tooling, no custom orchestrator, and no proprietary platform. It required treating the container as part of the model's runtime behavior rather than a packaging afterthought bolted on after the research work was done. The teams that struggle most with this approach usually aren't lacking Docker knowledge; they're applying web-service intuition to a workload that behaves nothing like a web service. If you're mid-migration on something similar, I'd genuinely ask: are you optimizing your image for build convenience or for what actually happens the moment traffic hits a cold node? Those answers are rarely the same, and figuring out which one you've been solving for is usually the first real fix.

By Pruthvi Raj Seknametla

Docker Containers Don’t Know Your Model Is Still Loading

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we'd agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas. What it didn't account for is that each replica needed almost three minutes just to pull a 14GB checkpoint and warm up CUDA kernels before it could answer a single request. The load balancer, seeing new pods report as running, immediately started routing traffic to them. For three minutes, a chunk of our users got 504s while perfectly healthy-looking pods sat there loading a model into memory. Nobody on the infra side had touched Docker that day. The incident wasn't a Docker bug. We assumed that container orchestration designed for web services would function the same way for processes that take minutes to become useful, rather than those that operate in milliseconds. Why LLM Containers Break the Usual Assumptions Packaging an LLM serving stack in Docker still makes sense for the same reason it always has; CUDA versions, driver compatibility, and Python ABI mismatches are miserable to manage across a fleet without a frozen artifact. But an LLM container carries baggage that a typical inference service doesn't. The weights are tens of gigabytes, not a few hundred megabytes. GPU memory is a single shared pool that one greedy container can quietly exhaust for everyone else on the box. And “ready” doesn't mean “process started”; it means the model is resident in VRAM and the CUDA graph is warmed, which can take minutes on a cold node pulling weights from object storage over the network. The Mistakes, in Order Our first version baked the model weights directly into the image, because it felt simpler: one artifact, one pull, done. In practice, it meant a 16GB image, painfully slow CI pushes, and a registry bill nobody wanted to look at. Worse, every time we bumped into a new fine-tuned checkpoint, we rebuilt and repushed the entire layer regardless of caching, because the COPY step touching gigabytes of weight files invalidates everything below it. Unlike a typical ML inference image, there's no meaningful caching win here at all; the layer is simply too big to ever be a cache hit across versions. We moved weights out to a mounted volume, fetched at container start from object storage, and never looked back. Second mistake, and this one actually cost us a production incident: we ran the container with Docker's default shared memory size. vLLM, which we used for serving, spins up worker processes that talk to each other over shared memory even on a single GPU. With the default 64MB /dev/shm, those workers would crash with cryptic bus errors under any real concurrency. The fix was almost embarrassingly small: Shell docker run --gpus all \ --shm-size=2g \ -e MODEL=mistralai/Mistral-7B-Instruct-v0.2 \ -e GPU_MEMORY_UTILIZATION=0.85 \ -e MAX_MODEL_LEN=8192 \ -p 8000:8000 \ llm-serve:latest The third mistake was more subtle and took longer to diagnose. vLLM's continuous batching reserves a large slice of GPU memory upfront for the KV cache, controlled by gpu_memory_utilization. We'd set that fraction high to maximize throughput, then bin-packed two replicas onto the same GPU to save cost. Under normal traffic, fine. During a burst of unusually long-context requests, such as someone summarizing a 6,000-word document instead of a tweet, the KV cache for that single batch ballooned, causing the container to run out of memory (OOM) mid-generation and taking down every other in-flight request in the same batch. This failure mode is more severe than a typical web service OOM because it not only drops the new request but also terminates queries that were already halfway through generating answers for paying customers. What We Actually Changed The readiness adjustment turned out to matter more than any Docker flag. We split liveness from readiness: liveness just checks that the process hasn't died; readiness fires a real, tiny generation request through the local API and only flips to healthy once that round trip succeeds. That alone killed the cold-start routing problem because the load balancer stopped trusting a merely alive process. We also gave up on bin-packing two replicas per GPU. In hindsight, treating GPU memory like it's as elastic as CPU or RAM was the actual root cause, not any single Docker setting. We implemented a model that uses one GPU, sets a conservative memory utilization ceiling, and enforces a request-level token limit at the proxy in front of the container, rather than inside it, because it is too late to make adjustments once the batch is already running. On the orchestration side, we stopped trying to scale-to-zero or scale aggressively off CPU-style metrics. Scale-to-zero is effective for web apps but doesn’t fit GPU-bound LLM serving, where cold starts can outlast traffic spikes. We kept a warm floor of replicas sized to baseline traffic and let a request queue absorb bursts instead of expecting new pods to materialize in time. It's less elegant than the autoscaling story everyone likes to tell, and it costs more in idle GPU time, but it's honest about what the hardware can actually do. What We Rejected, and Why We seriously considered dropping self-hosting altogether and routing through a managed inference API. For a side project, that's probably the right call — less to own, no GPU bin-packing headaches. We rejected it due to data residency requirements that prohibited sending raw text to a third party, and at our volume, managed pricing would quickly exceed our GPU costs. We also looked at Ray Serve and Triton early on, and they solve some of the issues more natively, but the team's Docker and Kubernetes muscle memory was strong enough that rebuilding on a new serving framework felt like trading one set of unknowns for another, at least for the first version. Key Takeaways Never bake multi-gigabyte model weights into the image — there's no caching benefit at that size, only slower pushes and bigger registry bills.Set shared memory explicitly; vLLM and similar multiprocess servers will fail under load with Docker's tiny default.Treat GPU memory utilization conservatively and avoid bin-packing replicas onto a single GPU unless you can guarantee a strict ceiling per container.Build a readiness assessment that performs a real generation, not just a process check; cold model loading will otherwise receive routed live traffic.Don't expect autoscaling to save you on cold-start timescales measured in minutes; a warm floor plus a queue is more honest than reactive scaling. Closing Thought None of these issues was really a Docker failure; the container did exactly what we told it to do. The failure was treating a multi-gigabyte, GPU-bound, slow-to-warm process like it was just another stateless web container that happens to need a GPU flag. I suspect that many teams will learn this lesson in the same way we did, during an incident on a Friday afternoon. Is it the right move to keep stretching Docker and Kubernetes to fit LLM serving, or is this the workload that finally pushes most teams toward purpose-built serving layers?

By Pruthvi Raj Seknametla

How Different Docker Engine Versions Led to Partial Traffic Unavailability in Docker Swarm

If different Docker Engine versions are running simultaneously in a Docker Swarm cluster, this may lead not to an obvious service outage but to a more subtle scenario: partial traffic degradation on individual nodes. In this case, the issue appeared on one of the manager nodes, Traefik started reporting an unavailable status (health=0) for the router-app service, and the cause, according to the working hypothesis, was related to differences in iptables rules and overlay networking between Docker 28.1.1 and 28.2.2. On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time. Context At the time of the incident, the Docker Swarm cluster consisted of 5 manager nodes, several dozen worker nodes, and approximately 40–50 services. External HTTP traffic passed through Traefik, deployed on each of the five manager nodes, and was then routed by Traefik to the backend application containers. One of the key services was router-app, responsible for building public transportation routes on the frontend. It was one of the critical entry-point services with the highest SLA, and any disruption to its availability could have led to severe penalties from the customer, so any deviation in its availability required an immediate response. Grafana dashboards showed application availability through Traefik health-check statuses. Those statuses were generated based on HTTP health-check endpoints implemented by the developers for most services, primarily the most critical ones. This was enough to quickly localize the problem at the ingress traffic level. The cluster had one important characteristic. Some nodes were running Ubuntu 20.04 (focal), while others were running Ubuntu 22.04 (jammy), and different APT repositories were pulling different Docker Engine versions. As a result, after another scheduled Docker update on the nodes, the production environment ended up with a mix of nodes running 28.1.1 and 28.2.2 at the same time. The docker node ls screenshot additionally confirmed that mixed versions were present not only on worker nodes but also on manager nodes, including swarm4 and swarm5. How the Incident Manifested The incident was detected not through user complaints and not through a general service outage, but through Traefik monitoring. The triggered alerts showed partial unavailability of one of the router-app containers, after which the health-check dashboard confirmed that the issue affected not the entire service but one of the manager nodes. This is important because the red blocks on the dashboard did not indicate a complete outage of router-app. It meant that Traefik on one of the manager nodes started receiving health=0 when checking that service’s backend endpoint, while the other manager nodes continued to see the backend as healthy. In practice, it looked like this: traffic through one of the manager nodes stopped reaching the router-app containers correctly, but Traefik, running on all five manager nodes, automatically excluded requests to the unhealthy entry point. As a result, from the outside the incident appeared as partial degradation rather than full unavailability. That is exactly what made the situation tricky. Fault tolerance limited the impact of the incident, but the underlying cause remained inside the cluster and continued to affect one of the entry points. What Docker Showed After localizing the issue to one of the manager nodes, it became clear that the cause should be sought not in router-app itself but in the network path between Traefik and the backend containers. At the same time, docker service ps did not show a widespread service failure, and the containers still appeared as running. The next useful signal came from the dockerd logs on swarm5. Repeated messages appeared there, including Peer delete operation failed, neighbor entry not found, and errors related to deleting FDB and neighbor entries for the VXLAN interface vx-001001-5lk08. For example: Plain Text Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.366807202Z" level=warning msg="Peer delete operation failed" error="could not delete fdb entry for nid:5lk08r7jjvtq5idqggzeygmlv eid:4e7d63d00fffaa6be7ce6362f47acd7912f7c11e5ac6e018393722decc16c210 into the sandbox:neighbor entry not found for IP 10.170.0.37, mac 02:42:0a:1b:14:2c, link vx-001001-5lk08" Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.765092537Z" level=warning msg="error deleting neighbor entry" error="no such file or directory" ifc=vx-001001-5lk08 ip=10.170.0.138 mac="02:42:0a:1b:14:65" Such messages were highly consistent with problems in Docker Swarm’s overlay network. In essence, Docker was trying to delete network records that were no longer present in the tables, which usually points to desynchronization of network state at the VXLAN, FDB, or neighbor-table level. By themselves, these messages still did not provide a complete explanation, but they pushed the investigation in the right direction. It became clear that the problem was not in the application’s business logic but in the network layer on one of the nodes. Additionally, docker node inspect self --pretty on swarm5 showed that from Swarm’s point of view the node looked normal: State: Ready, Availability: Active, Raft Status: Reachable, Leader: No, while Engine Version was already 28.2.2. This was an important point: the control plane still considered the node healthy, even though at the traffic-flow and network-state level it was already behaving differently. Diagnostics The investigation was carried out at the node level. The tools used included docker node ls, docker version, apt-cache policy docker-ce, as well as ip link show, bridge fdb show, ip neigh show, and comparisons of iptables chains across different nodes. The key fact became visible after docker node ls. The cluster was not homogeneous in terms of Docker Engine version: some manager nodes and some worker nodes were already running 28.2.2, while the others remained on 28.1.1. This led to the assumption that the issue might be at the iptables rules level. After that, iptables had to be compared separately on healthy and problematic nodes. On swarm5, a full rules dump was collected using the combination of iptables -S, iptables -t nat -S, and iptables -t mangle -S. Those rules showed the DOCKER, DOCKER-FORWARD, DOCKER-INGRESS, and DOCKER-USER chains, as well as ACCEPT, DROP, and DNAT rules for traffic through docker_gwbridge, published ports, and ingress routing.... To test the hypothesis, not only the problematic swarm5 but also the first manager node, swarm1, was compared, where a stable stack with Docker 28.1.1 had long been running. On swarm1, the output of iptables -S and iptables -t nat -S showed the expected picture: the DOCKER and DOCKER-INGRESS chains contained a full set of ACCEPT and DNAT rules for all published ports (80, 8080–8082, and dozens of internal service ports) with symmetric dport/sport pairs, while DOCKER-USER effectively boiled down to a clean RETURN. Taken together with the dump from swarm5, this reinforced the conclusion that on nodes running 28.1.1, the iptables configuration for ingress and routing was consistent, and the differences seen on 28.2.2 were related not to manual changes but to the behavior of Docker Engine itself. After that, iptables had to be compared separately on other nodes running different Docker versions. On nodes with 28.1.1, the DOCKER and DOCKER-USER chains and the associated rules were in the expected state, whereas on nodes with 28.2.2 some of the required rules were missing or the chains were reduced to a minimal RETURN. This explained the observed behavior well. The services remained running, Swarm did not appear broken, but external traffic and part of the overlay routing through a specific manager node were working incorrectly, causing Traefik on that node to report health=0 for router-app. It is worth noting separately that journalctl -u docker and docker service ps did not provide a simple direct cause for the incident. They did not show a picture of a general failure, so the conclusion had to be assembled from several sources: Traefik monitoring, dockerd logs, Docker versions, and the state of iptables on different nodes. Fix Once the main hypothesis had narrowed down to mismatched Docker Engine versions, the solution was fairly straightforward: return the cluster to a homogeneous configuration by rolling back to version 28.1.1 as the fastest solution. The rollback was performed for swarm4, swarm5, and all worker nodes where version 28.2.2 had already been installed. To do this, a specific package version was pinned via apt, then Docker was restarted, and the installed version was verified. One version of the commands looked like this: Shell apt-cache madison docker-ce | grep 28.1.1 apt-get install docker-ce=5:28.1.1-1~ubuntu.22.04~jammy \ docker-ce-cli=5:28.1.1-1~ubuntu.22.04~jammy \ containerd.io && systemctl restart docker && docker --version Additionally, it made sense to check the package sources and, if necessary, remove conflicting APT entries so that the nodes would no longer receive an unsuitable Docker version from another repository. In practice, it looked like this: Shell sudo rm /etc/apt/sources.list.d/download_docker_com_linux_ubuntu.list sudo apt update After the rollback, docker version was checked again, as well as the DOCKER and DOCKER-USER chains. After Docker Engine had been unified to 28.1.1 on both manager and worker nodes, the issue disappeared. From the perspective of external behavior, this was confirmed immediately. Health checks in Traefik returned to the green zone, and the partial unavailability of router-app on one of the manager nodes could no longer be reproduced. Root Cause Based on the available data, the most well-founded working version is this: in this environment, Docker Engine 28.2.2 formed or applied iptables rules related to DOCKER, DOCKER-USER, FORWARD, ingress, and overlay networking differently. In a mixed cluster, this led to one of the manager nodes no longer forwarding traffic correctly to the router-app backend containers, even though from the perspective of the control plane and service state this did not look like a direct failure. It is important here not to overstate what the data allows. This case does not prove a universal upstream bug in Docker 28.2.2 for all Swarm installations, but it does show that even closely related Docker Engine versions can affect the cluster’s network plane differently, especially when different Ubuntu distributions and different package sources are present in production at the same time. What Follows From This The first conclusion is simple: Docker Swarm is sensitive to Docker Engine version mismatches. If some manager or worker nodes have been updated while others have not, this can lead not only to version drift as an organizational problem, but also to practical issues with traffic, published ports, and overlay routing. The second conclusion is that after updating Docker, it is necessary to check not only docker version but also the node’s network behavior. The minimum set includes iptables -L DOCKER -v -n, iptables -L DOCKER-USER -v -n, checking published ports, ingress/overlay state, and health checks from the edge proxy. The third conclusion is that it is useful to maintain a single baseline stack across all nodes. One Ubuntu LTS distribution, unified repositories, and the same update order reduce the chance that cluster state will remain formally healthy while part of the network traffic is already being handled incorrectly. The fourth conclusion concerns update order. In our case, that was exactly how it happened, but it is worth noting separately. When Docker is updated in a Swarm cluster, it is better to update worker nodes first, then manager nodes, and the leader last, while after each stage separately checking the node’s behavior under real traffic conditions (service availability, correct routing, and published ports). In our case, enhanced monitoring was in place, so no additional manual checks of node behavior in traffic were required: if any part of the infrastructure became unavailable, we would promptly receive an alert. The final conclusion relates to monitoring. In this case, Traefik not only helped limit the impact of the incident by routing around the unhealthy node, but also provided the first precise signal that the problem was localized to a specific entry point rather than existing at the level of the entire service or the entire cluster.

By Denis Tiumentsev

Reliability Challenges in Multi-Cloud Environments: Why Two Clouds Are Often Harder Than One

The pitch for multi-cloud always sounds clean. Avoid vendor lock-in. Optimize costs by running workloads on whichever provider is cheapest for a given task. Improve resilience by distributing across independent failure domains. On paper, it's a compelling case. In practice, the teams living with multi-cloud deployments often describe something closer to the opposite: doubled operational complexity, halved observability, and a category of reliability problems that only exist because there are two clouds instead of one. A team I worked closely with made the move to multi-cloud workloads on AWS and ML inference pipelines on GCP because of better GPU availability and pricing at the time and spent the next eight months dealing with a class of incident they hadn't anticipated: failures that were neither the application's fault nor either cloud provider's fault but existed in the boundary between them. Data transfer latency spikes that only appeared under load. Authentication token expiry edge cases that only trigger during cross-cloud calls. Network policy interactions that passed every pre-production test and failed in production at 3 am. The problems weren't hard individually. They were hard because the diagnostic tools for each cloud pointed inward, and the failure lived in the space neither tool was looking at. The Visibility Gap at the Boundary The first thing that breaks in a multi-cloud architecture is unified observability, and it breaks before you notice. Each cloud provider ships excellent native monitoring tooling: CloudWatch on AWS, Cloud Monitoring on GCP, and Azure Monitor on Azure. Each is well-integrated with that provider's services and reasonably effective at surfacing problems within its domain. None of them are designed to tell you what's happening in the gap between providers. When a request originates in AWS, crosses a private interconnect or the public internet to GCP, gets processed, and returns a response, the AWS tooling sees a latency value that includes the round-trip to GCP. The GCP tooling sees the processing time on its end. Neither surface shows you the network transit time in isolation, the connection establishment overhead, or the variance in that transit time under different load conditions. You're looking at the sum when you need to see the components. The fix requires pulling observability out of both providers' native tooling and into a neutral layer. In practice, that means OpenTelemetry instrumentation at every service boundary, every outbound cross-cloud call tagged with a span that captures the full round trip from the caller's perspective, and shipped to a backend that neither provider controls. This sounds straightforward, and technically it is. The organizational friction is real: teams used to relying on provider-native dashboards resist the overhead of running a separate observability stack, and the political question of which team owns it when it spans two provider accounts is never as simple as it should be. The Configuration Drift Problem Here's a failure mode that didn't appear in the architecture review because nobody thought to look for it: configuration drift between environments. In a single-cloud deployment, there's usually a meaningful concept of a canonical configuration in the infrastructure-as-code that defines the state of the environment, version-controlled, reviewed, and applied through a pipeline. In a multi-cloud deployment, you have two canonical configurations, maintained by teams with different tooling preferences and release cadences, and the interaction between them is rarely explicitly modeled. The incident that made this concrete: a security team rotation updated the cross-cloud service account credentials in the AWS Secrets Manager. The GCP side that consumed those credentials was on a different rotation schedule and a different team. For eleven days, the system ran on cached credentials. On the twelfth day, the cache expired during a peak traffic window. The GCP inference pipeline started returning authentication errors. The AWS team saw timeouts. The GCP team saw auth failures. Neither alert correlated the two. The on-call rotation spent ninety minutes establishing that the credentials were the issue before they could even start on the fix. The lesson is that cross-cloud dependencies, credentials, certificates, API contracts, and network allowlists need to be modeled and monitored as first-class infrastructure, not as bilateral agreements between teams. In practice, this means a dependency inventory: an explicit record of every configuration element in Cloud A that depends on something in Cloud B, with ownership, rotation schedules, and health checks defined for each. Python # Cross-cloud dependency health check (Python) # Run on a schedule to detect drift before it causes an incident import boto3, google.auth, requests from datetime import datetime, timezone def check_cross_cloud_credential(secret_name: str, gcp_endpoint: str) -> dict: """ Validates that the credential stored in AWS Secrets Manager is currently accepted by the GCP service that consumes it. """ # Fetch current credential from AWS sm = boto3.client('secretsmanager', region_name='us-east-1') secret = sm.get_secret_value(SecretId=secret_name) credential = secret['SecretString'] # Probe the GCP endpoint with the current credential resp = requests.get( gcp_endpoint + '/health', headers={'Authorization': f'Bearer {credential}'}, timeout=5 ) return { 'secret': secret_name, 'valid': resp.status_code == 200, 'checked_at': datetime.now(timezone.utc).isoformat(), 'status': resp.status_code, } # Alert if valid=False — don't wait for production traffic to find out The health assessment above is intentionally simple. The sophistication isn't in the code; it's in the discipline of running it, alerting on failure, and treating a failed credential probe as an incident rather than a routine maintenance item. Teams that implement this pattern catch rotation mismatches days before they would have surfaced as production failures. Network Reliability: The Assumptions That Break Single-cloud architectures inherit a reasonably reliable network fabric. Traffic within an availability zone is fast and consistent. Traffic across zones adds a predictable overhead. Traffic across regions adds more. The provider manages the underlying network, and it generally behaves within well-understood parameters. Multi-cloud breaks that model. Traffic between providers crosses networks that no single provider fully controls, whether through a private interconnect (AWS Direct Connect to GCP via a colocation facility) or the public internet, with all the variance that entails. The latency distribution changes character: instead of a tight distribution with a predictable tail, you receive a wider distribution with a heavier tail, and the tail gets heavier under load in ways that are harder to predict and harder to reproduce in testing. The implication for application design is significant. Services that communicate across cloud boundaries need timeout and retry policies calibrated to a different latency distribution than services communicating within a single cloud. A retry budget designed for intra-cloud latency will either be too aggressive, triggering retries on latency spikes that would have resolved naturally, or too conservative, giving up on requests that would have succeeded with a longer timeout. Getting this right requires measured data from the actual cross-cloud path under realistic load, not assumptions imported from single-cloud experiences. Python # Cross-cloud call with calibrated timeout and retry policy import httpx from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type # Timeouts calibrated from p99 measurements of actual cross-cloud path CROSS_CLOUD_CONNECT_TIMEOUT = 2.0 # seconds CROSS_CLOUD_READ_TIMEOUT = 8.0 # wider than intra-cloud to absorb tail latency @retry( stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=10), retry=retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError)) ) async def call_gcp_inference(payload: dict) -> dict: async with httpx.AsyncClient( timeout=httpx.Timeout( connect=CROSS_CLOUD_CONNECT_TIMEOUT, read=CROSS_CLOUD_READ_TIMEOUT ) ) as client: resp = await client.post( GCP_INFERENCE_ENDPOINT, json=payload, headers=get_auth_headers() ) resp.raise_for_status() return resp.json() The timeout values in the example above aren't guesses; they're derived from measuring actual cross-cloud latency at the 99th percentile under production load. The read timeout of 8 seconds would be far too generous for an intra-cloud call, where p99 might be under 200 milliseconds. For a cross-cloud inference call with real tail latency, it's calibrated to let legitimate slow requests complete while still protecting against genuine hangs. The difference matters: a timeout set at 500ms on this path would generate false failures on every traffic spike. The Cost of Distributed Incident Response One reliability challenge that rarely makes it into architecture discussions is the human cost of multi-cloud operations during incidents. When a single-cloud incident fires, there's usually a clear owner, the team responsible for that environment, and a reasonably well-understood set of tools and runbooks. When a multi-cloud incident fires at the boundary, the ownership is ambiguous by definition. In hindsight, the team I worked with should have defined cross-cloud incident ownership explicitly before going to production, not as a policy document but as a named on-call rotation and a defined escalation path. What they had instead was an informal understanding that "whoever is paged first figures it out," which works until a cross-cloud incident fires at 3 a.m. and the first person paged has deep AWS expertise and no GCP access. That situation happened twice before anyone fixed it. The solution was a dedicated cross-cloud on-call rotation, not a separate team, but a monthly rotation in which the designated engineer was expected to have current working knowledge of both environments and appropriate access to both. It also required shared runbooks stored outside either provider's tooling, because documents stored in an AWS wiki are inaccessible if the incident involves AWS authentication failures. What I'd Do Differently The single most important investment before going multi-cloud is measuring the cross-cloud network path under realistic load and calibrating every timeout, retry budget, and circuit breaker to those measurements rather than to assumptions. This is an unglamorous workload: testing a network path, collecting latency distributions, and deriving timeout values, and it's almost always skipped in favor of getting to production faster. The debt shows up as a category of reliability problem that's challenging to diagnose because it looks like application errors but behaves like network variance. I'd also resist the organizational tendency to treat multi-cloud as a flag day, a point at which the system transitions from single-cloud to multi-cloud. The reliability problems at the boundary need to be understood incrementally, with each cross-cloud dependency introduced deliberately and monitored explicitly before the next one is added. The teams that go from single-cloud to multi-cloud in one large migration tend to discover all their boundary problems simultaneously under production load. When might it be advisable to avoid a multi-cloud approach? The honest answer is most of the time. The cases where multi-cloud genuinely earns its operational cost are narrow: regulatory requirements that mandate geographic or provider separation, specific technical capabilities that a single provider doesn't offer at the required scale or price point, or acquisition scenarios where two companies on different clouds need to integrate without immediate migration. If the primary driver is "avoiding vendor lock-in" as a philosophical position, the operational cost almost never justifies it. A single cloud with well-designed abstractions at the application layer provides most of the portability benefits without most of the operational burdens. Key Takeaways The reliability problems in multi-cloud environments often live at the boundary between providers in the gap that neither provider's native tooling is designed to illuminate. Neutral observability infrastructure spanning both environments is a prerequisite, not an enhancement. Cross-cloud dependencies on credentials, certificates, and network allowlists need to be explicitly inventoried and health-checked on a schedule. Rotation mismatches and configuration drift between providers are a common source of incidents that look unrelated until you discover the shared dependency. Timeout and retry policies for cross-cloud calls must be calibrated using measured latency data on the actual path under load. Assumptions imported from a single cloud experience will be wrong in ways that generate either false failures or genuine availability problems. Define cross-cloud incident ownership before going to production. Ambiguous ownership at the boundary can lead to increased resolution time during critical moments. Conclusion Multi-cloud is frequently sold as a resilience strategy and often experienced as a complexity tax. The resilience argument is real but conditional: if the cross-cloud architecture is well-instrumented, the boundary dependencies are explicitly managed, and the failure modes at the boundary are understood and designed for, then distribution across providers does improve resilience. If those conditions aren't met, multi-cloud primarily adds new failure modes without reliably eliminating old ones. The teams that make multi-cloud work well tend to share one characteristic: they treated the inter-cloud boundary as a first-class engineering concern from the beginning with its observability, its dependency management, and its own incident ownership. Teams that treated it as a network detail that would take care of itself consistently found that it required attention. The question worth sitting with before committing to multi-cloud is "Are you solving a real problem that a single cloud with better architecture can't solve, or are you building for a failure scenario, vendor lock-in, or catastrophic provider outage that is less likely than the operational problems you're about to introduce?" Multi-cloud at the wrong time, for the wrong reasons, creates the very fragility it's supposed to prevent.

By Pruthvi Raj Seknametla

LocalStack and Terraform: A Clean Local AWS Setup Guide

Running AWS resources locally is a game-changer for engineering velocity, cost optimization, and developer autonomy. Traditionally, testing cloud infrastructure required deploying directly to a staging or sandbox AWS account. This workflow introduced painful friction points: waiting for slow cloud provisioning cycles, tracking down orphaned resources that inflate the monthly bill, and requiring a constant, high-speed internet connection. LocalStack solves this by emulating core AWS services, such as S3, SQS, DynamoDB, and other services directly on your local machine inside a Docker container. When paired with Terraform, you can safely write, plan, and apply infrastructure-as-code (IaC) configuration blueprints against this local simulator. This guide walks you through the definitive "happy path" for configuring LocalStack and Terraform, followed by a robust troubleshooting handbook for common architecture-specific and container networking errors. This allows you to provision these mock resources cleanly. This allows testing Terraform code with local resources without incurring real AWS costs, requiring internet connectivity, or dealing with slow cloud provisioning cycles. The Happy Path Setup Step 1: Setting the Stage: Launching LocalStack With Docker To get started, we need our local AWS cloud stack running inside a container. We will pull the official LocalStack image, set up our credentials, and spin up the container. First, pull the latest official image to your local machine: Before firing up the container, head over to the LocalStack Web App Dashboard to grab your personal access token (PAT). While LocalStack offers an open-source community edition, advanced features or specific emulated APIs may check for a valid token. Export this token into your shell environment so the container can authenticate and activate premium features on startup: Shell export LOCALSTACK_AUTH_TOKEN="ls-..." Now, launch the container. We need to map the primary edge gateway port (4566), which routes all inbound AWS API requests, along with the standard range of ports used by individual internal services (4510-4559). We also pass our token as an environment variable: Shell docker run --rm -it \ -p 4566:4566 \ -p 4510-4559:4510-4559 \ -e LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN \ localstack/localstack Keep an eye on your terminal logs. LocalStack will quickly validate your token, pull your license configuration, and initialize the mock runtimes. You will see a clear notification when the edge proxy is fully ready to handle incoming API requests. Step 2: The S3 Sanity Check: Talking to LocalStack Before configuring our automation toolchain, let's run a quick manual sanity check using the standard AWS CLI. Because LocalStack runs entirely on your machine, we must override the default cloud routing by passing a custom --endpoint-url pointing to our local edge proxy. To verify that LocalStack is running and reachable, create a local S3 bucket and upload a test file using the AWS CLI. 1. Create a Bucket Shell aws s3 \ mb s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 2. Upload an Object Create a dummy text file and copy it into your new mock bucket: Shell aws s3 \ cp /tmp/demo.txt s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 3. List Objects Verify the object is safely stored inside the mock container: Shell aws s3 \ ls s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 Step 3: Writing the Blueprint: Configuring the Terraform Provider Now let's automate things. To instruct Terraform to deploy resources to our local simulator instead of the real AWS cloud, we must customize the AWS provider block. We enforce dummy credentials, bypass cloud-only identity validations, and explicitly force all API endpoints to route directly to http://localhost:4566. Providers Configuration Create a file named providers.tf with the following content: Markdown terraform { backend "local" { path = "terraform.tfstate" } required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "us-east-1" access_key = "mock_access_key" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true s3_use_path_style = true # Redirect all endpoints to LocalStack's edge port endpoints { apigateway = "http://localhost:4566" cloudwatch = "http://localhost:4566" dynamodb = "http://localhost:4566" ec2 = "http://localhost:4566" iam = "http://localhost:4566" lambda = "http://localhost:4566" rds = "http://localhost:4566" s3 = "http://localhost:4566" secretsmanager = "http://localhost:4566" sns = "http://localhost:4566" sqs = "http://localhost:4566" ssm = "http://localhost:4566" sts = "http://localhost:4566" } } SQS Resource Definition Next, define the SQS queue we want to provision. Create a file named main.tf: Markdown resource "aws_sqs_queue" "local_queue" { name = "my-local-queue" delay_seconds = 90 max_message_size = 2048 message_retention_seconds = 86400 receive_wait_time_seconds = 10 } output "queue_url" { value = aws_sqs_queue.local_queue.id } Step 4: The Moment of Truth: Initializing and Applying Configuration With our configuration defined, we can run Terraform. Ensure you are executing a native binary that matches your host system architecture (such as a native darwin_arm64 binary if you are working on an Apple Silicon machine) to prevent execution overhead. Initialize Terraform First, initialize the working directory to download the AWS provider plugins: Generate and Review the Plan Next, generate and review an execution plan. The plan output will detail our local queue configuration without attempting to connect to actual AWS endpoints: Apply the Plan Apply the plan to deploy the queue directly to LocalStack. Upon completion, Terraform will write your state file locally and output your new mock SQS queue URL: Step 5: Taking It for a Spin: Sending and Receiving SQS Messages To confirm that our Terraform-provisioned SQS queue is fully operational, let's capture the output URL and push a real message through it using the AWS CLI. 1. Send a Message Shell export QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-local-queue" aws sqs send-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL \ --message-body "Hello from LocalStack SQS" 2. Receive the Message Shell aws sqs receive-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL { "Messages": [ { "MessageId": "1235d997-f60a-4e86-b248-aff3f5f41dde", "ReceiptHandle": "NzYxOThkMDAtMWJiOC00OGVhLTllMDEtNTU3ZTY3ZGQ5M2I4IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktbG9jYWwtcXVldWUgMTIzNWQ5OTctZjYwYS00ZTg2LWIyNDgtYWZmM2Y1ZjQxZGRlIDE3ODI4OTI2MzcuMjg2ODc1NQ==", "MD5OfBody": "88dc2faa42b899c03e12fd3ac96d714b", "Body": "Hello from LocalStack SQS" } ] } Your terminal will return a successful JSON payload containing your message body, confirmation IDs, and MD5 hashes, proving that the local loop is entirely complete. Event Verification in LocalStack Logs Checking the LocalStack container console confirms the queue creation, message send, and message fetch operations were handled successfully: Troubleshooting Guide Even on a happy path, local container networks and mixed system architectures can throw a wrench into your workflow. Here is how to fix the most common bottlenecks. The Apple Silicon (M1/M2/M3) Rosetta Loop Symptom: The LocalStack container crashes unexpectedly on startup, or loops endlessly while attempting to launch internal components like local Lambda runtimes, throwing qemu: uncaught target signal 11 errors. The Cause: LocalStack occasionally spins up secondary processes or helper binaries inside the container. If Docker Desktop is forced to emulate an x86_64 architecture via Virtualization frameworks on an ARM64 Apple Silicon chip, the emulation layer can break during heavy nested execution. The Fix: Ensure your Docker Desktop configuration has Use Virtualization framework enabled under Settings -> General, and turn on Rosetta for x86/amd64 emulation under the Features in Development tab. Alternatively, force Docker to fetch the native ARM64 container image by updating your execution command to include the specific platform flag: Shell docker run --platform linux/arm64 --rm -it -p 4566:4566 localstack/localstack "Port Already in Use" Symptom: Docker fails to bind ports, displaying an error message like: Bind for 0.0.0.0:4566 failed: port is already allocated. The Cause: A previous instance of LocalStack didn't shut down cleanly, or another local development tool is monopolizing port 4566. The Fix: Option 1: Check for lingering Docker containers Often, a container crashed or was backgrounded but didn't release the port. Find any container using 4566: Shell docker ps -a | grep 4566 If a container shows up, stop and remove it (replace <CONTAINER_ID> with your specific ID): Shell docker stop <CONTAINER_ID> docker rm <CONTAINER_ID> Option 2: Kill native background processes If Docker isn't holding the port, another process on your host machine is. You'll need to find its Process ID (PID) and force-quit it. Find the PID: Shell lsof -i :4566 Kill it (look for the number under the PID column): Shell kill -9 <PID> Wrapping Up Combining LocalStack and Terraform gives you a lightning-fast, zero-cost, offline sandbox for cloud infrastructure development. Once your environment is configured correctly with a valid personal access token, precise Docker port mappings, and native toolchains matched to your host CPU, you can prototype, test, and tear down AWS configurations in seconds. No more waiting for slow cloud deployments or tracking down orphaned cloud resources. Happy local provisioning!

By Ammar Ekbote

Cloud Architecture

DZone's Featured Cloud Architecture Resources

Monthly Top Cloud Architecture Experts

The Latest Cloud Architecture Topics