Docker Containers Don’t Know Your Model Is Still Loading

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we'd agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas. What it didn't account for is that each replica needed almost three minutes just to pull a 14GB checkpoint and warm up CUDA kernels before it could answer a single request. The load balancer, seeing new pods report as running, immediately started routing traffic to them. For three minutes, a chunk of our users got 504s while perfectly healthy-looking pods sat there loading a model into memory.

Nobody on the infra side had touched Docker that day. The incident wasn't a Docker bug. We assumed that container orchestration designed for web services would function the same way for processes that take minutes to become useful, rather than those that operate in milliseconds.

Why LLM Containers Break the Usual Assumptions

Packaging an LLM serving stack in Docker still makes sense for the same reason it always has; CUDA versions, driver compatibility, and Python ABI mismatches are miserable to manage across a fleet without a frozen artifact. 

But an LLM container carries baggage that a typical inference service doesn't. The weights are tens of gigabytes, not a few hundred megabytes. GPU memory is a single shared pool that one greedy container can quietly exhaust for everyone else on the box. And “ready” doesn't mean “process started”; it means the model is resident in VRAM and the CUDA graph is warmed, which can take minutes on a cold node pulling weights from object storage over the network.

The Mistakes, in Order

Our first version baked the model weights directly into the image, because it felt simpler: one artifact, one pull, done. In practice, it meant a 16GB image, painfully slow CI pushes, and a registry bill nobody wanted to look at. Worse, every time we bumped into a new fine-tuned checkpoint, we rebuilt and repushed the entire layer regardless of caching, because the COPY step touching gigabytes of weight files invalidates everything below it. Unlike a typical ML inference image, there's no meaningful caching win here at all; the layer is simply too big to ever be a cache hit across versions. We moved weights out to a mounted volume, fetched at container start from object storage, and never looked back.

Second mistake, and this one actually cost us a production incident: we ran the container with Docker's default shared memory size. vLLM, which we used for serving, spins up worker processes that talk to each other over shared memory even on a single GPU. With the default 64MB /dev/shm, those workers would crash with cryptic bus errors under any real concurrency. The fix was almost embarrassingly small:

docker run --gpus all \
  --shm-size=2g \
  -e MODEL=mistralai/Mistral-7B-Instruct-v0.2 \
  -e GPU_MEMORY_UTILIZATION=0.85 \
  -e MAX_MODEL_LEN=8192 \
  -p 8000:8000 \
  llm-serve:latest

The third mistake was more subtle and took longer to diagnose. vLLM's continuous batching reserves a large slice of GPU memory upfront for the KV cache, controlled by gpu_memory_utilization. We'd set that fraction high to maximize throughput, then bin-packed two replicas onto the same GPU to save cost. Under normal traffic, fine. During a burst of unusually long-context requests, such as someone summarizing a 6,000-word document instead of a tweet, the KV cache for that single batch ballooned, causing the container to run out of memory (OOM) mid-generation and taking down every other in-flight request in the same batch. This failure mode is more severe than a typical web service OOM because it not only drops the new request but also terminates queries that were already halfway through generating answers for paying customers.

What We Actually Changed

The readiness adjustment turned out to matter more than any Docker flag. We split liveness from readiness: liveness just checks that the process hasn't died; readiness fires a real, tiny generation request through the local API and only flips to healthy once that round trip succeeds. That alone killed the cold-start routing problem because the load balancer stopped trusting a merely alive process.

We also gave up on bin-packing two replicas per GPU. In hindsight, treating GPU memory like it's as elastic as CPU or RAM was the actual root cause, not any single Docker setting. We implemented a model that uses one GPU, sets a conservative memory utilization ceiling, and enforces a request-level token limit at the proxy in front of the container, rather than inside it, because it is too late to make adjustments once the batch is already running.

On the orchestration side, we stopped trying to scale-to-zero or scale aggressively off CPU-style metrics. Scale-to-zero is effective for web apps but doesn’t fit GPU-bound LLM serving, where cold starts can outlast traffic spikes. We kept a warm floor of replicas sized to baseline traffic and let a request queue absorb bursts instead of expecting new pods to materialize in time. It's less elegant than the autoscaling story everyone likes to tell, and it costs more in idle GPU time, but it's honest about what the hardware can actually do.

What We Rejected, and Why

We seriously considered dropping self-hosting altogether and routing through a managed inference API. For a side project, that's probably the right call — less to own, no GPU bin-packing headaches. We rejected it due to data residency requirements that prohibited sending raw text to a third party, and at our volume, managed pricing would quickly exceed our GPU costs. We also looked at Ray Serve and Triton early on, and they solve some of the issues more natively, but the team's Docker and Kubernetes muscle memory was strong enough that rebuilding on a new serving framework felt like trading one set of unknowns for another, at least for the first version.

Key Takeaways

• Never bake multi-gigabyte model weights into the image — there's no caching benefit at that size, only slower pushes and bigger registry bills.
• Set shared memory explicitly; vLLM and similar multiprocess servers will fail under load with Docker's tiny default.
• Treat GPU memory utilization conservatively and avoid bin-packing replicas onto a single GPU unless you can guarantee a strict ceiling per container.
• Build a readiness assessment that performs a real generation, not just a process check; cold model loading will otherwise receive routed live traffic.
• Don't expect autoscaling to save you on cold-start timescales measured in minutes; a warm floor plus a queue is more honest than reactive scaling.

Closing Thought

None of these issues was really a Docker failure; the container did exactly what we told it to do. The failure was treating a multi-gigabyte, GPU-bound, slow-to-warm process like it was just another stateless web container that happens to need a GPU flag. I suspect that many teams will learn this lesson in the same way we did, during an incident on a Friday afternoon. Is it the right move to keep stretching Docker and Kubernetes to fit LLM serving, or is this the workload that finally pushes most teams toward purpose-built serving layers?