Kubernetes Rolling Updates Are Lying to You — and Your Users Are Paying for It

There is a moment during a rolling update when everything looks fine.

The old pods are draining. The new pods are coming up. The readiness probes are passing. The deployment controller is happy. You watch the replica counts shift in your terminal and think this is working exactly as designed.

Then your on-call phone buzzes.

In a 5G signaling deployment I worked on, this happened during what should have been a routine update. Every infrastructure signal was green. The Kubernetes event log was clean. The new pods passed their readiness checks within a few seconds of starting up.

Session establishment failures jumped 18%.

The pods were running. They just weren’t ready. And our probes had no way to tell the difference.

The Problem With HTTP Probes

Readiness probes are one of Kubernetes’ better ideas. Before they existed, you’d route traffic to a pod and hope the application had finished initializing. Probes let you say: Don't send traffic until the pod tells you it’s ready.

The default implementation looks something like this:

readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

Kubernetes hits /healthz. Gets a 200. Marks the pod ready. Routes traffic.

For a stateless web service, this is fine. The HTTP endpoint responding and the application being ready to handle requests are essentially the same thing. Load a config file, spin up a thread pool, and you’re good.

The assumption breaks the moment your application has to integrate with something before it can serve real traffic.

What Integration Actually Means

In telecom environments, network functions don’t just start — they register. A SIP proxy needs to complete handshakes with upstream session border controllers. A Diameter node needs to establish peer connections with authentication servers. A GTP function needs to synchronize state with its neighbors.

These aren’t instantaneous. Depending on the environment, full registration can take anywhere from ten seconds to over a minute after the process has started.

During that window, here’s what happens:

1. Pod starts.
2. /healthz returns 200 (process is up).
3. Readiness probe passes.
4. Kubernetes routes traffic to the pod.
5. Pod receives a request that requires upstream registration.
6. Registration isn’t complete yet.
7. Request fails.

From Kubernetes’ perspective, everything worked. The probe passed. Traffic was routed. From your user’s perspective, their session failed.

The problem isn’t telecom-specific. Any of these patterns hit the same issue:

• gRPC services that need to warm connection pools before handling load
• Message queue consumers that need to establish broker connections and claim partitions
• Database-backed services that need to run migrations or seed connection pools
• Service mesh participants that need sidecar handshakes before traffic can flow

The HTTP probe can only tell you the first thing. It says nothing about the second.

What the Incident Actually Looked Like

In the 5G deployment, the readiness probe was checking a lightweight internal endpoint that returned 200 as soon as the main process thread started. Two seconds after startup, at most.

What it wasn’t checking: whether the pod had completed SIP registration with the upstream proxy and re-established session state with the session management function.

That took closer to thirty seconds. Sometimes longer under load.

So for that thirty-second window during each pod’s rollout, we were routing live signaling traffic to pods that were alive but unregistered. Requests that hit those pods during registration would fail. The failures were silent—no crash, no obvious error, just session establishment attempts that didn’t complete.

The 18 percent failure rate during the update window wasn’t from a bug in the new code. It was from routing traffic to pods before they were actually ready to handle it.

The first time it happened, it resolved on its own, and we wrote it off as a fluke. The second time, it happened during a peak traffic period, and the impact was hard to ignore. The third time never happened because we’d fixed the probe by then.

How to Fix It

The fix is to replace the HTTP probe with something that checks actual readiness — not process readiness, but integration readiness.

For the signaling service, that meant an exec probe that called a script checking registration state:

readinessProbe:
exec:
command:
- /bin/sh
     - -c
     - "check-registration-status.sh && exit 0 || exit 1"
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 18

The script did two things: it verified that the SIP registration with the upstream proxy had been completed and confirmed that the session management peer connection was established. Until both returned healthy, the pod stayed out of rotation.

The tuning matters:

• initialDelaySeconds: 15 – gives registration time to begin before Kubernetes starts checking
• periodSeconds: 5 – checks frequently enough to catch completion quickly
• failureThreshold: 18 – allows up to 90 seconds for registration to complete before Kubernetes gives up and restarts the pod

For non-telecom workloads, the same pattern applies. A gRPC service might check whether its connection pool is warmed:

readinessProbe:
exec:
command:
     - /bin/sh
     - -c
     - "grpc_health_probe -addr=:9090 && pool-ready-check.sh"
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 12

The principle is always the same: don’t ask Kubernetes to decide your application is ready based on evidence that doesn’t reflect what ready actually means for your workload.

One More Thing Worth Knowing

If you fix your readiness probe and your rolling updates get slower, that’s correct behavior—not a regression.

Before the fix, pods were being marked ready before they actually were. Updates appeared fast because Kubernetes was moving quickly through the rollout. After the fix, updates take longer because Kubernetes now waits for real readiness before routing traffic.

The slowdown is the probe working. Your update velocity was an illusion.

The Short Version

If your readiness probe is an HTTP check against a /healthz endpoint, ask yourself what that endpoint actually knows about your application’s readiness state.

If the answer is "it knows the process started," you have a process-up probe, not a readiness probe.

The distinction matters only when something goes wrong during a rollout — exactly when you need your probes to be accurate.

Related Research

The production autoscaling findings referenced in this article informed peer-reviewed research on workload-aware Kubernetes autoscaling for telecom infrastructure, published in IEEE Xplore: https://ieeexplore.ieee.org/document/11501289