Zone-Aware Routing in Kubernetes: Reducing Latency, Improving Resilience, and Lowering Cloud Costs

This guide explains zone-aware routing from a Kubernetes-first point of view.

It covers:

• why zones matter in cloud platforms
• which topology labels Kubernetes places on nodes
• how Kubernetes first tried to solve locality through Service
• what gaps remained after those Service-based features
• how Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation

Why Zones Matter

In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost.

That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See:

• AWS Architecture Blog: Overview of Data Transfer Costs for Common Architectures
• Amazon EC2 pricing: Data Transfer

This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones.

The Topology Information Kubernetes Already Has

Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes.

The two most important well-known labels are:

• topology.kubernetes.io/region
• topology.kubernetes.io/zone

According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense.

Reference:

• Kubernetes well-known labels: topology.kubernetes.io/zone

In managed clusters, these labels are commonly present by default.

Here is the kind of node data Kubernetes typically exposes:

apiVersion: v1
kind: Node
metadata:
  name: ip-10-0-12-34.ec2.internal
  labels:
    kubernetes.io/hostname: ip-10-0-12-34.ec2.internal
    topology.kubernetes.io/region: us-east-1
    topology.kubernetes.io/zone: us-east-1a

That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing.

The Original Service Model

The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods.

At the beginning, the model was simple:

• a Service selected a set of Pods
• kube-proxy programmed forwarding rules
• traffic could be sent to any healthy endpoint behind the Service

That was excellent for reachability and abstraction, but it had no built-in notion of zone locality.

The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a.

Kubernetes' First Attempts to Improve Locality Through Services

Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works.

Internal Traffic Policy

One early mechanism was internalTrafficPolicy: Local.

This tells kube-proxy to use only node-local endpoints for cluster-internal traffic.

Example:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
  internalTrafficPolicy: Local

Reference:

• Kubernetes Service Internal Traffic Policy

This helps with node locality, but it is not zone-aware routing.

Its limitations are important:

• it is node-local, not zone-local
• if a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspective
• it is too strict for many multi-zone workloads that want zonal preference, not node affinity

So this was useful, but it did not really solve multi-zone locality.

Topology Aware Routing With Services

Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing.

This works through two components:

1. The EndpointSlice controller looks at endpoint and node topology.
2. kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone.

Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation:

apiVersion: v1
kind: Service
metadata:
  name: zone-aware-backend
  annotations:
    service.kubernetes.io/topology-mode: Auto
spec:
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 8080

Conceptually, the flow looks like this:

This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users.

Traffic Distribution Preferences

Kubernetes later added trafficDistribution as a clearer way to express routing preferences.

In current Kubernetes documentation, the relevant zone-level preference is:

• PreferSameZone

The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up.

Example:

apiVersion: v1
kind: Service
metadata:
  name: zone-aware-backend
spec:
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 8080
  trafficDistribution: PreferSameZone

Reference:

• Kubernetes Service trafficDistribution

This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee.

In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution.

What Gap Remained After Service-Based Locality

Kubernetes Services improved a lot, but they still left several gaps.

The Behavior Is Best Effort

Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing.

Examples include:

• too few endpoints
• impossible balanced allocation
• missing topology labels on one or more nodes
• missing hints for one or more endpoints
• no hinted endpoint for the local zone

That is correct for safety, but it means the behavior is heuristic and conditional.

It Assumes a Certain Traffic Shape

Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone.

If most traffic originates from one zone, local subsets can overload while the global service still looks healthy.

It Is Scoped to the Service Datapath

This is the most important architectural gap.

Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane.

In particular, they do not by themselves define:

• how an L7 gateway proxy should understand its own zone
• how an Envoy-based gateway should configure locality-aware upstream load balancing
• how a gateway controller should express stricter local preference versus simple best-effort locality
• how policy should attach to particular routes, gateways, or backends

That left room for Gateway API implementations to expose richer locality controls.

Why Gateway API Implementations Stepped In

Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane.

That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design.

Reference:

• Gateway API overview

This is where zone-aware routing became more explicit at the gateway layer.

Instead of relying only on kube-proxy's Service behavior, gateway implementations can:

• understand the proxy's own locality
• read backend endpoint locality
• configure the underlying proxy's load balancer directly
• expose locality policies as route or backend-attached configuration

Example of How Envoy Gateway Addresses the Gap

Envoy Gateway supports two paths:

1. Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistribution
2. Configuring zone awareness directly through BackendTrafficPolicy

Reference:

• Envoy Gateway zone-aware routing
• Envoy zone-aware routing

Example BackendTrafficPolicy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: zone-aware-routing
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: zone-aware-routing
  loadBalancer:
    type: RoundRobin
    zoneAware:
      preferLocal:
        minEndpointsThreshold: 1
        force:
          minEndpointsInZoneThreshold: 1

That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing.

Example of How kgateway Addresses the Gap

kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics.

At a high level, kgateway combines:

• Gateway proxy locality configuration
• Backend-attached load-balancing policy
• Native Envoy locality-aware upstream load balancing
• Endpoint locality metadata that Envoy can use directly

Architectural Summary

The progression looks like this:

1. Kubernetes Service solved stable discovery and reachability.
2. internalTrafficPolicy improved node-local routing, but not zonal routing.
3. Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.
4. Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves.

Practical Takeaways

• Kubernetes already provides the topology metadata needed for zone-aware decisions.
• Service-native locality is useful, but it is heuristic and scoped to the Service datapath.
• Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.
• Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane.

Where Zone-Aware Routing Matters in Practice

Zone-aware routing usually becomes worth the added operational attention when one or both of these are true:

• The workload has a tight latency budget, especially at p95 or p99
• The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material

Common examples include:

• Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delay
• Financial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latency
• Large SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly cost
• AI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical

Worked Example: Multiplayer Gaming Backend

Suppose a regional game API runs gateway proxies and backend pods in three zones.

Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c.

Assume the following:

• 25,000 requests per second reach the player-state service from zone-a
• the combined request and response payload is about 40 KiB per call
• cross-zone traffic is billed at a representative $0.01 per GB
• without zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c

Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths.

That means the traffic volume from zone-a to the player-state service is about:

• 25,000 x 40 KiB per second, or roughly 1 GB/s total
• if two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone traffic
• over a 30-day month, that is about 1.7 million GB
• at $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path

That is the cost side. The latency side can matter even more for the player experience.

If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect:

• matchmaking responsiveness
• session join time
• the smoothness of player state or presence updates
• how stable the system feels during traffic spikes and retries

This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control.

Worked Example: Large SaaS Control Plane

The same logic applies outside gaming.

Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline.

Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways:

• it removes avoidable cross-zone traffic from the steady-state hot path
• it reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance

For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment.

Conclusion

Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata.

The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must.

Further Reading

• Kubernetes Service
• Kubernetes Topology Aware Routing
• Kubernetes Service Internal Traffic Policy
• Kubernetes well-known topology labels
• Gateway API overview
• AWS Architecture Blog: Data transfer costs