Testing, Deployment, and Maintenance

Sponsored By: NutanixThe following is sponsored content. It may not reflect the views of our editorial staff. The Kubernetes scaling problem nobody talks about Enterprise platform teams encounter the same pattern repeatedly: a Kubernetes platform works well enough that nobody wants to change it. This happens gradually as teams make reasonable technology choices: selecting different ingress controllers, secrets management tools, CD platforms, or observability software. Individually, none of these decisions is a problem. Months later, however, they’ve created a Kubernetes environment that only a handful of people understand. As soon as that one person gets sick or leaves the company, maintaining or improving the platform becomes much more difficult. Mark Dastmalchi-Round, a Solutions Architect at Nutanix with decades of experience in platform engineering, describes the pattern in blunt terms: “Configuration drift, exacerbated by the fact that multicloud is increasingly becoming the new reality.” Over time, that drift compounds. Companies get acquired, technology merges, and silos form. Suddenly, organizations are managing clusters that look nothing alike and are often held together by institutional knowledge. As a solution, proprietary overlays have sought to address these issues, with mixed results. They tend to reduce overall surface area (fewer choices lead to fewer points of divergence), but often at a cost to portability and extensibility, which is what made Kubernetes so attractive in the first place. A more durable approach is to build on Kubernetes-native primitives, adding governance and operational consistency without replacing the workflows teams already use. The remainder of this article will demonstrate what that looks like in practice. What an open platform actually means in enterprise Kubernetes “Open platform” is a common phrase in the Kubernetes ecosystem, but it’s worth defining what that term actually means in practice. Dastmalchi-Round defines an open platform as one that “exposes industry-standard APIs and, where possible, uses pure upstream open-source projects.” The distinction isn't whether the platform is open source. It's whether it relies on Kubernetes-native APIs and tooling or introduces proprietary CRDs, workflows, and CLIs that make migration difficult. As he notes, "You can still get lock-in with open source, because if it is only one vendor's solution and they layer all of their stuff on top of standard tooling, you are now dependent on their abstractions." The difference is easier to see when comparing an open platform with a proprietary overlay. Comparing Open Kubernetes Platforms and Proprietary Overlays Dimension Open Platform (NKP) Proprietary Overlay Core CRDs Standard upstream (Cluster API, FluxCD, Helm) Vendor-specific, migration cost is high GitOps engine FluxCD (CNCF project) Proprietary sync engine App packaging Helm + OCI (industry standard) Custom catalog format Monitoring stack Pure upstream CNCF (Prometheus, Grafana) Wrapped / vendor-branded Exit cost Clusters survive platform removal Manifests tied to platform APIs Third-party tooling Works if it runs on Kubernetes Requires certified integration Nutanix Kubernetes Platform (NKP) applies these principles by building on upstream Kubernetes components rather than replacing them. As Dastmalchi-Round puts it, the real test is what survives if you remove the platform. "With NKP, the clusters are pure upstream Kubernetes,” says Dastmalchi-Round. “The monitoring stack is pure upstream CNCF projects. GitOps is provided by FluxCD. Your manifests and charts are standard Helm." In other words, the operational tooling may change, but the underlying applications and deployment artifacts remain portable. Raw manifests to managed artifacts: Helm and OCI packaging in NKP Most enterprise teams start with a collection of Kubernetes YAML manifests that work for a single application or environment. While those manifests are typically stored in version control, they aren't easily reusable across environments, self-service for other teams, or packaged in a way that supports consistent versioning and rollback. Helm addresses those limitations by packaging manifests into versioned, parameterized charts. For existing applications, the process typically starts by converting Kubernetes manifests into a standard Helm chart, either manually or with tools such as Helmify. The result is a familiar Helm project structure built around Chart.yaml, parameterized templates, and a values.yaml file, giving teams a reusable deployment artifact instead of a collection of static manifests. Deployment-specific settings, such as image tags, replica counts, and resource limits, move into a values.yaml file, while the underlying templates remain unchanged. Those deployment-specific settings are defined in the chart's values.yaml file. For example: # values.yaml — the self-service interface for application teams replicaCount: 2 image: repository: registry.example.com/myapp tag: "2.1.0" pullPolicy: IfNotPresent resources: limits: cpu: 500m memory: 256Mi requests: cpu: 250m memory: 128Mi ingress: enabled: true host: myapp.internal.example.com annotations: kubernetes.io/ingress.class: "traefik" serviceAccount: create: true name: "myapp-sa" Versioning makes deployments reproducible across environments while providing a clear history of releases. Teams can promote the same chart through development, staging, and production with confidence, then roll back to a previous version if needed. OCI registries address the next challenge: distributing and versioning those charts. Instead of relying on a separate chart repository, teams can store Helm charts alongside container images as immutable, versioned artifacts. Because chart versions can't be overwritten, deployments are reproducible and easier to audit. The approach also fits existing registry workflows. Organizations using Harbor, Amazon ECR, or similar registries can manage container images and Helm charts in the same place, using the same authentication, access controls, and security policies. For example: # Package the chart locally helm package ./myapp --version 2.3.0 # Authenticate to the OCI registry (same registry as your container images) helm registry login registry.example.com \ --username $REGISTRY_USER \ --password $REGISTRY_PASSWORD # Push is stored as an OCI artifact alongside container images helm push myapp-2.3.0.tgz oci://registry.example.com/charts # Any team can pull without touching the source repo helm pull oci://registry.example.com/charts/myapp --version 2.1.0 # Inspect the chart before deploying helm show values oci://registry.example.com/charts/myapp --version 2.1.0 The goal of packaging is to create a self-service deployment model. Once packaged, Helm charts are registered with the NKP catalog, where they appear alongside built-in platform applications as versioned deployment artifacts. Application teams can deploy them by configuring only the settings that vary between environments, while platform teams focus on maintaining reusable application catalogs instead of manually managing deployments. FluxCD deployments, overrides, and upgrades Once Helm charts are stored in an OCI registry, FluxCD keeps deployed clusters aligned with the desired state defined in Git. It continuously reconciles each cluster against that source of truth, automatically correcting configuration drift. In multi-cluster environments, each cluster follows the same reconciliation process using its own configuration. NKP's FluxCD implementation centers on two resources: HelmRepository, which points to the OCI registry, and HelmRelease, which specifies the chart version, configuration values, and target namespace. # Source: points FluxCD at your OCI chart registry apiVersion: source.toolkit.fluxcd.io/v1beta3 kind: HelmRepository metadata: name: internal-charts namespace: flux-system spec: type: oci url: oci://registry.example.com/charts interval: 5m # poll for new chart versions every 5 minutes # Release: declares desired state for a specific deployment apiVersion: helm.toolkit.fluxcd.io/v2beta3 kind: HelmRelease metadata: name: myapp-production namespace: production spec: interval: 10m chart: spec: chart: myapp version: "2.3.0" sourceRef: kind: HelmRepository name: internal-charts namespace: flux-system values: replicaCount: 3 resources: limits: cpu: 1000m memory: 512Mi ingress: host: myapp.prod.example.com Although teams interact with NKP through its web interface, those actions are ultimately represented as standard Kubernetes resources. Configuration changes become declarative objects that FluxCD reconciles like any other GitOps workflow, making the deployment model transparent and compatible with standard Kubernetes tooling without relying on proprietary deployment workflows. Teams typically promote the same chart version from development to staging and production while applying environment-specific overrides through HelmRelease values rather than modifying the chart itself. Promotion becomes a Git commit instead of a manual deployment, with FluxCD automatically reconciling and applying the change. FluxCD also provides continuous drift detection. If someone manually changes a resource in the cluster, FluxCD restores it to the state defined in Git during the next reconciliation cycle. Rolling back a deployment is simply a Git revert, with Git history providing a complete audit trail of configuration changes. How to integrate third-party tools without losing openness Enterprise platform teams are often asked to integrate tools such as vulnerability scanners, cost management dashboards, and application performance monitoring (APM) platforms. The tools themselves aren't the problem. The problem is managing each one through a separate deployment and maintenance process, increasing operational complexity over time. NKP addresses this by treating third-party software like any other platform application. Whether it's an upstream open-source project or a commercial product distributed as a Helm chart, it follows the same Helm-over-OCI packaging model and is deployed and managed through FluxCD. The outcome is a consistent deployment and lifecycle workflow across both first- and third-party applications. For example, an upstream Helm chart such as Redis can be published to the NKP catalog and managed through the same deployment workflow as a first-party application, avoiding the need for a separate integration process. Because this approach relies on standard Kubernetes resources, Helm charts, Git, and Kubernetes RBAC, those workloads remain portable across platforms. As Dastmalchi-Round summarizes, "If it works on Kubernetes, it will work on NKP." Dastmalchi-Round notes that the biggest integration challenges typically come from tools that rely on rigid deployment models, particularly older operator-based packages that expose little configuration. "A few years ago, there was a trend of people overusing the operator pattern for packaging applications," he says. "Operators have their uses, but when they became the distribution artifact, they often resulted in big, opaque blobs running in your cluster. If they didn't do exactly what you needed, you were out of luck." As more vendors have adopted Helm-based packaging, those limitations have become less common. Examples of Third-Party Tool Integrations in NKP Integration Type Packaging Model Configuration Upgrade Path NKP Catalog Security scanner (e.g., Trivy) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Custom Grafana dashboard Helm chart + ConfigMap Dashboard JSON in Git Chart version update Yes Cost management (e.g., OpenCost) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Service mesh (e.g. Istio) Helm chart via OCI IstioOperator CRDs in Git Controlled chart upgrade Yes Legacy operator-only tool Operator bundle Operator-managed CRDs Operator version update Requires evaluation In practice, the less a tool depends on proprietary deployment mechanisms, the easier it is to integrate, manage, and move between Kubernetes platforms. Conclusion: the platform that gets out of the way NKP doesn't replace Kubernetes workflows—it builds on them. Helm packages applications, OCI registries distribute them, Git defines the desired state, and FluxCD keeps deployments in sync. Instead of introducing proprietary workflows, NKP brings these familiar tools together with the governance, lifecycle management, and self-service capabilities required for enterprise-scale operations. It standardizes these workflows across any environment, including public clouds, on-premises, and edge locations. For enterprise teams, the value lies in achieving consistency without sacrificing portability. As Dastmalchi-Round notes, the question isn't whether lock-in exists, but how costly it is to leave. By relying on upstream Kubernetes components, Helm charts, and GitOps workflows, organizations retain portable applications and deployment artifacts even if they choose a different platform in the future. In the end, an open platform shouldn’t be defined by its licensing model. It should be defined by how much of your platform remains yours if you decide to move on.

An autoscaling policy can be wrong for months without a single error firing. It isn't built to fail loudly; it's built to keep response times steady, and it'll keep doing exactly that even while making the worst possible call for a GPU-bound job. The mismatch hides in plain sight because nothing looks broken. It stops doing its job without ever raising an alarm, and the first sign usually isn't an alert but a cost report or a training job stuck in a queue. Here's a fairly standard Kubernetes Horizontal Pod Autoscaler config:  YAML apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler spec:   minReplicas: 2   maxReplicas: 10   metrics:     - type: Resource       resource:         name: cpu         target:           averageUtilization: 70 For a stateless web service, this is close to perfect. A pod gets added, utilization dips, another request comes in, utilization climbs again. The whole loop runs slowly enough for the cooldown window to work exactly as intended: plenty of time to observe and react. A training job doesn't move like that. It sits at zero for two days, then needs ten GPUs immediately, then drops back to zero the second the job finishes. CPU utilization barely registers the change, because CPU was never the constraint to begin with. So the autoscaler, watching the wrong metric entirely, does nothing useful. Triggerworks well forbreak down for CPU utilization  Steady, request-driven traffic  GPU-bound training jobs  Queue depth / GPU utilization  Bursty, batch-oriented AI workloads  Legacy web services  Autoscaling wasn't wrong here, exactly. It kept solving the problem it was built for, one that had already stopped being the problem sitting in front of it.   The GPUs Were Right. The Data Never Arrived. There's a second version of this same trap that's easier to miss. Even with the right trigger metric, GPUs can sit idle waiting on data they can't ingest fast enough. Storage throughput and network bandwidth that worked for traditional applications can become bottlenecks when training jobs move terabytes at scale. An idle GPU waiting on data still costs money, but it rarely appears as an autoscaling problem. When the Infrastructure Looks Fine, and the Model Doesn't  Once a model is live and behaving, the infrastructure looks fine. CPU healthy, memory healthy, no alerts firing. Somewhere down the line, though, a flagging rate or an approval rate starts drifting, and nothing in the infrastructure layer notices. Prometheus, Grafana, and OpenTelemetry confirm the service is healthy. None of them tell you whether the model's decisions are still good. That's the split most teams don't plan for going in: infrastructure health and model health are two completely different signals, and only one of them shows up in the tools most cloud teams already trust. Data Quality Still Determines AI Performance  Trace either failure back far enough and it rarely ends at the model. McKinsey's research, AI Data Readiness: The Key to Scaling Impact, found more than two-thirds of high-performing organizations name data, not model selection, not compute, as the real constraint on scaling AI. It shows up constantly in practice: a CRM system, a billing platform, and a support desk defining the same customer three different ways. MLOps tooling can track model versions and deployments, but it cannot fix unreliable data underneath the model. Versioning is not the same as fixing. Models rarely fail because they cannot process data. They fail because they process unreliable data with the same confidence as accurate data. The Regulator's Question Has No Engineering Answer  Eventually, someone always asks the harder question, and it usually isn't an engineer who asks it. A lending platform turns an application down, and the applicant pushes back. A regulator wants to know exactly how that decision got made. Without an audit trail connecting that specific outcome back to the specific inputs the model saw, there's no real answer to give, regardless of how accurate the model has been on average. That almost never blocks a proof of concept. It blocks production, on a timeline nobody controls.  Cloud Placement Becomes a Production Decision for AI Workloads  There's a fourth complication sitting underneath all of this, one that surfaces even later. Where a workload actually runs stops being a footnote once AI enters the picture. AI workloads introduce new constraints around hardware availability, latency, cost, and regulatory requirements. Some workloads have to stay within a specific country's borders for regulatory reasons. Others only perform well on hardware a specific provider happens to offer. A team standardized on one cloud for everything else discovers, usually the hard way, that AI doesn't respect that standardization.  The challenge is no longer choosing one cloud provider. It is deciding where each workload can run effectively while balancing performance, cost, and compliance. What Gets Built Before the Next Incident, Not After None of these four problems — autoscaling, observability, data, governance, and placement — show up in a pilot. That's exactly why they're expensive.  The autoscaling policy either scales for GPU load or it doesn't. The observability stack either catches a model quietly getting worse, or it only notices when a server goes down. The data feeding the model is either governed enough to trust or it isn't. An audit trail either exists before the first real customer sees an output, or it gets built after a regulator asks for one. Someone has either mapped out where each workload needs to run, or that decision is still riding on wherever the last project happened to land.  Right now, real value is going to the teams that got the boring infrastructure work right, not the teams with the fanciest model. 

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs

Over the past year, I've been rebuilding parts of an incident response stack for a client, and the biggest surprise wasn't the AI features themselves. It was how much of the underlying workflow had to change to make those features useful. You can't just bolt an LLM onto a 2015-era ticketing tool and call it AIOps. The queue structure, the alert taxonomy, even the way runbooks are written all need to change. I've written before about the agent side of this shift, in AI Agent Architectures: Patterns, Applications, and Implementation Guide and Observability and DevTool Platforms for AI Agents. This two-part series is the other side of that coin: what happens when you point those same agent patterns at your own production systems instead of at somebody else's AI application. Same reasoning loop, different target. This first part covers incident management specifically, including a category I skipped in my earlier tool roundups: dedicated "AI SRE" agents like Traversal, Resolve.ai, and Cleric, which behave differently from the AIOps platforms most of us grew up with. Part 2 goes past incidents into ITOps, chaos engineering, SLO management, on-call toil, and the rest of what fills an SRE's week. A note on the numbers below: vendor-reported accuracy and MTTR figures in this space move fast and come from the vendors themselves. I've flagged those clearly rather than presenting them as independently verified benchmarks. Where SRE Pain Actually Lives Before getting into tools, it helps to remember what SREs spend their time on. Most postmortems I read over the years have the same three complaints: Too many alerts, not enough signalCorrelating five different dashboards to find one root causeWriting the same postmortem summary for the fourth time this quarter None of these are new problems. What's new is that large language models are actually decent at the second and third ones, if you feed them clean data, and a newer crop of agents is starting to chip away at the first one too. The Traditional Incident Pipeline Here's roughly what an incident used to look like before AI got involved, at most mid-size shops I've worked with: Traditional incident pipeline Every arrow in that diagram is a human doing manual correlation work. That's fine when you have ten services. It falls apart at three hundred, and it's part of why I keep coming back to the point I made in Infrastructure as Code: How Automation Evolved to Power AI Workloads: scale problems in ops rarely get solved by hiring more people to stare at more dashboards. Where AI Fits Into the Pipeline Today The shift isn't "AI replaces the engineer." It's AI collapsing steps B through F into something closer to a single triage step, with the engineer reviewing a proposed root cause instead of hunting for one from scratch. AI-aided pipeline Notice the engineer never disappears from this diagram. They just move from being the one who does the correlation to the one who checks the correlation. That distinction matters, because it changes what you hire and train for. I made a version of this same argument about production-grade agents generally in the Shipping Production-Grade AI Agents refcard: an agent needs a human review layer, or you're just moving risk around instead of removing it. If you want a deeper look at how these review loops are actually structured under the hood, I broke that down recently in Loop Engineering: The Layer After Prompt, Context, and Harness Engineering. Incident Management: What Changed A few concrete things have improved in incident management tools over the last two years: Alert correlation got better. Tools like BigPanda, Moogsoft, and PagerDuty's AIOps features now cluster related alerts using pattern recognition instead of static rules. A database timeout, three downstream service errors, and a spike in 500s used to show up as four separate pages. Good correlation engines now group them as one incident with a suggested cause. Similar-incident retrieval works reasonably well. If your org has a decent history of past incidents with clean postmortems, tools can now surface "this looks like INC-4471 from March" with real accuracy. This only works if your postmortem data isn't garbage, which is a bigger blocker than people admit. Draft postmortems save real time. Not because the AI writes a good postmortem on the first try, but because staring at a blank page is the slowest part of writing one. A rough draft built from the incident timeline, Slack thread, and metrics gives engineers something to edit rather than create. What's newer, and worth its own section, is a class of tools that don't just correlate what you already collected. They go get new evidence during the incident, the way a senior engineer would. The New Category: Dedicated AI SRE Agents This is the part of the landscape that's moved fastest since I last wrote about agent tooling. A handful of startups have built agents whose entire job is investigating production incidents autonomously, not just clustering alerts that already exist. Traversal leans on causal machine learning rather than a general-purpose LLM wrapper. Instead of pattern-matching against similar past incidents, it builds a model of causal dependencies across your services and traces the actual chain of cause and effect, down to the specific deploy or config change that started the failure. It reports strong root-cause accuracy in production at large enterprises and is used for both alert triage and live incident investigation. The pitch is narrower than "full AIOps platform," and that narrowness is the point. Resolve.ai takes a broader angle. It was built by the team that created OpenTelemetry, and it positions itself as an agentic teammate across the whole production lifecycle: investigating incidents, but also touching capacity questions, config drift, and guided code changes. Where Traversal is a specialist in root cause, Resolve.ai is closer to a generalist you'd loop in on almost anything production-related, with the incident work as the anchor use case. Cleric sits in a similar space to both, with a specific focus on autonomous alert triage. It runs a multi-source investigation the moment an alert fires, pulling metrics, logs, traces, and deploy history in parallel, and returns an evidence-backed hypothesis before an on-call engineer has finished opening their second dashboard tab. It runs with read-only access by default, which matters a lot for teams still building trust in the category, and it was named a Gartner Cool Vendor in AI for SRE and Observability in 2025. That "read-only by default" design decision is exactly the kind of guardrail I argued for in Trust No Agent: How to Secure Autonomous Tools on Your Machine: an agent's blast radius should be a deliberate design choice, not an afterthought. Causely and NeuBird round out the space with slightly different angles: Causely focuses on causal reasoning to find the single root cause behind a storm of cascading alerts, and NeuBird targets enterprise IT environments with LLM-driven telemetry analysis at large scale. Here's roughly where an AI SRE agent sits in the pipeline compared to the AIOps correlation tools from the last section: AI SRE agent pipeline The key difference from the earlier diagram: this agent isn't just correlating signals you already collected in a dashboard. It's actively going out and querying your systems the way a human on-call engineer would, forming a hypothesis, testing it, and either confirming or discarding it before it ever pages a person. That's a meaningfully different capability than clustering alerts by similarity, and it's why this category gets its own row in any serious comparison. If you're weighing whether to build this kind of investigation loop yourself versus buying one of these platforms, it's worth reading MCP vs Skills vs Agents With Scripts: Which One Should You Pick? first, since the architecture decision behind "agent that calls tools live" versus "agent with a fixed skill set" applies just as much to SRE tooling as it does anywhere else. What This Actually Brings to the SRE Persona It's worth being specific about what changes for the person on-call, not just what the vendor deck claims: Fewer 2 a.m. investigations that start from zero. The agent has usually already ruled out the obvious suspects by the time a human looks at the page, so the engineer starts from a hypothesis instead of a blank terminal.Less tool-hopping. A lot of incident time isn't spent thinking; it's spent switching between Datadog, Grafana, the CI pipeline, and Slack. An agent that queries all of them in parallel removes a genuinely tedious chunk of the job.A written trail for free. Because the agent's investigation is itself a structured log of what it checked and why, you get a decent postmortem skeleton as a byproduct, not a separate task.A new failure mode to watch for. Engineers can start trusting the proposed root cause without checking the evidence trail, especially under pager pressure. That's a habit worth actively training against, not assuming away. None of this replaces the on-call engineer's judgment. It changes the shape of their shift from "gather evidence, then decide" to "review evidence, then decide," which is faster but only as trustworthy as the evidence the agent actually gathered. Comparing the Tool Landscape Here's how some of the major players stack up on where they've actually invested in AI, versus where it's mostly a checkbox feature. I've split this into two tables, because lumping AIOps correlation platforms in with dedicated AI SRE agents hides a real difference in what these tools do. Established AIOps and incident platforms: ToolAlert CorrelationRoot Cause SuggestionAuto-Drafted PostmortemsPredictive CapacityOwnership / StatusPagerDuty (AIOps)StrongModerateYesLimitedIndependent, public companyMoogsoftStrongStrongNoNoAcquired by Dell Technologies (2023)BigPandaStrongModerateLimitedNoIndependent, privateDatadog (Bits AI)ModerateStrongYesModerateBuilt in-house by DatadogServiceNow (Now Assist)ModerateModerateYesStrong (ITOps)Built in-house by ServiceNowDynatrace (Davis AI)StrongStrongLimitedStrongBuilt in-house by Dynatraceincident.ioModerateLimitedYesNoIndependent, privateRootlyModerateLimitedYesNoIndependent, private Dedicated AI SRE agents: ToolCore ApproachActs Autonomously?Best FitFounded / BackingTraversalCausal ML across dependency graphInvestigation autonomous, remediation guardedTeams with strong existing observability wanting sharper RCA2023, Sequoia and Kleiner PerkinsResolve.aiBroad agentic reasoning over code, infra, telemetryInvestigation autonomous, remediation opt-inTeams wanting one agent across incidents, capacity, and config2024, Greylock-led seedClericMulti-source parallel investigationRead-only by defaultTeams new to AI SRE agents, wary of write access2024, Zetta Venture PartnersCauselyCausal reasoning on cascading alertsInvestigation onlyEnvironments with alert storms and unclear blast radiusPrivate, early stageNeuBirdLLM-driven telemetry analysis at scaleInvestigation, guided remediationLarge enterprise IT environmentsPrivate, early stage A caveat worth stating plainly: I haven't run rigorous side-by-side benchmarks on all of these, and vendor claims move faster than reality, especially in the AI SRE agent table where most of these companies are one to three years old and evolving month to month. Treat both tables as a directional map, not a scorecard, and validate against your own alert volume before picking one. Deterministic AI vs. Generative AI in These Tools This distinction gets muddled in vendor marketing, so it's worth separating clearly. AspectDeterministic / ML-based (older AIOps)Generative AI (LLM-based, newer)ApproachStatistical pattern matching, clustering, anomaly detectionLanguage model reasoning over logs, tickets, chat history, and live queriesPredictabilityHigh, same input gives same outputLower, outputs can vary between runsStrengthCorrelation, anomaly detection at scaleSummarization, hypothesis generation, natural language explanation, draftingWeaknessPoor at explaining "why" in plain languageCan hallucinate a plausible-sounding but wrong root causeWhere it shows upDynatrace Davis AI, Moogsoft's original correlation engineDatadog Bits AI, ServiceNow Now Assist, Traversal, Resolve.ai, ClericTrust level neededCan often auto-remediateNeeds human review before action Most modern platforms now run both in tandem: the deterministic layer does the anomaly detection and correlation, and the generative layer explains it in plain English, forms hypotheses, and drafts the writeup. That combination is doing more real work than either piece alone, and it's basically the same pattern I described for agent observability generally in the AI agent architectures piece linked earlier: a fast, boring, reliable layer underneath a slower, flexible reasoning layer on top. Where We're Headed in Part 2 Incident response gets the spotlight because it's the loudest part of the job, but if you track where an SRE's actual week goes, a lot of it isn't firefighting at all. It's chaos testing, SLO math, on-call scheduling, and the slow grind of writing and maintaining runbooks nobody reads until 3 a.m. In Part 2, I'll walk through where AI is showing up in ITOps specifically, and then go further into chaos engineering, SLO and error budget management, on-call toil reduction, and capacity planning, the quieter parts of the job that determine whether the incident tools in this article even have a fighting chance.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

We built an LLM pipeline to help a large network operations team stay on top of trouble tickets. It ran quietly in production until the moment it was supposed to earn its keep. In early 2026, a major winter storm swept across a wide region and knocked out power to more than a million people; network equipment failed in bulk, tickets poured in, and the summarizer meant to help engineers triage the chaos went dark. The root cause was not a bug in the usual sense. There was no null pointer and no bad deploy. We hit the Azure OpenAI tokens-per-minute (TPM) limit, our retries made it worse, and we had no fallback. This is the anatomy of that failure, and the architecture we built afterward to treat an LLM like the rate-limited, non-deterministic dependency it actually is. The uncomfortable theme up front: our system was busiest during precisely the event it existed to handle. Demand and failure were correlated. If you put an LLM in front of any incident-driven workload, this will eventually be your story too. What the System Did Tickets in this environment originate from many channels, including network alarms, customer calls, emails, and proactive checks by operations staff. But by the time our pipeline sees them, they are already incidents and cases in ServiceNow. Our scope starts there. ServiceNow streams ticket events out of the box through Stream Connect into Kafka. Our application, running in Azure and orchestrated with LangGraph, consumes those events, retrieves related context from Azure AI Search, and calls the Azure OpenAI API to produce three kinds of summary: Status notifications for the customers affected by an outage,Ticket summaries for the technicians actively working a ticket, andExecutive summaries that roll up what is happening across a region. The value is simple. Ticket logs are long, noisy, and full of machine-generated entries. A technician picking up a ticket, or a manager gauging the blast radius of an outage, does not want to read pages of log. They want five sentences. The LLM gave them five sentences, and on a normal day it sat comfortably within quota. The Failure Timeline Then the storm hit. Equipment failed in bulk. The storm drove power outages past a million customers across a wide region, and our network equipment failed along with the grid. The alarm systems did exactly what they were designed to do: they fired, in volume.Tickets surged. They grew to roughly six times our baseline.The token load surged far faster. This is what caught us. Our load is not measured in requests; it is measured in tokens. Storm tickets did not just arrive more often, each carried a longer log (more alarms, more correlated events). So, a ~6× jump in tickets became closer to a ~15× jump in tokens per minute.We hit the TPM ceiling. Azure OpenAI began returning 429 Too Many Requests with a Retry-After header.Retries deepened the throttle. Every layer that could retry, did. That included the SDK, our wrapper, and LangGraph nodes re-running on failure, all in near-unison, with no jitter. Each retry wave slammed the limit together and pushed our effective token rate higherwhile we were already over budget. And because every retry of a generation is another paid, token-billed call, the retries spent the very budget we had blown..There was no fallback. When retires were exhausted, there was nowhere to go. There was no cheaper model and no degraded path. Summarization simply stopped The cascade. Customer status notifications stalled, technicians lost the ticket summaries they rely on, and executive summaries went stale. So, the team fell back to reading raw logs by hand. Summarization stayed degraded, on and off, for a multi-hour stretch, until we manually provisioned extra capacity and hand-routed traffic to other models to limp through the worst of it. The shape of the overload, with illustrative numbers to make the dynamic concrete: Metric Normal day Storm Summaries per minute ~40 ~240 (≈6×) Tokens per summary (log + context + output) ~3,300 ~8,000 (longer logs) Token demand ~132K TPM ~1.9M TPM Token quota ~250K TPM ~250K TPM Result ~53% utilization ~7.7× over → sustained 429s (The figures are illustrative estimates that preserve the real proportions, not exact production measurements.) The punchline is in the third row: a ~6× rise in tickets became a ~15× rise in tokens. That is the trap of a token-metered dependency, and the rest of this article is what it taught us. How the original outage cascaded — and why naive retries made it worse. Root Cause: An LLM is a Token-Metered Dependency, Not a Request-Metered One Most writing on resilience, including circuit breakers, retries, and bulkheads, is framed around microservices, and most of it applies here. But an LLM API breaks a few assumptions those patterns quietly rely on, and each broken assumption showed up in our incident. 1. The limit is tokens, not requests. Classic rate-limit thinking counts calls; Azure OpenAI quota is measured in tokens per minute. Your load therefore depends on the size of your inputs — and for a summarizer that is the worst possible coupling: it burns the most quota exactly when documents are longest, which during an incident is exactly when logs are longest. A request-rate dashboard would have looked merely elevated while our token rate was off the chart. 2. Retries spend the budget you are already over. On a normal REST API, a retry is cheap. On a token-metered, pay-per-token backend, every retried generation is another full charge against the limit you just exceeded. Naive retries do not just fail to help. They actively deepen the throttle. 3. Synchronized retries are a self-inflicted DDoS. With no jitter, failed calls backed off by the same amount and returned together, re-tripping the limit on a clock. It is the classic retry storm, amplified by point #2 because each retry is token-expensive. 4. No fallback means peak demand is a single point of failure. One model, one deployment, one path is fine until that path is throttled, and it will be throttled at peak. 5. Demand correlates with failure. A summarizer for incident tickets is, by definition, busiest during incidents. The load spike and the operational emergency are the same event. Capacity planned for the average is capacity planned for the calm before the thing you actually built the system for. The Fix: Classify, Route by Severity, and Govern the Token Budget The redesign treats the LLM as a scarce, metered resource and spends it deliberately, turning the frantic, manual capacity-adding and model-rerouting we did by hand during the storm into a permanent, automatic capability. Schedule in Redis, not Kafka. Our Kafka topics are shared by many interfaces and kept generic, so we could not repurpose them for prioritization. Instead, our consumer reads the generic stream and pushes work into Redis priority queues, where all the scheduling logic lives. Kafka stays the durable ingestion layer — a natural backpressure buffer, so a storm surge piles up safely in the log instead of hammering the model, and consumer lag becomes our early-warning storm metric. Classify with a tiny model. A small, local ML classifier scores each ticket by priority, severity (P1–P5), and customer impact, using fields already on the ServiceNow ticket. It is deliberately not an LLM call: during a storm every Azure OpenAI token is contested, so spending premium tokens just to decide how to spend premium tokens is exactly backwards. When the classifier is unsure, it routes up, because under-serving a real P1 is far worse than over-spending on a P4. Route by severity to isolated capacity. Each tier gets the cheapest treatment that still meets its need: Severity Routes to Why P1 / P2 Premium model deployment (related incidents coalesced into one regional rollup) High stakes, exec-facing; worth the tokens P3 / P4 Separate, cheaper model deployment "Good enough" at a fraction of the tokens P5 Non-LLM extractive summary (error counts, key fields, first/last events) Zero tokens; also a universal degraded mode The key trick is that the cheaper tier is a different model, so it draws from a different Azure OpenAI quota pool — a flood of low-severity tickets cannot cannibalize the premium tier's TPM. This needs no provisioned throughput; two standard deployments on different models give you quota isolation for free. Govern the token rate. A shared, Redis-backed token budget gates every LLM call: we estimate a job's tokens before dispatch and only proceed if the rolling per-minute budget allows, per deployment. Retries use bounded exponential backoff with jitter and honor Retry-After; the first worker to see a 429 sets a global cooldown the whole fleet respects, so the retry storm cannot form. Low-priority queues age and get promoted so they are never starved, and at-least-once delivery is made safe with idempotency keyed on ticket plus log version. Put together, the request path becomes: ServiceNow → Stream Connect → Kafka → classifier → Redis priority queue → token governor → the right model (or extractive fallback). The queue absorbs the spike, the governor respects the ceiling, and severity routing decides who gets the scarce premium tokens when there are not enough to go around. The redesigned pipeline: tickets are classified by severity, scheduled through Redis with a token governor, and routed to isolated model tiers. What We Expect (By Design) With this in place, the same storm should behave very differently. The 429 cascade cannot recur by construction. The governor caps dispatch at quota, so overflow becomes bounded queue lag — low-priority summaries delayed by minutes — rather than total failure.Premium capacity is protected. Routing roughly the top 15% of tickets to the premium tier and coalescing related incidents keeps it within quota even under the surge.Cost falls. Moving the bulk of volume to a cheaper model and the long tail to zero-token extraction projects on the order of a 50–65% blended token-cost reduction. Takeaways Plan capacity in tokens, not requests: Your load is driven by input size, which spikes exactly when you can least afford it.Design for the spike, not the average: Assume demand correlates with failure.Make retries jittered, bounded, and 'Retry-After'-aware: Remember each retry costs tokens.Tier your models by importance: Put cheap or non-LLM paths under the long tail, and isolate premium capacity on its own quota pool.Always keep a degraded mode: A rough summary delivered beats a perfect one that never arrives.

Logging is one of the oldest practices in software engineering, yet in distributed systems it remains one of the most poorly implemented. Most teams log, but very few log well. The gap between having logs and having useful logs becomes painfully visible the moment a production incident occurs at 2 AM across a system running dozens of microservices. This article focuses on structured logging: what it is, where teams consistently go wrong with it, and the concrete practices that separate log data you can actually act on from log noise that burns engineering hours during incidents. If you are building or operating distributed systems today, structured logging is not optional. It is the foundation on which every other observability signal- traces, metrics, alerts- depends. What Structured Logging Actually Means Structured logging means emitting log entries as machine-readable key-value pairs rather than arbitrary free-text strings. Instead of this: Plain Text [ERROR] 2026-07-10 03:14:22 - Failed to process payment for user 84729, reason: timeout You emit this: JSON { "timestamp": "2026-07-10T03:14:22Z", "level": "error", "service": "payment-service", "event": "payment_processing_failed", "user_id": 84729, "reason": "timeout", "duration_ms": 3001, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" } The difference sounds cosmetic. It is not. The first format requires regex parsing and string matching to extract meaning. The second is immediately queryable, aggregatable, and, crucially, correlatable with traces and metrics from other services handling the same request. The Five Mistakes Distributed Systems Teams Make With Logs 1. Logging Without Context Propagation In a monolith, a single log line tells you where in the codebase an event occurred. In a distributed system, a log line without a correlation identifier tells you almost nothing. If Service A calls Service B which calls Service C, and Service C fails, you need a shared identifier, typically a trace ID, that threads through all three services' logs so you can reconstruct the full request journey. The fix is context propagation: passing a trace ID through every request, injecting it into every log entry, and configuring your logging library to include it automatically. In practice, this means integrating your logging setup with OpenTelemetry or a similar tracing framework from day one, not as an afterthought. When your log entries include trace_id and span_id fields, you can jump from a log entry to its full distributed trace in a single query; that capability compresses incident diagnosis from hours to minutes. 2. Inconsistent Field Naming Across Services In a microservices architecture developed by multiple teams, field-naming inconsistencies compound into a real problem at scale. One service logs user_id, another logs userId, a third logs uid. One service logs errors under error, another uses err, another uses exception. When you need to query across services during an incident, this inconsistency forces per-service query variations, slowing everything down. Establish and enforce a logging schema across your organization. Define a canonical set of field names for common concepts, user identifiers, request identifiers, error fields, latency fields, and make that schema part of your service standards. Libraries like structlog in Python or logrus/zap in Go make it straightforward to enforce common fields at the logger initialization level, so teams can't easily deviate from the schema accidentally. 3. Logging at Wrong Severity Levels Severity level misuse is endemic. INFO logs that should be DEBUG. Application errors logged as WARN because the developer did not want to trigger alerts. Business logic exceptions logged as ERROR when they are expected and handled. Over time, this degrades the signal value of severity levels to the point where teams stop filtering by level entirely. Adopt and document clear severity semantics for your organization: DEBUG: information useful only during active development; should not run in productionINFO: normal operational events (service started, request received, job completed)WARN: unexpected conditions that are recoverable and do not require immediate actionERROR: failures that require investigation; every ERROR should eventually be investigated or suppressed with documented justificationFATAL: unrecoverable failures; service cannot continue Treat severity levels as a contract with your future on-call self. 4. Over-Logging Hot Paths High-throughput services that log every incoming request at INFO level generate enormous log volumes that create three problems: storage costs escalate, log search performance degrades, and genuinely important events get buried in noise. A service processing 10,000 requests per second generates over 860 million log lines per day from request logging alone. Use sampling for high-frequency, low-severity log events. Most observability platforms and log monitoring tools support log sampling natively; you configure a sampling rate for specific log patterns, keeping representative data without keeping everything. For example, sample 1% of successful payment processing logs but keep 100% of error logs. This dramatically reduces volume while preserving signal fidelity where it matters. 5. Treating Logs as a Standalone Signal Logs become exponentially more powerful when they are correlated with traces and metrics. A spike in error logs is interesting. An error log spike correlated with a latency metric increase correlated with a trace showing a database connection timeout is actionable in seconds. Teams that treat logs as independent from their other observability signals are leaving significant diagnostic capability on the table. If you are not already running OpenTelemetry, start there. It provides a unified SDK for instrumenting logs, traces, and metrics in a way that ensures they carry shared context identifiers. Once your logs carry the same trace IDs as your distributed traces, your observability signals become correlated by default, not by manual investigation. A Practical Logging Schema to Start With Here is a minimal structured logging schema that covers the majority of production use cases across distributed services: JSON { "timestamp": "ISO-8601 UTC", "level": "debug|info|warn|error|fatal", "service": "service-name", "version": "1.4.2", "environment": "production", "event": "snake_case_event_name", "message": "Human-readable description", "trace_id": "OpenTelemetry trace ID", "span_id": "OpenTelemetry span ID", "user_id": "optional", "request_id": "optional", "duration_ms": "optional, numeric", "error": { "type": "TimeoutError", "message": "Connection timed out after 3000ms", "stack": "optional, omit in high-volume paths" } } This schema is opinionated but extensible. Services add domain-specific fields as needed while every entry maintains the common fields that make cross-service correlation possible. Conclusion Structured logging in distributed systems is not about logging more; it is about logging intentionally. The practices that separate teams who resolve incidents in minutes from teams who spend hours in log archaeology come down to four things: consistent field naming, trace context propagation, disciplined severity usage, and treating logs as a correlated signal rather than an isolated one. Get these right, and your logs become a first-class observability asset during incidents. Get them wrong, and you have the worst of both worlds: high storage costs and low diagnostic value. The patterns outlined here are not theoretical; they are the difference between incident response that feels like detective work and incident response that feels like reading a timeline.

I work as a data analyst at a legal services company. Part of my work involves protecting sensitive data during the Test Data Management (TDM) process. Many other departments in the company need test data to develop an application. Copying the production data for test sounds like a good plan. But because the test environment usually has lower cybersecurity requirements, this will cause customer privacy data leaks. So, my job is to mask the sensitive data to protect customer privacy. When it comes to my job, the first thing that comes to many people’s minds is that my work involves masking sensitive data. For example, changing the email address from [email protected] to [email protected]. Masking data is indeed important, but before we jump to the masking step, there's one basic question: Which column contains sensitive data, and how can I find it? In this article, I will introduce a pipeline designed to identify sensitive data columns before masking steps. Structure of the Pipeline Please find the identifying sensitive data pipeline structure workflow chart below: Identifying Sensitive Data Pipeline Structure Workflow Before I start introducing each stage, I’d like to mention two points. The first point: The original intent behind this pipeline structure design was to save time spent locating sensitive data. Server usage is billed based on duration. In a perfect world, the system would balance efficiency and accuracy. However, in practice, efficiency takes precedence in order to cut costs. The second point: The pipeline also had to preserve data usability for testing. In some cases, data privacy controls must be designed in a way that does not break core application workflows. For key columns such as primary keys and foreign keys, they need to preserve join functions and application workflows. So in practice, we usually leave them unchanged. Apply Column Name and Pattern Matching First, and quite intuitively, many columns' names are really straightforward and can be easily identified. For example, full name, phone number, and email. After the very first easy screening, some columns can be identified by hardcoded Python scripts, based on the specific column name pattern. However, there is an issue at this stage. I can identify columns containing sensitive data using customer email. But if there is another column named customer email address that hasn't been included in the hardcoded script, I won't be able to detect it. Besides that, relying solely on column names isn't always reliable. Take the notes column, a free-text field, for instance. It often appears as an optional field after the main information has been entered. Most people will leave it blank or write some insignificant things. But sometimes customers do write something, such as Our CEO Everett would like you to prioritize processing the ABC document. Please email them to [email protected] as soon as you finish, and then call 123-456-7890 to notify him. If I don't mark this column as need masking, the customers' private information will be exposed. Check Historical Decisions After the initial filtering step, I will check the historical decisions database for specific columns, such as the notes column mentioned earlier. If the database indicates that the historical decision for column notes is to mask it, then that column will be masked during the current round. Even if the notes in this specific round contain no sensitive data. For example, no privacy-related information is mentioned. There is no guarantee that the data in the next refreshed cycle will remain free of sensitive information. Send Ambiguous Columns to AI for Review and Analyze Sample Values Here comes the highlight of the entire pipeline. Sometimes, column names are somewhat ambiguous. Or it's unclear whether certain rows contain sensitive data. Let's take the notes column mentioned earlier again. It might be empty. Or it could contain a message like When food is delivered, please ring the doorbell and call my wife Bobi, thereby the sensitive information gets leaked. I started using the spaCy library from Python for Natural Language Processing (I will refer to this term as NLP later in this article). While spaCy isn’t a Large Language Model (I will call this term LLM), it certainly performs NLP analysis. However, the sampling process was time-consuming. I would sample the entire dataset if it had fewer than 50,000 rows, but randomly select 50,000 rows if it exceeded that limit. In a later version of the workflow, I switched to OpenAI: this time, I just need to select a sample of 100 rows and send them via API to the AI/LLM for analysis. The AI then generates a masking recommendations database, which will undergo manual review later. Accuracy improved significantly after we began using LLMs. It rose from 80% with spaCy to approximately 93% after switching to OpenAI. This 93% figure was determined by having human analysts conduct a column-by-column analysis in parallel with my development of the pipeline and automation scripts. So the result is benchmarked against manual reviews. Furthermore, this figure represents an average obtained after two rounds of actual TDM data masking operations and several additional rounds of testing. Regarding the remaining 7% of errors, false positives accounted for about 90%, and false negatives for only 10%. This is important because missing sensitive data is much more serious than over-flagging a column for review. Compared to manually analyzing a medium-sized schema containing 100 tables for 64 hours. An automated script can complete the analysis in just 2 hours. However, please note that this 2-hour timeframe does not include the time required for subsequent manual review. Human Review and Store Recommendations and New Decisions After the AI/LLM finishes analysis, human analysts will review the mask recommendations database generated by the AI. Each row in the database generates a report containing the user ID, database name, table name, column name, masking suggestion, masking rule, and analysis date. Then, humans will review the mask suggestions and corresponding masking methods. For example, the AI-generated mask suggestion database is: AI-generated Mask Suggestion Database Example As a human analyst, at this stage, I can review the masking suggestion generated by the AI. I would agree with the suggestion to mask the data. However, regarding the masking rule, I would review it and change it to set it to a blank value. Manual review needs to randomly sample 500 rows and analyze them individually to reach a final mask decision. In this new process, human analysts only need to review a single row of AI-generated mask decisions and mask rules. The switch saves time significantly. During a new round of the TDM data masking process, some new columns will be identified by AI and flagged as requiring masking. The new masking decision will be added to the existing historical decisions database after manual review. Send to Data Governance and Send to Business Customer and Get Feedback After our TDM team identifies and masks the sensitive data columns, we submit our results to the Data Governance department for a secondary manual review. Their review process differs slightly from ours. Our team focuses on using business knowledge to determine whether a column contains sensitive data. And we’re also responsible for developing more efficient identification & masking procedures. However, the Data Governance department needs to review and provide more accurate masking decisions. Because their team members have better knowledge of how to decide whether a column should be masked and of the appropriate masking method. After our two departments conducted two rounds of manual review, we sent the masked data results to our business customers' departments. They will use this data for testing and provide us with feedback based on their specific needs. For example, we recommended masking customer_id with a generated synthetic number. But doing so will change primary and foreign keys, thereby breaking database linkages. So, our business customer departments advised us against masking those columns. Conclusion and Future Improvement Plans Successfully masking sensitive data begins with accurately identifying the columns containing such data. Many people skip this and jump straight to the more interesting masking process. In my view, however, getting this step wrong will fail the rest of the workflow as well. The pipeline I designed isn't perfect. And I have a few ideas for improving the "Apply column name and pattern matching" component in the future. Since we’ve already used OpenAI, why not let the AI detect new patterns when analyzing ambiguous columns? We could have the AI generate a dynamic pattern database that updates automatically with every refresh cycle. It would also help us continuously update and refine our historical decisions database.

The Problem: Our p99 Was 3-5 Seconds Our PyFlink pipeline was missing its latency SLO by seconds. The pipeline itself was straightforward: consume events from Kafka, transform them, serialize them as Protobuf, and write the results to downstream systems. Yet under production load, p99 end-to-end latency was consistently in the 3-5 second range. Profiling pointed us to an unexpected bottleneck: we were deserializing Protobuf messages in Python, even though the Flink runtime processing our stream was JVM-based. Every record that entered the Python path had to cross the JVM-to-Python process boundary, get parsed by a Python UDF, and then cross back. The business logic wasn't the problem. The doorway was. We moved Protobuf deserialization to Flink's JVM-side Protobuf format and kept Python for orchestration and SQL. In our environment, p99 dropped to approximately 500 milliseconds, with less code and a pipeline that is easier to reason about. Verified on AWS Managed Service for Apache Flink (formerly Kinesis Data Analytics). Why Python-Side Deserialization Is So Expensive The naive PyFlink architecture looks like this: A Kafka source table declared with a generic format (raw, json, or a SimpleStringSchema), so every record arrives as opaque bytes or a string.A Python map() or UDF that imports generated _pb2.py classes and calls ParseFromString() on every message.Downstream transforms and sinks. Two costs hide in step 2, and they compound at high throughput. The process boundary. PyFlink is not Python running inside Flink; it is a JVM runtime coordinating with a separate Python execution environment. Every record that enters the Python execution path incurs overhead associated with moving data between the JVM and Python, and depending on the operator and execution mode, that can involve serialization and inter-process communication in both directions. For a per-record deserialization UDF on a latency-sensitive pipeline, that overhead is paid before the actual business transformation begins. Per-record parse cost. Even when Python's Protobuf implementation uses its native backend, parsing in a Python UDF still requires the record to enter the Python execution path. When the workload is latency-sensitive and high-throughput, the combination of serialization, inter-process communication, Python execution, and parsing overhead can become significant. In our case, profiling showed that this path was a major contributor to our latency. In our pipeline, these two costs together accounted for the bulk of the gap between a 3–5 second p99 and the ~500ms target we needed, before the enrichment logic even began executing. The Key Realization: PyFlink Already Runs on the JVM Here's the insight that changes the architecture: if Protobuf is declared at the table DDL level, Flink's Kafka connector deserializes it with its native, optimized JVM-based Protobuf format before any data reaches the Python side. Your columns simply arrive typed and ready. Python's role shrinks to what it's genuinely good at in this stack: orchestration and SQL. No rewrite to Java. No change to how jobs are deployed. Just a different declaration of intent. The trade is that Flink's native Protobuf format needs compiled Java message classes on the classpath; it does not consume .proto files or Python _pb2 modules directly. That means adding a small build step to your workflow, which we'll cover below. Implementation The pipeline splits into two declarative jobs. Job 1: JSON In, Protobuf Out The source table reads the raw JSON topic; the sink table declares format = 'protobuf' and points at the compiled Java class. The JVM handles typed-row-to-Protobuf encoding. SQL -- SOURCE: raw JSON payload as STRING plus Kafka record timestamp CREATE TABLE source_events_json ( event_data STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = '${INPUT_JSON_TOPIC}', 'properties.bootstrap.servers' = '${KAFKA_BOOTSTRAP_SERVERS}', 'scan.startup.mode' = 'latest-offset', 'format' = 'raw' ); -- SINK: Protobuf out to Kafka (JVM handles typed row to Protobuf) CREATE TABLE sink_events_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent' ); -- TRANSFORM: pure SQL, no Python UDFs INSERT INTO sink_events_pb SELECT JSON_VALUE(event_data, '$.id') AS id, JSON_VALUE(event_data, '$.organization_id') AS organization_id, ROW( UNIX_TIMESTAMP(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '')), CAST(EXTRACT(NANOSECOND FROM CAST(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '') AS TIMESTAMP_LTZ(9))) AS INT) ) AS event_ts, CAST(JSON_VALUE(event_data, '$.is_active') AS BOOLEAN) AS is_active, JSON_VALUE(event_data, '$.event_type') AS event_type FROM source_events_json; Note what's absent: no ParseFromString(), no _pb2.py imports, no Python deserialization loop. The Python program registers DDL and runs SQL. Job 2: Protobuf In, OpenSearch Out Downstream, the sanitized Protobuf topic becomes a typed source, using the same protobuf.message-class-name property, plus ignore-parse-errors so a malformed record can't poison the pipeline. SQL -- SOURCE: Protobuf from the sanitized Kafka topic CREATE TABLE kafka_source_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'scan.startup.mode' = 'latest-offset', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent', 'protobuf.ignore-parse-errors' = 'true' ); -- SINK: OpenSearch (JSON) CREATE TABLE opensearch_sink ( id STRING, organization_id STRING, event_ts TIMESTAMP_LTZ(3), is_active BOOLEAN, event_type STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'opensearch-2', 'hosts' = '${OPENSEARCH_ENDPOINT}:443', 'index' = 'acme-events-v1', 'format' = 'json' ); INSERT INTO opensearch_sink SELECT id, organization_id, TO_TIMESTAMP_LTZ(event_ts.seconds * 1000, 3), is_active, event_type FROM kafka_source_pb; The Build Step: Getting Java Classes Onto Flink's Classpath The one genuinely new piece of workflow is compiling your .proto definitions to Java and packaging them into the job's fat JAR. The essential Maven pieces: XML <dependencies> <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>3.25.5</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-protobuf</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>${flink.connector.kafka.version}</version> </dependency> <!-- plus your sink connectors, e.g. flink-connector-opensearch2 --> </dependencies> Two practices that made this maintainable for us: Version-control the generated Java sources (or generate them in CI from a single canonical .proto repo) and pull them in with build-helper-maven-plugin's add-source, rather than compiling .proto files in every consuming project. One schema source of truth, many consumers.Shade everything into one JAR with maven-shade-plugin, excluding signature files (META-INF/*.SF, *.DSA, *.RSA). On AWS Managed Flink, pass it via the job's JAR configuration; on self-managed Flink, drop it in lib/ or use --classpath. The full workflow: define the .proto, compile it to Java with protoc, package the fat JAR, put it on Flink's classpath, author the PyFlink job with the DDL above, then deploy and watch end-to-end p99. How We Measured the Improvement We measured end-to-end p99 latency as the time from a record landing on the source Kafka topic to the corresponding OpenSearch write being acknowledged MetricBeforeAfterp99 latency3-5s~500msSustained throughput~5,000 events/sec~5,000 events/secFlink parallelism128Python UDF parsingYesNoJVM/Python boundary on hot pathYesNoProtobuf decodingPythonJVM Results End-to-end p99 latency around 500 milliseconds in our environment at production load, down from a 3-5 second baseline, by eliminating per-record JVM-to-Python crossings and Python-side parsing on the hot pathLess code. The deserialization UDFs, the _pb2 imports, and their error handling all disappeared. What remains is DDL plus SQLSimpler and easier to operate. The pipeline now relies on Flink's Kafka connector and Protobuf format for serialization and parsing, with built-in parse-error handling, instead of hand-rolled Python parsing When This Optimization Won't Help Moving Protobuf decoding to the JVM won't automatically solve every latency problem. If your pipeline's critical path is dominated by sink backpressure, network latency, external API calls, state access, or checkpointing overhead rather than deserialization, changing the serialization path may have little effect on end-to-end latency. This optimization is most valuable when profiling specifically shows that Python execution and JVM/Python data movement are significant contributors to the critical path, which is why we'd recommend profiling first rather than applying this as a default change. When You Should Still Use Python UDFs This pattern is not "never write Python UDFs." It's "keep them off the per-record deserialization path." Python remains the right tool when: The transformation genuinely needs Python libraries (ML feature computation, model inference, specialized parsing that has no SQL equivalent).Throughput is modest and developer velocity matters more than the last hundred milliseconds.You're prototyping. Even then, declare the format natively from day one anyway; it costs nothing and you won't have to migrate later. If a UDF is unavoidable on a hot path, at least let the JVM do the deserialization first so the UDF receives typed columns rather than raw bytes. Gotchas Worth Knowing Before You Ship Property syntax varies by Flink version. Some versions use format = 'protobuf'; newer key/value descriptors prefer value.format = 'protobuf'. Check your version's docs.Enums: surface them as STRING if you need ergonomic SQL manipulation, or keep them numeric with a lookup table.Schema evolution: favor backward-compatible, additive changes with defaults. Because the compiled Java classes are baked into the JAR, a schema change means a rebuild and redeploy, so make that a deliberate, versioned step in CI rather than an afterthought. ignore-parse-errors is your safety net during rollout windows, but monitor the drop counter so it doesn't silently eat data.Benchmark end-to-end, not just the UDF: source lag, operator latency, and sink acknowledgments under production load patterns. Deserialization wins can be masked, or dwarfed, by sink backpressure.Security: lock down OpenSearch credentials and TLS; pin Kafka client versions compatible with your Flink release. Closing Thoughts We didn't rewrite the pipeline in Java. We removed an unnecessary per-record JVM-to-Python boundary from the hot path and let Flink's JVM-native Protobuf format do the work it was designed to do. If your PyFlink job parses Protobuf messages in Python today, check whether Flink's native format support can move that work into the JVM-side execution path. For latency-sensitive pipelines, eliminating unnecessary Python boundaries may be one of the highest-leverage optimizations to investigate, especially when profiling shows that serialization and Python execution are on the critical path.

Why Most Platforms Fail to Become Products Many companies are heavily investing in internal developer platforms (IDPs) with the expectation that they will speed up delivery and governance, and increase developer productivity. Despite significant investment in Kubernetes, CI/CD, observability, security tooling, and cloud infrastructure, many platforms struggle to gain adoption. The reason is simple: they are built and operated like infrastructure projects, not products. Infrastructure teams are often very focused on technical excellence: automation, scalability, reliability, and compliance. Developers, on the other hand, are interested in a different goal — getting their applications into production quickly and safely without having to go through so much complexity. IDP is successful when developers choose it voluntarily because it makes their lives easier. That shift requires platform architects to think less like infrastructure engineers and more like product managers. Building an IDP is like operating an airport. Nobody travels because they love airports. They travel because they want to reach a destination efficiently. Similarly, developers do not care about Kubernetes clusters, pipelines, secrets management, or observability stacks. They care about shipping features to customers. The platform's job is to make the journey smooth, fast, and safe. This article explores the core practices that differentiate successful product-centric platforms from infrastructure-centric ones. Practice 1: Start With Developer Journeys, Not Technology Choices Imagine constructing a shopping mall by selecting elevators, security systems, and air-conditioning units before understanding customer traffic patterns. The result is often technically impressive but operationally frustrating. The same happens with developer platforms. Architects should first map the customer journey (developer journey) before designing platform capabilities. Many platform initiatives begin with questions like: Which Kubernetes distribution should we use?Which GitOps framework is best?Which CI/CD tool should be standardized? These are important questions, but they should not be the starting point. Successful platform architects begin by understanding developer workflows: How does a new service get created?How long does environment provisioning take?Where do deployment delays occur?What causes support tickets?Which activities are repetitive and manual? The goal is to identify friction and eliminate it. Organizations using platforms based on technologies like Red Hat OpenShift, IBM Cloud Kubernetes Service, or other cloud-native platforms have found that developers adopt only when the platform team focuses on reducing the friction in workflow rather than adding more infrastructure features to the platform. Practice 2: Create Golden Paths, Not Golden Handcuffs A highway encourages drivers to use the fastest route while still allowing exits when necessary. Successful IDPs behave like highways. Developers naturally choose the Golden Path because it is easier and safer than building everything from scratch. One of the most powerful concepts in modern platform engineering is the Golden Path. A Golden Path provides: Recommended architecturesStandard deployment patternsPre-approved security controlsBuilt-in observabilityAutomated CI/CD workflows Developers should be able to move fast along a paved road while retaining flexibility for unique requirements. Platform teams that leverage services from cloud provider environments often realize that standardized self-service templates drive significantly higher adoption than restrictive governance models. Practice 3: Make Self-Service the Primary Interface Every banking transaction once required a visit to a physical branch. Today, customers expect to do everything from a mobile app. Developers hope for the same experience from inside their own software. Nothing kills developer productivity faster than dependency queues. Consider a common case of dependency queues. Open a ticket for infrastructure.Wait for approval.Wait for provisioning.Request secrets.Request monitoring.Request deployment access. Weeks can pass before development even begins. Modern platforms must provide self-service experiences where developers can do the following without opening tickets. Create environmentsProvision databasesConfigure pipelinesAccess observability dashboardsRequest infrastructure resources An IDP should function like a digital banking application—secure, streamlined, and available on demand. Below is the Product-Centric IDP reference architecture. Developers consume platform capabilities through self-service experiences, while the platform embeds security, observability, governance, and delivery capabilities and exposes them through Golden Paths. Practice 4: Treat Platform APIs as Products A power drill might have sophisticated engineering in it. Users judge it by a very simple standard: “Can I drill a hole fast and reliably?" Many platform teams are focused on infrastructure automation and not developer experience. Each API, template, workflow, and portal interaction is a product interface. Questions worth asking include: Is the API predictable?Is documentation clear?Are error messages actionable?Is onboarding intuitive?Can developers discover capabilities easily? Developers evaluate IDPs the same way. They are not interested in the complexity underneath. They care about usability. This principle is especially important when integrating observability services, cloud provisioning layers, or deployment automation platforms. For example, IBM Cloud's managed services can significantly simplify operational complexity, but value is realized only when developers experience that simplicity through intuitive platform workflows. Practice 5: Build Observability into the Platform, Not Around It Imagine when you are driving a car without any speedometer, fuel gauge or warning indicators. You may still reach your destination but the risk increases dramatically. Observability is the dashboard for software systems. Observability is often treated as an afterthought. A team deploys an application and later attempts to add: MetricsLogsTracesDashboardsAlerting This approach creates inconsistency and operational blind spots. Platform teams should embed observability from day one. Every service created through the platform should automatically include: Logging standardsDistributed tracingMetrics collectionHealth monitoringService dashboards Whether organizations use IBM Cloud Observability, Instana, OpenTelemetry, Prometheus, Grafana, or other solutions, the platform should make observability automatic rather than optional. Practice 6: Make Security Invisible but Ubiquitous When entering a modern office building, people rarely think about security. Access badges, surveillance, and emergency controls are built into the environment — the building is secure without requiring employees to become security experts. The same principle applies to IDPs. In immature environments, security is seen as a series of checkpoints, review meetings, manual compliance approvals, vulnerability assessments, and audit evidence collection. Developers find it as friction because it arrives late in the delivery lifecycle. Traditional security models operate as gates. Platform-centric security operates as guardrails. The objective is not fewer security controls — it is fewer manual interactions. Build Secure-by-Default Golden Paths Every new service created through the platform should automatically inherit: Secure CI/CD pipelines with dependency and container image scanningSecret detection and policy enforcementAccess control standards and audit loggingEncryption best practices Automate Policy Enforcement Manual compliance verification is one of the biggest sources of deployment delays. Platform teams should adopt policy-as-code (PaC) approaches that automatically validate deployment configurations, infrastructure standards, and regulatory controls. Instead of asking, "Did someone review this configuration?" the platform asks, "Does this configuration satisfy our policies?" Reduce Security Cognitive Load Developers should not need deep expertise in every security domain. The platform should abstract identity management, secrets management, certificate management, and vulnerability remediation workflows—particularly in hybrid and multi-cloud environments where security complexity grows rapidly. A useful measure of progress: the percentage of security controls inherited from the platform versus manually implemented by application teams. The higher the inheritance rate, the lower the cognitive load. Practice 7: Measure Platform Success Like a Product A gym owner does not measure success by counting treadmills—they measure it by member outcomes. Platform teams should apply the same logic. Traditional infrastructure metrics like cluster utilization, pipeline counts, and resource consumption tell you whether the platform is running. They do not tell you whether it is working for developers. Product-oriented platform teams focus on: Developer satisfactionPlatform adoptionTime to first deploymentDeployment frequencyLead time for changes If developers still circumvent the platform, no amount of technical sophistication matters. The Developer Experience Scorecard Measuring developer experience requires balancing sentiment, effort, and adoption. High-performing platform teams track four key measures: Metric What It Measures How to Collect Developer Satisfaction Score (DSS) Overall platform sentiment Quarterly survey, 1–10 scale Platform NPS Willingness to recommend the platform "How likely are you to recommend this platform?" scored 0–10 Ease-of-Use Score How intuitive common workflows feel Per-task rating, 1–5 scale Developer Effort Score How much work is required to achieve an outcome Survey question on effort per task Together, these reveal not just whether developers are using the platform but whether they genuinely value it. Satisfaction Is a Leading Indicator Most delivery metrics lag behind—deployment frequency (e.g., lead time, incident count) and other metrics. Developer satisfaction is a leading indicator. Developers discover friction long before it is observable from the data. A declining DSS today will result in a decline in productivity and adoption tomorrow. Listening early allows platform teams to respond before problems grow into organizational challenges. The real measure of success is not how many developers use the platform—it is how they feel while using it. The IDP Health Dashboard High-performing platform teams monitor a balanced set of metrics across four categories: Category Metrics Sentiment DSS, Platform NPS, Ease-of-Use ratings Adoption Golden Path adoption, self-service usage, onboarding rates Friction Support ticket volume, documentation search failures, manual approval requests Productivity Time to First Deployment (TTFD), environment provisioning time, lead time for changes A platform succeeds not when developers are forced to use it, but when they prefer to use it. Practice 8: Reduce Cognitive Load Relentlessly The automotive industry spent decades simplifying the driving experience so drivers could focus on reaching their destination rather than understanding the mechanics of their vehicles. IDPs should do the same. As organizations evolve into cloud-native architectures, developers are expected to navigate containers, Kubernetes, CI/CD, IaC, security policies, service meshes, observability tools, and compliance requirements all at once. Each one solves a very important problem individually. As a whole, they overwhelm developers and take focus away from developing business capabilities. A successful platform is not one that exposes every infrastructure capability. It is one that hides unnecessary complexity while providing simple, intuitive paths to outcomes. The goal of platform engineering is not to eliminate complexity. It is to absorb complexity so developers don't have to. Common indicators of excessive cognitive load: Developers struggling to find documentationFrequent support requests for routine tasksLong onboarding times for new servicesMultiple handoffs between teamsTool sprawl across the engineering ecosystem Reduce Tool Sprawl Every tool a developer must learn introduces new interfaces, terminology, documentation, and configuration models. Platform teams should create a unified experience through a developer portal, service catalog, or platform API, that minimizes the number of decisions and interfaces developers encounter. Minimize Context Switching Every transition between tools, teams, or approval processes introduces cognitive overhead. Platform teams should ask: Can this be automated? Can these steps be consolidated? Can approvals be replaced with automated guardrails? The goal is fewer interruptions between code creation and deployment. Platform Teams Are Complexity Brokers Complexity never disappears — it moves. Organizations can either push complexity onto every development team, or centralize and manage it within the platform. High-performing platform teams choose the latter, absorbing operational, security, infrastructure, and compliance complexity so application teams can focus on features. Practice 9: Obsess Over Time to First Deployment The first experience developers have with a platform often determines whether they embrace it or avoid it. Imagine a shopping mall where opening a new store requires twelve forms, multiple approval queues, and manual setup of every utility. Store owners would go elsewhere. The best malls provide ready-made spaces where businesses can start operating almost immediately. Developer platforms should do the same. High-performing platform teams focus relentlessly on Time to First Deployment (TTFD) — the time between creating a service and successfully deploying it. The Biggest Contributors to Poor TTFD Bottleneck Root Cause Fix Manual infrastructure provisioning Ticket-driven approval chains Self-service IaC, service catalogs, platform portals CI/CD pipelines built from scratch No standard templates Pre-built, reusable pipeline templates Security reviews at the end Late-stage compliance gates Shift left — embed scans and policy checks in Golden Paths Observability setup delays Manual metrics/dashboard configuration Auto-provision logging, tracing, and health checks by default Too many decisions Choice overload at onboarding Provide Golden Paths with sensible defaults Measure Every Stage Stage Target Service creation < 5 mins Repository creation Automated Pipeline creation Automated Infrastructure provisioning < 10 mins First build < 5 mins First deployment < 15 mins Observability enablement Automatic TTFD = Provisioning Time + Setup Time + Approval Time + Deployment Time Many organizations discover that approval time is larger than all technical activities combined. The fastest platforms replace approvals with automated guardrails. Practice 10: Build a Platform Community, Not Just a Platform Team Cities flourish when residents contribute feedback and shape growth. Cities planned entirely from a central authority often struggle to meet citizen needs. IDPs are no different. The best platforms evolve through continuous collaboration. Platform teams should create feedback loops through office hours, community forums, developer councils, internal documentation reviews, and experience surveys. Developers become co-creators rather than consumers. Community Health Metrics Running community mechanisms is not enough — each one needs a way to know whether it is working. Track these six indicators to measure community health: Metric What It Measures Healthy Signal Monthly Active Community Members Developers engaging in forums, channels, or office hours Steady growth quarter over quarter Developer-to-Developer Answer Rate % of forum questions answered by non-platform-team members Above 40% indicates a self-sustaining community External Contributions per Quarter Pull requests or documentation edits from application teams Increasing trend Roadmap Items from Community Input % of platform backlog items originating from developer feedback Above 50% signals product-centric culture Office Hours Repeat Attendance Rate % of attendees who return across multiple sessions Above 60% indicates ongoing value Support Ticket Deflection Rate % of issues resolved via community before a ticket is opened Rising deflection reduces platform team toil The ultimate sign of a mature platform community is a change in how developers talk about the platform—from something that happens to them to something they help shape. Practice 11: Think in Products, Roadmaps, and Customer Value Smartphones succeeded because manufacturers continuously improved user experience. Customers did not buy phones because of processor specifications. They bought outcomes—better communication, productivity, and convenience. Developers adopt platforms for the same reason. The strongest indicator that a platform is becoming a product is a change in language. Instead of asking: What infrastructure should we standardize? Platform teams begin asking: What developer problems should we solve next? Which user journeys create the most friction?Which capabilities deliver the highest value?What does our product roadmap look like? Features matter only when they improve the developer experience. Practice 12: Design for Platform Reliability, not Just Application Reliability Imagine a city that invests heavily in building roads, bridges, and public transport for its citizens, but has no maintenance crew, no traffic monitoring, and no plan for when a bridge closes. The infrastructure exists, but without reliability commitments, citizens cannot depend on it. Internal developer platforms face exactly the same risk. Most platform engineering conversations focus on the reliability of applications running on the platform — uptime, error rates, latency SLOs for customer-facing services. What is rarely discussed is the reliability of the platform itself. Yet the platform is load-bearing infrastructure for every engineering team in the organisation. When the CI/CD pipeline degrades, every team's delivery stops. When the service catalog is unavailable, no new services can be provisioned. The platform's reliability is a multiplier — a single failure can simultaneously impact dozens of teams. Define Platform SLOs Before Developers Define Them for You Platform teams that do not define their own Service Level Objectives will find that developers define them informally — through frustration, workarounds, and loss of trust. Effective platform SLOs cover the experiences developers depend on most: Pipeline availability — what percentage of CI/CD pipeline executions succeed without infrastructure-related failures?Provisioning latency — how long does environment or resource provisioning take at the 95th percentile?Portal availability — is the developer portal and service catalog accessible during working hours?Golden Path build time — how long does a standard pipeline template take to complete? These are the experience metrics developers encounter every day. A platform team that publishes and tracks these SLOs operates as a reliable internal service provider. A team that does not is invisible until something breaks. IDP Maturity Model Stage Characteristics Infrastructure Platform Standardized infrastructure, clusters, CI/CD tooling Self-Service Platform Service catalogs, automation, infrastructure on demand Developer Platform Golden Paths, integrated observability and security, DevEx focus Platform Product Platform roadmaps, adoption metrics, developer satisfaction measurement Adaptive Platform Continuous feedback loops, AI-assisted operations, continuous platform evolution Most organizations do not start with a Platform Product. They evolve toward it. The goal of the maturity model is not to reach the highest stage overnight, but to identify the next set of capabilities that will improve developer experience and platform adoption. High-performing platform teams treat platform maturity as a journey rather than a destination. Assessing Your Current Stage To identify where your platform currently sits, ask three diagnostic questions: How do developers access platform capabilities today? If the answer is "by opening a ticket," the platform is at the infrastructure stage. If developers provision resources on demand without human approval, they are at the self-service stage or beyond.Do developers choose the platform voluntarily or use it because they must? Voluntary adoption driven by speed and simplicity signals a developer platform or platform product. Mandatory usage with frequent workarounds signals an earlier stage.Does the platform team maintain a product roadmap prioritized by developer feedback? A yes here is the clearest indicator of a platform product. The absence of a roadmap almost always reflects an infrastructure or self-service mindset. Moving to the Next Stage Each stage has a single dominant unlock that drives progression: Infrastructure → Self-Service: Replace ticket-driven provisioning with self-service automation and a service catalog.Self-Service → Developer Platform: Introduce Golden Paths that embed security, observability, and CI/CD by default.Developer Platform → Platform Product: Establish a formal platform roadmap, measure developer satisfaction (DSS, NPS), and treat developer feedback as a product backlog.Platform Product → Adaptive Platform: Build continuous feedback loops, introduce AI-assisted operations, and invest in platform telemetry that proactively surfaces friction before developers report it. The most common mistake is attempting to skip stages. Teams that build Golden Paths before self-service exists create well-designed paths nobody can access independently. Teams that adopt satisfaction metrics before Golden Paths exist measure friction without the tools to address it. Progress through the stages in order. The IDP Architect's Checklist Before launching any new platform capability, ask: ✅ Does this feature remove friction from a developer workflow? ✅ Can developers access it through self-service? ✅ Is it aligned with a Golden Path? ✅ Is observability included by default? ✅ Is security built into the platform? ✅ Is governance automated rather than manual? ✅ Can success be measured through developer outcomes? ✅ Does it reduce cognitive load? ✅ Does it improve Time to First Deployment? ✅ Would developers choose this platform if they had alternatives? If the answer to several of these questions is "no," the capability is probably infrastructure-focused rather than product-focused. Final Thoughts The future of platform engineering is not about building more infrastructure. It is about delivering better developer experiences. The most successful IDPs combine the discipline of site reliability engineering (SRE), the automation of cloud-native technologies, and the mindset of product management. Whether your foundation runs on IBM Cloud, OpenShift, hyperscaler cloud services, or a hybrid environment, the winning formula remains the same: Treat developers as customers. Treat the platform as a product. Treat developer productivity as the ultimate business metric. When platform architects embrace this mindset, platforms stop being collections of tools and start becoming accelerators of innovation—and that's when platforms truly become products.

By Josephine Eskaline Joyce CORE

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen. Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel. In this blueprint, we will build a secure multi-tenant execution environment. We will containerize the agent runtime using Docker, provision a GKE cluster with Axion nodes, isolate the network, and orchestrate the execution layer using a robust Java backend. Step 1: Containerizing the Agent Runtime The first step is establishing a baseline execution environment. We want this Docker image to be as lightweight as possible to reduce the attack surface, while containing the necessary runtimes for the LLM to execute its logic. Dockerfile # Use a minimal Alpine base image to reduce attack surface FROM python:3.11-alpine # Create a non-root user for execution RUN addgroup -S agentgroup && adduser -S agentuser -G agentgroup WORKDIR /sandbox # Copy the execution wrapper script COPY --chown=agentuser:agentgroup execute_payload.py /sandbox/ # Enforce non-root execution USER agentuser # Prevent Python from writing pyc files and buffering stdout ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 CMD ["python", "execute_payload.py"] To make this functional, we need an entrypoint script that safely reads the LLM-generated code from an injected environment variable or a mounted volume, executes it, and captures the output. Here is a simplified execute_payload.py implementation: Python import os import sys import traceback def main(): # In a production environment, this payload might be injected via # a Kubernetes Secret or a secure sidecar proxy. encoded_payload = os.environ.get("AGENT_PAYLOAD", "") if not encoded_payload: print("Error: No payload provided.") sys.exit(1) try: # Execute the untrusted code within this isolated process # Security constraints are handled by the container and gVisor layers exec(encoded_payload, {"__builtins__": __builtins__}, {}) except Exception as e: print(f"Execution Error: {str(e)}") traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() Even if a malicious script breaks out of the Python runtime, it will find itself as an unprivileged user inside a minimal Alpine container. Step 2: Provisioning GKE With Axion and Agent Sandbox Google Axion (N4A) processors provide excellent performance per watt, making them ideal for running hundreds of concurrent, lightweight agent tasks. We will create a cluster and explicitly enable the sandbox feature. Shell # Create the GKE cluster with Sandbox enabled gcloud container clusters create agent-sandbox-cluster \ --region us-east4 \ --enable-sandbox \ --sandbox type=gvisor \ --release-channel regular # Create a dedicated node pool using Axion N4A instances gcloud container node-pools create axion-agent-pool \ --cluster agent-sandbox-cluster \ --region us-east4 \ --machine-type n4a-standard-4 \ --num-nodes 3 \ --node-labels dedicated=untrusted-agents \ --tags untrusted-workload Applying node labels ensures that trusted core microservices do not accidentally end up on the same physical infrastructure as untrusted agent execution environments. Step 3: Enforcing Network Isolation Compute isolation is useless if the untrusted code can scan your internal network or exfiltrate data to the public internet. We must deploy a strict NetworkPolicy to default-deny all egress traffic from our sandboxed namespace. YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-agent-egress namespace: isolated-agents spec: podSelector: matchLabels: app: agent-executor policyTypes: - Egress egress: # Only allow DNS resolution - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP # Allow outbound only to a specific internal API gateway if needed # - to: # - ipBlock: # cidr: 10.0.0.50/32 Step 4: Deploying the Sandboxed Workload With the network secured, we define the Kubernetes deployment. By setting the runtimeClassName to gvisor, Kubernetes routes the container lifecycle through the GKE Agent Sandbox rather than the standard container runtime. YAML apiVersion: apps/v1 kind: Pod metadata: generateName: dynamic-agent-task- namespace: isolated-agents labels: app: agent-executor spec: # Instruct GKE to use the Agent Sandbox (gVisor) runtimeClassName: gvisor # Ensure these pods only land on our Axion node pool nodeSelector: dedicated: untrusted-agents restartPolicy: Never containers: - name: execution-environment image: your-registry/agent-runtime:v1.0.0 env: - name: AGENT_PAYLOAD valueFrom: secretKeyRef: name: task-payload-secret key: payload # Drop all unnecessary Linux capabilities securityContext: runAsUser: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" volumeMounts: - name: temp-storage mountPath: /tmp volumes: - name: temp-storage emptyDir: {} Step 5: Orchestrating the Execution via Java Spring Boot To bring this architecture together, the control plane must dynamically spin up these sandboxed pods whenever an AI agent decides it needs to run code. In a modern distributed system, this is typically handled by a core backend microservice. Using the Fabric8 Kubernetes Client in a Java Spring Boot application provides a highly resilient way to orchestrate these ephemeral workloads programmatically. Java import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.client.KubernetesClient; import org.springframework.stereotype.Service; @Service public class AgentOrchestratorService { private final KubernetesClient kubernetesClient; public AgentOrchestratorService(KubernetesClient kubernetesClient) { this.kubernetesClient = kubernetesClient; } public String executeUntrustedCode(String tenantId, String pythonCode) { // 1. Create a Kubernetes Secret containing the code payload String secretName = createPayloadSecret(tenantId, pythonCode); // 2. Load the sandbox Pod template and inject the specific payload secret Pod sandboxedPod = kubernetesClient.pods() .inNamespace("isolated-agents") .load(getClass().getResourceAsStream("/k8s/agent-pod-template.yaml")) .item(); // 3. Launch the pod dynamically via the API server Pod runningPod = kubernetesClient.pods() .inNamespace("isolated-agents") .create(sandboxedPod); // 4. Await completion and extract the logs safely kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .waitUntilCondition(pod -> pod.getStatus().getPhase().equals("Succeeded") || pod.getStatus().getPhase().equals("Failed"), 30, java.util.concurrent.TimeUnit.SECONDS); String executionLogs = kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .getLog(); // 5. Clean up the ephemeral resources kubernetesClient.pods().delete(runningPod); kubernetesClient.secrets().withName(secretName).delete(); return executionLogs; } } The Defense in Depth Strategy This architecture relies on a strict defense in depth model. If an LLM hallucinates a malicious payload or a user deliberately attempts prompt injection to compromise the platform, the attacker faces multiple independent barriers. The code executes as a non-root user in a minimal Alpine environment with a read-only filesystem. Network access is completely blocked by native Kubernetes policies. Finally, any attempt to exploit kernel vulnerabilities is intercepted by the gvisor runtime boundary running on dedicated Axion hardware. By combining these layers, engineering teams can build and scale trustworthy Agentic AI platforms without risking the integrity of their core cloud infrastructure.