Building Internal Developer Platforms on Kubernetes: The Abstraction Problem Nobody Warns You About

Introduction

The meeting that changed the platform team's direction was not a technical one. It was a conversation with a product engineer who had been at the company for eight months and had never successfully deployed to production without help from someone on the platform team. Not because she lacked skill. 

She was smart, experienced, and had successfully launched production systems at two previous jobs, but getting a working service into production meant dealing with fifteen different configuration files across four repositories, figuring out how Helm values files and Kustomize overlays worked together, and knowing which of the three CI pipeline templates to use based on whether the service needed a sidecar, a job scheduler, or neither.

She had read the documentation. It was accurate. It just didn't tell her what to do when the documented path didn't match the state of her specific service in her environment. The platform team had built powerful infrastructure. They had not built a usable platform.

That distinction between infrastructure and platform is where most Kubernetes-based internal developer platform efforts go wrong, and it's worth being precise about what it means.

Infrastructure vs Platform: A Practical Distinction

Infrastructure is the machinery: the Kubernetes clusters, the networking layer, the CI pipelines, the secrets management system, and the monitoring stack. A platform is the interface that makes that machinery accessible to developers who aren't Kubernetes experts without requiring them to become ones. 

The confusion between the two produces a situation that's extremely common in engineering organizations: a technically sophisticated infrastructure that's effectively only usable by the people who built it.

The test for whether you have a platform or just infrastructure is simple: can a developer who joined three months ago deploy a new service to production without asking anyone for help? Not by following a tutorial someone wrote last year that may or may not still be accurate, but through tooling that guides them through a current, correct process. If the answer is no, you have infrastructure. The platform is the missing layer.

This statement is not an argument against complexity in the underlying system. Kubernetes is complex, and that complexity exists for beneficial reasons: flexibility, programmability, and a rich ecosystem. The platform layer should absorb the complexity, rather than exposing it to every developer who needs to ship a service.

What the First Attempt Got Wrong

The infrastructure team built the first version of the internal platform in their spare time, juggling it with other priorities. It consisted of a set of Helm chart templates, a GitHub Actions workflow library, and a wiki with deployment instructions. 

This approach is how most internal platforms start, and it has a predictable failure mode: the templates encode the assumptions of the people who wrote them, the wiki goes stale within weeks, and the gap between the documented process and the actual state of the infrastructure grows invisibly until it becomes a significant tax on every developer who hits it.

The fundamental mistake was treating platform work as documentation work rather than product work. A wiki is not a platform. A set of templates that require understanding to use correctly is not a platform. A platform is software that makes the correct path the easy path, that validates inputs before they cause problems downstream, and that fails loudly and helpfully rather than silently and mysteriously.

The second attempt started from a different premise: the platform is a product, developers are its users, and the measure of success is whether they can do their jobs without needing the platform team.

The Abstraction Layer: Custom Resources and Admission Webhooks

The technical decision that made the most difference was introducing a custom resource definition (CRD) that represented a service in the platform's domain model, not a Kubernetes Deployment or Service, but a higher-level construct that encoded the platform's opinionated defaults and generated the underlying Kubernetes objects from a much simpler specification.

# Platform-level CRD: what developers actually write
apiVersion: platform.company.com/v1
kind: AppService
metadata:
  name: payment-api
  namespace: production
spec:
  image: payment-api:v1.4.2
  tier: backend          # drives resource limits, network policy
  replicas: 3
  port: 8080
  healthCheck: /healthz
  env:
    DATABASE_URL:
      secretRef: payment-db-credentials

This twelve-line manifest replaced the hundred-plus lines of Kubernetes YAML that developers had previously been required to write and maintain. The controller running in the cluster, a standard Kubernetes operator built with controller-runtime, read the AppService resource and generated the Deployment, Service, HorizontalPodAutoscaler, PodDisruptionBudget, and NetworkPolicy that the platform's standards required, with defaults applied consistently across every service.

The key design decision was what to expose in the CRD and what to hide. The tier field is a prime example: rather than exposing resource requests and limits directly, which requires understanding what values are appropriate for the cluster, the CRD accepts a tier label (frontend, backend, worker, batch) that maps to a predefined resource profile. 

A backend tier service receives a specific CPU and memory allocation appropriate for the cluster's node types. A batch tier service receives a different profile with different eviction priorities. The developer specifies intent; the platform enforces the appropriate configuration.

# Controller logic: tier maps to resource profile (Go pseudocode)
func resourceProfileForTier(tier string) corev1.ResourceRequirements {
    profiles := map[string]corev1.ResourceRequirements{
        "frontend": {
            Requests: corev1.ResourceList{
                corev1.ResourceCPU:    resource.MustParse("100m"),
                corev1.ResourceMemory: resource.MustParse("128Mi"),
            },
            Limits: corev1.ResourceList{
                corev1.ResourceCPU:    resource.MustParse("500m"),
                corev1.ResourceMemory: resource.MustParse("256Mi"),
            },
        },
        "backend": {
            Requests: corev1.ResourceList{
                corev1.ResourceCPU:    resource.MustParse("250m"),
                corev1.ResourceMemory: resource.MustParse("256Mi"),
            },
            Limits: corev1.ResourceList{
                corev1.ResourceCPU:    resource.MustParse("1000m"),
                corev1.ResourceMemory: resource.MustParse("512Mi"),
            },
        },
        // batch, worker profiles follow same pattern
    }
    return profiles[tier]
}

Admission webhooks complemented the CRD by catching misconfiguration before it reached the cluster. A validating webhook checked every AppService manifest against a set of rules: the image tag must not be 'latest,' the health verification path must respond within the cluster, and secret references must exist in the target namespace and return a clear error message describing exactly what was wrong and how to resolve it. This shifted error detection from 'runtime, after deployment' to 'submission time, before anything breaks,' which dramatically reduced the debugging load on both developers and the platform team.

The Golden Path and Its Limits

The CRD and controller approach works well when services fit the platform's model. Here's where things became challenging: not all services fit the platform's model. A service that needed custom init containers, a service that required a specific affinity rule because of a hardware dependency, and a batch job with a complex retry policy that didn't map cleanly to the tier abstraction. Each of these required either extending the CRD or breaking the abstraction and falling back to raw Kubernetes YAML.

The temptation is to keep extending the CRD to cover every case. Resist it. A CRD that tries to expose every Kubernetes feature is just a more complicated way to write Kubernetes YAML, and it loses the simplicity that made the abstraction valuable. 

The better model is a golden path, the CRD for the 80% of services that fit the standard model, and a documented escape hatch for the 20% that don't. The escape hatch is raw Kubernetes resources, maintained by the teams that need them, with the platform team providing support rather than ownership.

The key is being honest with developers about which path they're on. A service using the AppService CRD gets platform-managed defaults, automatic updates when the platform evolves, and first-class support. A service using raw Kubernetes resources owns its own configuration and gets best-effort support. That distinction in the support model is what makes the trade-off legible rather than arbitrary.

What I'd Do Differently

In hindsight, the most important investment was the admission webhook, and I'd build it earlier. The CRD and controller took significant time to design and implement. 

The webhook could have been built in a few days and would have immediately improved the developer experience by catching misconfiguration at submission time rather than deployment time. Validation before generation is higher-leverage than generation that might produce something invalid.

I'd also measure platform adoption from day one. Which teams are using the AppService CRD? Which teams are on raw Kubernetes? What's the conversion rate of new services to the platform abstraction? Without that data, the platform team relies on intuition instead of evidence to guide their investment. 

The teams that adopt slowly are often the ones with the most valuable feedback about where the abstraction doesn't fit, and they're also the teams most likely to be quietly maintaining fragile custom configurations that will become incidents later.

When should you not build a CRD-based platform abstraction? If you have fewer than fifteen to twenty engineers deploying services, the overhead of designing, building, and maintaining a CRD-based platform abstraction almost certainly exceeds the value. 

Helm charts and excellent templates get you most of the way there with a fraction of the complexity. The operator pattern earns its cost when you have enough services that inconsistency becomes a real operational problem when the differences between how services are configured start causing incidents and nobody can tell you why a particular service is configured the way it is.

Key Takeaways

Infrastructure and platform are different things. Infrastructure is the machinery; a platform is the interface that makes machinery accessible without requiring expertise in its internals. Most Kubernetes-based IDPs stop at infrastructure.

Custom resource definitions let you define a domain model that encodes your platform's opinions. Developers specify intent (tier, replicas, port); the controller generates the correct Kubernetes objects with consistent defaults applied.

Admission webhooks shift error detection from runtime to submission time. A clear error message at kubectl apply is worth more than a mysterious pod crash two minutes later.

Maintain a golden path for the majority of services and a documented escape hatch for the rest. A CRD that tries to cover every Kubernetes feature loses the simplicity that justified building it.

Conclusion

The platform engineer's job is to make complexity disappear, not by eliminating it, but by absorbing it into tooling so that the people building products don't have to carry it. 

That's a harder problem than building the infrastructure itself, and it requires a fundamentally different mindset: less systems engineering, more product thinking. Who are the users? What tasks do they need to accomplish? Where does the current experience fail them?

The teams building internal developer platforms who get the process right tend to look, from the outside, like they have unusually productive engineering organizations. Individual contributions ship faster, incidents caused by misconfiguration drop, and the platform team spends less time on support and more time on improvements. The causal chain runs directly from platform quality to engineering output, even though it's usually measured differently.

The open question is whether the CRD-based abstraction model scales to genuinely heterogeneous service fleets, the kinds of organizations where services span multiple languages, multiple deployment patterns, and multiple infrastructure dependencies. 

The golden path works when most services look similar enough that a shared abstraction is useful. What occurs to the platform model when 40% of services utilize the escape hatch? At that point, is the abstraction still earning its cost, or is it adding complexity without delivering the simplicity it promised?