Enterprise-Wide Kubernetes

Kubernetes is being increasingly adopted by enterprises as a multi-cloud OS that provides common constructs for designing and managing all applications.

As enterprise Kubernetes adoption grows, a pattern we are seeing is that enterprises will have several Kubernetes clusters. As enterprise teams start using Kubernetes for business-critical applications they also realize the benefits of centralizing cloud-native operational skills. This often leads to the formation of a "platform" team that provides operations and governance for Kubernetes clusters and workloads across the enterprise.

In this post, we will discuss how these platform teams can design and operate Kubernetes clusters enterprise-wide.

A Kubernetes cluster is itself composed of several components. A significant business and technical advantage of betting on Kubernetes enterprise-wide is being able to compose your own "best-in-class" stack using certified components from the lush ecosystem nurtured by the Cloud Native Computing Foundation (CNCF).

The diagram below shows the major components of a Kubernetes stack. Each one of these is discussed further below:

Infrastructure

From market-priced cloud instances, to bare-metal servers for databases, to GPUs for machine learning — there are a number of choices for the compute, network, and storage to use for Kubernetes workloads. Almost all major infrastructure vendors and cloud providers are building support for Kubernetes. Some as a managed service purpose-built for their infrastructure, and others via Kubernetes plugins for networking (CNI) and storage (CSI). Will enterprises use a single vendor solution, or look to manage several heterogeneous clusters?

Kubernetes Distributions

StackSense research recently published a great post categorizing the different types of Kubernetes distributions available from enterprises vendors — from custom to 100% upstream. It's not atypical for enterprises to use a combination of these!

Kubernetes Enterprise Security

Security for Kubernetes is a large topic that spans image scanning, image provenance, runtime host and cluster scanning, secrets management, RBAC, access control, isolation, network segmentation, and audit trails. The good news is that solutions have rapidly matured and are enterprise-ready.

Kubernetes Logging

Log collection, rotation, aggregation, and streaming needs to be supported for both Kubernetes cluster components and application workloads. Another key consideration is how to provide the right logs to the right teams — can your existing solutions do that or are new solutions required?

Kubernetes Monitoring

As with logging, metrics need to be collected from both cluster components and applications running in clusters. Prometheus, a CNCF project, has become a de facto standard for collecting Kubernetes metrics. However, metrics need to be collected, and forwarded, and presented across clusters to the right teams and individuals. Another common question is how APM (Application Performance Management) tools fit it, and whether they are still needed.

Kubernetes Load-Balancing (Ingress)

A Kubernetes Ingress provides Layer 7 load-balancing for workloads within a cluster. An Ingress can also act as a reverse proxy to route requests to backend pods (service instances) based on HTTP path or header information. There are several choices available. However, the choice of Ingress will impact availability of advanced features like end-to-end encryption.

Build Tools and Version Control for Kubernetes

A key value proposition for adopting Kubernetes is to increase developer agility. This requires integration with continuous delivery and version control tools. Build orchestration tools like Jenkins can push changes to clusters. An emerging approach is to drive all infrastructure and cluster changes using version control tools, like Git. In this case, your infrastructure will pull changes from Git. Whether you choose a pull or a push approach, its important to design a change management pipeline for both images and Kubernetes manifests.

Image Registries

Container images are stored in a central registry, which allows different hosts to pull images. Popular registries, like Artifactory, Nexus, and Harbor, offer several features including scanning images for security purposes. It's also important to design image provenance systems so that only scanned and trusted images are deployed in production environments.

Application Management with Kubernetes

Kubernetes is not the end-game. The goal of running Kubernetes is to manage containerized applications. Managing applications requires integration of application metrics, health, state, accounting, security, troubleshooting and diagnostics, remediation, audit trails, managing image and manifest changes across a delivery pipeline, secure remote access, and several other related operational concerns.

Obviously, there are choices and trade-offs involved in composing a Kubernetes stack for enterprise use.

Nirmata, and our partners, are committed to helping the enterprise in their cloud-native journey. I am excited to announce that we are sponsoring a new BrightTalk webinar series. where we will be discussing each of the major topics posted in this blog. Each episode will be co-hosted by a domain expert as our featured guest.

Our first episode features Paulo Renato from the Microsoft Azure team. Paulo and I will discuss Kubernetes infrastructure from cloud providers and address questions like:

1. What are the enterprise options for selecting Kubernetes distributions?
2. When should an enterprise choose a managed Kubernetes service?
3. How should enterprises evaluate managed Kubernetes services from the top cloud providers?
4. What else matters for enterprise Kubernetes infrastructure?

Join us to gain insights into the value of Kubernetes for your enterprise applications! You can sign-up for free here.

If you are bringing Kubernetes to your enterprise, I would love to hear from you and meet you at one of our webinars or meetups!