In today’s technology landscape, Kubernetes plays a vital role. Container usage is only increasing, with 86 percent of IT leaders prioritizing containerization for more applications. (Cloud Container Adoption In The Enterprise, 2020) 

Kubernetes is an open-source container management system designed by Google that brings significant agility, automation, and optimization to those running containers in a DevOps environment.  

Thanks to Kubernetes’ extensibility, it’s able to meet the most diverse requirements and constraints in application development, and sits firmly in the prime spot as the most common open-source container orchestration framework.  

However, enterprise organizations who are operating and serving customers at scale need to consider how best to leverage the complexity of Kubernetes to fully take advantage of wider geolocated cloud features. 

Kubernetes is originally designed to support workload consolidation in a single cluster. However, there are many problem scenarios that require a multi-cluster approach to optimize performance and results. These can include workloads across regions, limiting outage blast radius, compliance issues, hard multitenancy, security, and specialized solutions. For example, enterprise-scale organizations might leverage multiple Kubernetes clusters spread across multiple clouds or hybrid clouds with on-premises and with one or more cloud providers.  

Due to the performance nature of modern cloud-native applications, Kubernetes environments need to be highly distributed. Proper multi-cluster management ensures consistent operations across all environments, enterprise-grade security, and workload management.  

Below is a visual representation of an example Kubernetes multi-cluster architecture:      

Leveraging multiple Kubernetes clusters can solve many of the aforementioned problem scenarios, but managing multiple clusters is a difficult task. As the cluster grows, the complexity in managing them also increases. There are multiple operations that occur in a well-managed Kubernetes cluster, such as adding/removing a node, securing the cluster, upgrading it from time to time, maintaining the cluster, etc.   

These operations' complex nature at scale requires a proper Kubernetes multi-cluster management system to monitor everything happening on all the clusters. Taking a best practice approach, SREs in operations teams can better monitor the health of the cluster environment and maintain application performance when multi-cluster architecture is more uniform. 

Kubernetes multi-cluster operations include: 

• Creating, updating, and deleting Kubernetes clusters across different environments (data centers; private, hybrid, and public clouds; and at the Edge) 
• Updating the control plane and compute nodes 
• Managing application life cycles across hybrid environments 
• Scaling, securing, and upgrading clusters (perhaps even provider-independent) 
• Maintaining and updating multiple nodes 
• Searching, finding, and modifying any Kubernetes resource 
• Implementing Role-Based Access Control (RBAC) over the clusters. (For example, an admin can have access to all clusters, but a developer might only need access to the dev cluster.) 
• Defining the resource quotas among the clusters 
• Creating pod budget policies 
• Creating network and governance policies 
• Defining taints and toleration on the clusters 
• Scanning for risks and vulnerabilities  

One of the important strategies for managing multiple Kubernetes clusters in the industry is to have a proper cluster isolation pattern. The isolation we're discussing here improves the security within an application.   

To achieve the isolation boundary, you must optimize the use of Kubernetes namespaces. These will help you in defining the scope of cluster usage and its resources by a team or application. Namespaces are a logical way to break a cluster up into logical constructs and also allow you to define RBAC; pod security policies; network policies (to separate workloads); and quotas.  

Here’s an example of this isolation in practice: You may need to deploy a three-service app. Service A and Service B should not be able to talk to each other, yet Service C needs to talk to both A and B. So then you could leverage the namespace and add an additional compliance level by keeping them in different clusters. 

There are many popular tools available for Kubernetes multi-cluster management. In no particular order, these include:  

• Cluster management 
  • D2iQ Kommander 
  • Rancher 
  • Red Hat Advanced Cluster Management 
• Deployment 
  • CloudBolt 
  • Google Anthos 
  • Helm 
  • Kubespray 
• Security and monitoring 
  • Aqua 
  • cAdvisor 
  • Falco 

Many of the above tools share the following features for managing multiple clusters with ease and visibility: 

• Central cluster management with automatic application delivery 
• Reduced operation cost by unifying all management 
• Increased application availability by deploying across distributed clusters 
• Centralized policy management and enforcement for ease of security compliance 
• Accelerated application development to production 
• Global DNS provision to distribute traffic across multiple clusters in a balanced way 

Automation and policy management are crucial to successfully managing multiple clusters in multiple environments. While it is not always possible to automate everything on day one, it should remain a top priority to automate as much of your cluster deployment and operations as possible as you move forward. 

Benefits of Multi-Cluster Adoption

Depending on the business or product requirements, you will need to optimize an appropriate multi-cluster strategy when considering the following: 

Risk Mitigation 

By applying a policy-based governance approach to a multi-cluster architecture, engineers can automatically monitor and ensure security and configuration controls meet industry compliance standards or self-imposed corporate standards in a desired state model. 

Geolocations or Cloud Locations 

A multi-cluster strategy can also help address the typical issues that administrators and site reliability engineers (SREs) face when working across a range of environments. Multi-clustering can optimize performance across multiple data centers, private clouds, and public clouds that run Kubernetes clusters.  

Leveraging multi-clusters around geolocation can improve performance for consumers as it reduces latency over regions and also facilitates better disaster recovery. 

Team-Based Structure 

One of the best practice approaches to working in Kubernetes is to create smaller multi-clusters and multiple environments (development and production) per cluster. That way, they will all be connected to each other over the organization network. The two main reasons for such an approach are to separate concerns and limit the blast radius of unforeseen events. This architectural framework approach sees the formation of multiple Kubernetes clusters across an organization. The advantage of this approach is that teams can leverage Kube features like service discovery across the wider organization while maintaining complete control of their own resources. 

Consider your project and business needs from the outset when thinking about adopting a multi-cluster strategy. For example, do you really need hard multitenancy? Hard multitenancy is an approach that assumes all tenants are high risk and as such requires the most stringent of responses, which will automatically require a multi-cluster approach. After that, consider your compliance requirements and if you have the operational capacity to handle the overheads of a multi-cluster architecture.  

Governance is about synchronizing clusters and enforcing centralized policy management. Kubernetes governance refers to a set of rules created through policies that need to be enforced on all Kube clusters. This is a critical component for enterprises being production-ready and working at scale in Kubernetes. Typically, this process means enforcing conformance rules across Kubernetes multi-clusters as well as applications running in those clusters.  

It would help if you had a solid plan in place to be production-ready at scale. Kubernetes governance may sound dull, but it’s going to pay off in the long run, especially if implemented across a large organization. Suppose you continue to grow the number of clusters in use without governance. In that case, they will exist in different pockets with different rules and implementations, which basically translates to a huge amount of extra time and work for your teams to manage in the near future. However, there are only a few very important components for creating successful Kubernetes governance, and these are outlined below.   

When considering a successful Kubernetes governance strategy, the first component is ensuring that good multi-cluster management and visibility are well in place. It is necessary to maintain control over how and where clusters are provisioned and configured, as well as which versions of software can be used. For visibility, ideally, you should be able to centrally view and manage clusters to better optimize resources and troubleshoot issues. Improved management practices and better visibility can also save you the headache of having to deal with a number of security risks and performance issues down the road.  

Next, you need to have an authentication and access management system. Having centralized authentication and authorization is going to help your organization in a big way. Not only will it make the process of login easier, but it’ll also help track who is doing what, so you can ensure the right people are performing the right tasks at any given time.  

Finally, for Kubernetes governance, you should optimize policy management. Think about how Kubernetes is going to impact your engineering culture and how to find the right balance for developers’ flexibility within it. Governance — with an appropriate level of flexibility — ensures that you can meet customer needs and deploy critical services in a uniform and consistent way. 

 The two governance dimensions consist of:  

• Policy scope: where a specific rule should be applied, enforced, or verified 
• Policy targets: what should be enforced and verified 

When you think about policy targets for governance, there can be many aspects. Here, we are discussing security policies, network management, access management, and image management.  

Limit User Access  

In security policies for Kubernetes governance, it’s important to limit users' access on the pods in the clusters. Users of the cluster should have well-defined access depending on the privilege, which is based on their role. To achieve this, implement security policies in place that will have rules and conditions related to access and privileges. In these policies, define that the containers should have only read access to the file system and enforce allowPrivilegeEscalation=false in the policy so that the containers and child processes cannot be subject to privilege changes.  

Network Policies

Network policies play a very important role in defining which service can talk to each other. Here, define which pod and service can communicate with each other and where the isolation must be established. This comes under the pod security in Kubernetes governance. These policies help you control the traffic inside the Kubernetes clusters. These network policies can be pod-based, namespace-based, or IP-based, depending on your requirement for governance. In these policies, you define entities such as spec, podSelector/namespaceSelector, policyTypes, ingress(rules defined for traffic), egress, etc.  

Access Administration and Management 

In access management, while configuring the RBAC (role-based access control) policy, admins need to take the responsibility to limit access to cluster resources. There are multiple Kubernetes objects such as Role, ClusterRole, RoleBinding, and ClusterRoleBinding, which helps you define access to the cluster resources in detail. 

Image Management 

Leveraging public Docker images can improve speed and agility when developing applications, however, there are a lot of vulnerable Docker images out there, and using them on the production cluster can be very risky.  

Image management is also a part of Kubernetes governance. Any and all images to be used on the cluster must be audited first for any vulnerabilities. There are a number of approaches to vulnerability scanning — how and where you check will depend on your preferred workflows. However, it is recommended that images be tested and passed before deploying on the cluster.  

Hacking activities have increased exponentially in recent years, and hackers keep on finding loopholes in the system to gain access for faulty intentions. So, be very vigilant when adopting best practices to ensure that you only use official, clean, and verified Docker images on the cluster. 

These policies are a must for better Kubernetes governance. After these policies and rules are in place, verify that they are all enforced in the organization as a part of internal compliance rules and policies. Monitor teams and projects comprehensively to ensure that everyone in the organization follows the policies defined and does not bypass them.   

Open Policy Agent (OPA), a CNCF project, is an excellent way to create and enforce such policies across an enterprise at scale. As shown below, every request will go through the OPA, and the request decision will be based on the policies defined for the Kubernetes cluster. If the request is policy-compliant, it will be executed. If the request is violating the policies defined, it will be rejected by the OPA. 

The complexity of managing a Kubernetes cluster only increases as the size of the cluster increases. Cluster sprawl can begin with multiple different application teams using it for different projects. The solution to this is identifying and implementing best practice governance early and quickly to avoid major leg work in the future. Implement multiple smaller clusters and a single centralized management system to keep track of all the clusters and optimize observability (for monitoring, alerting, capacity control, and cost management, etc.). 

This approach will help in reducing complexity and ensure that applications running on the clusters are highly available. You will also see an improvement in performance because you can deploy your application on a cluster in the nearest available region where your application needs to be deployed. Implementing governance early makes sure that everything is working as per rules defined in the policies and the employees of the organization comply with it.  

The key advantages of optimizing multi-cluster management and governance include: 

• Reduced complexity 
• Enhanced cluster visibility  
• Better application availability 
• Improved monitoring and logging 
• Ensured conformance and compliance 
• Reduced latency 
• Improved disaster recovery 
• Ability to deploy legacy and cloud-native applications across multiple cloud/hybrid environments 
• Enhanced cloud solutions 
• Standardization with built-in automation to reduce operational costs and improve efficiency  
• Better security posture (through visibility, centralized control, consistency, and standardization) 

There are many established vendors like D2iQ, Red Hat OpenShift, Google Anthos, Rancher, and VMware Tanzu that are capable of supporting you in the management and governance of Kubernetes multi-clusters at scale. These vendor offerings provide varying levels of openness, speed to deployment, cost of ownership characteristics, reliability, multi-cloud support, and the ability to help with smart (AI/ML) cloud-native applications. While any one of these vendors can help you reap the benefits and solve challenges faced while working with Kubernetes at scale, it’s best to do your due diligence before committing.  

For those interested in following an open-source path for multi-cluster management and governance (perhaps to reduce OPEX costs), there are a lot of open-source tools available. Tools like K9s, Kontena Lens, WKSctl, cAdvisor, and Weave Scope can help you overcome the basic challenges involved in managing and governing Kubernetes multi-clusters.  

The benefits of investing in a Kubernetes management platform lie predominantly in the cost savings an organization can experience by making sure everything in production runs smoothly and that applications can run at scale across the globe without any hiccups. Optimizing such a tool can also help you reduce manual effort amongst your engineers through the multiple automated solutions available in their features. Here too, the ability of any one of these vendors to help you reduce the workforce power involved and the bandwidth required to manage multiple Kubernetes clusters across a large-scale enterprise can vary. Research and test before selecting.  

So, when considering how to manage multi-cloud, multi-distribution clusters, DevOps teams should take a look at these platforms as an alternative to overcome the pain of multi-cluster management and governance on their own. 

2020. Cloud Container Adoption In The Enterprise. [PDF] Forrester Consulting. Available at: <https://www.capitalone.com/tech/cloud-container-adoption-report/> [Accessed 30 December 2020].