{{announcement.body}}
{{announcement.title}}
Refcard #292

Advanced Kubernetes

Kubernetes is a distributed cluster technology that manages container-based systems in a declarative manner using an API. There are currently many learning resources to get started with the fundamentals of Kubernetes, but there is less information on how to manage Kubernetes infrastructure on an ongoing basis. This Refcard aims to deliver quick, accessible information for operators using any Kubernetes product.

Published: Mar. 14, 2019    |    Modified: Jan. 16, 2020
6,560

Brought to you by

Mesosphere
Free PDF for easy Reference
refcard cover

Written by

author avatar Chris Gaun PM, Mesosphere
asset cover
Refcard #292

Advanced Kubernetes

Kubernetes is a distributed cluster technology that manages container-based systems in a declarative manner using an API. There are currently many learning resources to get started with the fundamentals of Kubernetes, but there is less information on how to manage Kubernetes infrastructure on an ongoing basis. This Refcard aims to deliver quick, accessible information for operators using any Kubernetes product.

Published: Mar. 14, 2019    |    Modified: Jan. 16, 2020
6,560
Free PDF for easy Reference

Written by

author avatar Chris Gaun PM, Mesosphere

Brought to you by

Mesosphere
Table of Contents

About Kubernetes

Basics

Kubectl

Managing Nodes

Test Plan

Security and Troubleshooting

And More

Section 1

About Kubernetes

Kubernetes is a distributed cluster technology that manages container-based systems in a declarative manner using an API. Kubernetes' core open-source source code is governed by the Cloud Native Computing Foundation (CNCF) and counts all the largest cloud providers and software vendors as its contributors. There are currently many learning resources to get started with the fundamentals of Kubernetes, but there is less information on how to manage Kubernetes infrastructure on an ongoing basis. This Refcard aims to deliver quickly accessible information for operators using any Kubernetes product.

For an organization to deliver and manage Kubernetes clusters to every line of business and developer groups, ops needs to architect and manage both the core Kubernetes container orchestration and the necessary auxiliary solutions — e.g. monitoring, logging, and the CI/CD pipeline.

Gartner predicts that by 2020, more than 50% of global organizations will be running containerized applications in production, up from less than 20% today. This research provides an overview and actionable advice for organizations implementing modern containerized apps, data services, and machine learning.

According to 451 Research, the application container market size was $762 million last year and will grow by over 3.5x to $2.7 billion by 2020. Organizations can prepare for the rapid growth in containerized applications, real-time data, and machine learning by taking steps to properly architect the network. Kubernetes differs from the orchestration offered by configuration management solutions in that it provides a declarative API that collects, stores, and processes events in an eventually consistent manner.

A few traits of Kubernetes include:

  • Abstraction: Kubernetes abstracts the application orchestration from the infrastructure resource and as-a-service automation. This allows organizations to focus on the APIs of Kubernetes to manage an application at scale in a highly available manner instead of the underlying infrastructure resources.

  • Declarative: Kubernetes' control plane decides how the hosted application is deployed and scaled on the underlying fabric. A user simply defines the logical configuration of the Kubernetes object, and the control plane takes care of the implementation.

  • Immutable: Different versions of services running on Kubernetes are completely new and not swapped out. Objects in Kubernetes, say different versions of a pod, are changed by creating new objects.


This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 2

Basics

Simplified Kubernetes architecture and components

Control Plane, Notes, and Persistent Storage

Kubernetes' basic architecture requires a number of components.

API Server: The main way to communicate with Kubernetes clusters is through the Kubernetes API Server. A user (and other Kubernetes components) sends declarative events to the Kubernetes API Server through HTTPS (port 443). The API Server then processes events and updates the Kubernetes persistent data store — e.g. etcd. The API Server also performs authentication and authorization. For more information, see the Kubernetes documentation here.

etcd Cluster: As Kubernetes collects declarative events, it needs to store them. Kubernetes uses etcd for this persistent storage, providing a single source of truth of all Kubernetes objects. etcd requires an odd number of nodes. A loss of etcd storage results in a loss of the cluster's state, so etcd should be backed up for disaster recovery. For more information, see the Kubernetes documentation here.

Controller Manager: Kubernetes uses a control plane to perform non-terminating control loops to observe the state of Kubernetes objects and reconcile it with the desired state. For more information, see the Kubernetes documentation here.

Kubelet (Agent): Kubernetes clusters have a worker node called a Kubelet that has several functions. For example, Kubelets create Pods from the Podspec. For more information, see the Kubernetes documentation here.

Scheduler: Kubernetes relies on a sophisticated algorithm to schedule Kubernetes objects. The scheduling takes into account filters such as resources, topology, and volume, then uses prioritization setting, such as affinity rules to provision pods on particular Kubernetes worker nodes. For more information, see the Kubernetes documentation here.

Kube-Proxy: Each Kubernetes cluster worker nodes have a network proxy for connectivity. For more information, see the Kubernetes documentation here.


This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 3

Kubectl

Below are some useful commands for IT professionals getting started with Kubernetes. A full list of Kubectl commands can be found at the reference documentation.

Making Life Easier

Finding Kubernetes command short name:

 kubectl describe 

You can find out more about using Kubectl Aliases here and context switching among Kubernetes clusters here.


This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 4

Managing Nodes

Sometimes, it is necessary to perform maintenance on underlying nodes. In those cases, it is important to use eviction and make sure the application owners have set a pod disruption budget.

To properly evict a node, use:

  • Cordon node:  kubectl cordon $NODENAME 
  • Drain node:  kubectl drain 
    • Respects  PodDisruptionBudgets 
  • Uncord node:  kubectl uncordon $NODENAME 

This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 5

Test Plan

According to the CNCF, most Kubernetes clusters are still provisioned and managed software distributions (instead of public cloud provider options). For these clusters, once a new cluster is up and running, it is imperative to devise a test plan and design a run book to make sure it is operational.

For new public cloud providers or new versions of Kubernetes on a provider, it is also important to run a testing plan. Below is testing plan used as part of D2iQ's testing of Kubernetes. After each step, validate that the cluster is working properly.

Test Clusters

  1. Provision a highly available Kubernetes cluster with 100 Kubernetes nodes.
  2. Scale that cluster down to 30 nodes after it has finished provisioning.
  3. Provision 3 additional highly available Kubernetes clusters with 5 nodes.
  4. Scale all 3 clusters (in parallel) to 30 nodes simultaneously.
  5. Provision 16 more highly available Kubernetes clusters with 30 nodes.
  6. Kill 5 Kubernetes nodes on a single cluster simultaneously.
  7. Kill 3 control-plane nodes on a single cluster (fewer if the nodes will not automatically reprovision).
  8. Kill the  etcd  leader.

Test Workloads

  1. Provision storage (potentially using CSI and a specific driver).
    • Test different provider specific storage features.
  2. Run the e2e cluster loader with higher end of pods per node (35-50).
  3. Run Kubernetes conformance testing to stress the cluster.
  4. Provision Helm to the Kubernetes cluster.
  5. Test specific Kubernetes workloads (using Helm charts).
    • Deploy Nginx
    • Deploy Redis
    • Deploy Postgres
    • Deploy RabbitMQ
  6. Deploy services with  type=loadbalancer  to test load balancer automation.
    • Test a different provider's specific load balancer features.
  7. Expose a service using Ingress.

This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 6

Security and Troubleshooting

Troubleshooting With Kubectl

The Kubernetes command line kubectl provides many of the resources needed in order to debug clusters.

CHECK PERMISSIONS

TROUBLESHOOTING COMMAND EXAMPLE
Check your permissions $ kubectl auth can-i $ kubectl auth can-i create deployments --namespace dev
Check permissions of other users $ kubectl auth can-i [Options] $ kubectl auth can-i create deployments --namespace dev chris

PENDING CHECK PERMISSIONS

troubleshooting command
General issue checking $ kubectl describe pod <name of pending pod>
Check to see if pod is using too many resources $ kubectl top pod
Check node resources $ kubectl top node
Get node resources $ kubectl get nodes -o yaml | grep
Get all pod resources $ kubectl top pod --all-namespaces --containers=true

You can also remove pods from the environment.

WAITING ON INCORRECT IMAGE INFORMATION

troubleshooting command
Check YAML URL spec:
containers:
— name: example
image: url:port/image:v
Pull an image onto desktop to determine whether registry and image information is correct $ docker pull <image>

Also check that the secret information is correct.

CRASH LOOPING DEPLOYMENT

  • Use ctrl + C to exit the crash loops
  • Troubleshoot
  • Roll back deployment
troubleshooting command
General issue checking $ kubectl describe deployments
Rolling back deployment $ kubectl rollout undo [Deployment Name]
Pausing a deployment $ kubectl rollout pause [Deployment Name]

This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 7

And More

The Advanced Kubernetes Refcard also includes:

  • General Debugging
  • Troubleshooting With Curl
  • Feeding Results to Other Things
  • Troubleshooting With jq
  • Security

Important Resources


This is a preview of the Advanced Kubernetes Refcard. To read the entire Refcard, please download the PDF from the link above.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}