Please Don’t Evict My Pod: QoS Class

It was the second day of the long weekend; I was watching Money Heist on Netflix (a good one to watch, free recommendation by a human), and in-between, I got the Slack notification on one channel, “Is something wrong with our application?” By the time I started my MacBook to check, another Slack message came: "Application is not running, pod status says Evicted.” Luckily, the application was in a pre-prod environment, but we want the pod up and running. 

It looks like an effortless task; restart the pod, right? We did, but the pod was evicted again in some time. The application code was simple and shouldn’t be consuming many resources of the K8s cluster. Frankly, before this, we hadn't encountered such an issue, and we weren’t sure about the solution, and there start a saga of pod eviction resolution. 

Eviction is a complex topic with many factors, objects and policies involved. Let’s get started with basics and slowly build the related corresponding concepts. In this first post, I cover the pod phase, status, container status, state, some commands, sample K8 objects to imitate the eviction and QoS class of the pods, and correlating all of them.

What Do You Mean by Evicted?

Is evicted a pod phase or status? How is pod status is related to container status, and what is the container state? Don’t worry; we will try to answer all of these questions. Let’s start with the easiest and most generic one first, the pod phase. It is a simple, high-level summary of where the pod is in its lifecycle. Below are the five possible pod phase values

• Pending: accepted by the Kubernetes, either not scheduled by the Kube-scheduler or downloading the container image.
• Running: pod bounded to a node, and all the containers are running.
• Failed: all containers in the pod have terminated, and at least one container execution leads to failure. Failure means task ends with a non-zero status, or the Kubernetes system killed it.
• Succeeded: all containers terminated with zero status, and the pod will not restart.
• Unknown: failed to determine the state of the pod.

From the programmatical perspective, phase is just a String type instance variable in the object PodStatus. The below sketch depicts the relationship between various object types.

Let’s explore the next one, pod status  (PodStatus). PodStatus is an aggregated view of all the containers within a pod. The PodStatus object consists of a bunch of array instance variables based on ContainerStatus. Also, it's worth noting some other important instance variables like reason andqosClass. Finally, I want to highlight two important points:

• The information reported as Pod status depends on the current  ContainerState.
• The instance variable reason value is a brief CamelCase message indicating details about why the pod is in this state. 

Hold on, did you read the last line thoroughly? Read it once more and keep a note of it. Now, we are moving further to explore container status and state. The ContainerStatus object represents complete information about a container in a pod. Among several instance variables, it’s worth noticing the state one; data type of state is ContainerState. There are only three permissible value of container state,

• Waiting: the default state of the container. Pulling an image, applying a secret, and several other operations happen in this state.
• Running: indicates the container is running without any issue.
• Terminated: container completed the execution and stopped running. Either the container execution is successful, i.e. zero exit code or failed because of some issue. Object ContainerStateTerminated represents the terminated state. This object has exitCode & reason instance variables, which show what happened with the container. Evicted is one of the values of the reason variable. Here is the source code for the Evicted container state.

Command to Get Evicted Pods

kubectl get po -a --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted"))

Simulate Pods Eviction

---

apiVersion: v1

kind: Namespace

metadata:

  name: evict-example

---

apiVersion: scheduling.k8s.io/v1

kind: PriorityClass

metadata:

  name: high-priority

  namespace: evict-example

value: 1000000

globalDefault: false

description: "This priority class should be used for XYZ service pods only."

---

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2

kind: Deployment

metadata:

  namespace: evict-example

  name: pod-with-defined-resources-example

spec:

  selector:

    matchLabels:

      app: pod-with-defined-resources-example

  replicas: 3 # tells deployment to run 3 pods matching the template

  template:

    metadata:

      labels:

        app: pod-with-defined-resources-example

    spec:

      priorityClassName: high-priority

      containers:

      - name: pod-with-defined-resources-example

        image: abhioncbr/pod-evict:latest

        imagePullPolicy: IfNotPresent

        resources:

          requests:

            memory: "570Mi"

            cpu: "100m"

          limits:

            memory: "570Mi"

            cpu: "100m"

---

apiVersion: v1 # for versions before 1.9.0 use apps/v1beta2

kind: Pod

metadata:

  namespace: evict-example

  name: pod-oom-killed-example

spec:

  restartPolicy: Never

  containers:      

    - name: pod-oom-killed-example

      image: abhioncbr/pod-evict:latest

      imagePullPolicy: IfNotPresent

      resources:

        requests:

          memory: "10Mi"

          cpu: "100m"

        limits:

          memory: "10Mi"

          cpu: "100m"

---

apiVersion: v1 # for versions before 1.9.0 use apps/v1beta2

kind: Pod

metadata:

  namespace: evict-example

  name: pod-evict-burstable-example

spec:

  restartPolicy: Never

  containers:

  - name: pod-evict-burstable-example

    image: abhioncbr/pod-evict:latest

    imagePullPolicy: IfNotPresent

    resources:

       requests:

         memory: "30Mi"

         cpu: "100m"

       limits:

         memory: "1000Mi"

         cpu: "100m"

---

apiVersion: v1 # for versions before 1.9.0 use apps/v1beta2

kind: Pod

metadata:

  namespace: evict-example

  name: pod-evict-best-effort-example

spec:

  restartPolicy: Never

  containers:

  - name: pod-evict-best-effort-example

    image: abhioncbr/pod-evict:latest

    imagePullPolicy: IfNotPresent

The code file will create the following Kubernetes objects

• evict-example namespace.
• high-priority PriorityClass.
• pod-with-defined-resources-example Deployment with 3 replicas, each with resource request & limits 570Mi
• pod-oom-killed-example pod.
• pod-evict-burstable-example pod.
• pod-evict-best-effort-example pod.

The Command for Creating and Deleting Objects

xxxxxxxxxx

kubectl apply -f pod-evict-resources.yaml # for creation of objects. kubectl get pods -n evict-example         # for listing all pods. 

kubectl delete all --all -n evict-example # for deletion of objects.

All these resources are created with the perspective of simulating eviction in a node having approx 2GB of memory resources. Please adjust the memory values according to your needs. Also, the image used for the container on the pod is simple Python based abhioncbr/pod-evict. 

Note: The Python script contains an infinite loop for continuous more memory usage.

xxxxxxxxxx

import time

some_str = ''

while True:

 some_str = some_str + ' * hello * ' * 1024000

 print(some_str)

 # sleep for sometime

 print('Before: %s' % time.ctime())

 time.sleep(20)

 print('After: %s\n' % time.ctime())

# sleep again for sometime

print('Before: %s' % time.ctime())

time.sleep(360)

print('After: %s\n' % time.ctime())

How Kubelet Decides to Evict the Pod

Kubelet evicts pods based on the QoS. What the hell is QoS? You may have noticed a variable qosClass in object PodStatus. Is it the same? It is! Okay, let's explore it further to find out what it means and what part it played in pod eviction. Before that, we have to visit the resources request and limit for the pods.

• Requests: The minimum resources required by the container. If an available node resource is less than the request values, the pod will stay Pending. Requests are at the container level, and if two more containers are present, than the total resources required will be the sum of both container request.
• Limits: The maximum resources of a node that a pod can utilize. If a container tries to consume more than the mentioned limits, pod gets terminated with error reason OOMKilled or CPU throttled.

Based on the presence, non-presence of the resources and their values, Kubernetes assign three different classes of QoS to the pod. The following are the three classes.

• Guaranteed: All the containers of a pod have requests and limits, and their values request and limit values are equal too.
• Burstable: At least one container in a pod with either request or limits values and not satisfying the Guaranteed class condition as Burstable.
• BestEffort: All containers in a pod without any requests and limits configuration falls into the BestEffort class.

If you check pods created above in the simulation step, all three types of pods are present. You can check by using the following command.

xxxxxxxxxx

kubectl get pods -n evict-example 

kubectl describe pod pod-with-defined-resources-XXXXXXX 

kubectl describe pod pod-evict-burstable-example 

kubectl describe pod pod-evict-best-effort-example

Cool, we made the understanding of resources and limits and how Kubernetes determines the QoS class of a pod. Only one last thing left, how it is related to eviction?

Kubelet evicts pod in series, first the BestEffort one than Burstable one and in last Guaranteed one.

• BestEffort: pods are the first ones considered for eviction
• Burstable: second ones to be considered after BestEffort.
• Guaranteed: least likely to be evicted in-case of eviction.

In our case, our application pod was the Burstable type. We converted it into the Guaranteed type and saved our evening of the long weekend.

Our quest for finding the details about why Kubelet started pod eviction continued for several weeks. I know this post turned to be a long one, and I am concluding it here. In my next post, I will be covering more about soft and hard eviction thresholds signals, pod disruption budget, priority class, resource quotas, limit ranges and more. Thanks for reading and looking for feedback.