{{announcement.body}}
{{announcement.title}}

Please Don’t Evict My Pod: QoS Class

DZone 's Guide to

Please Don’t Evict My Pod: QoS Class

If you've recently recieved a notification that your pod was evicted, you might be wondering why. Take a look at what you can do about it here.

· Cloud Zone ·
Free Resource

It was the second day of the long weekend; I was watching Money Heist on Netflix (a good one to watch, free recommendation by a human), and in-between, I got the Slack notification on one channel, “Is something wrong with our application?” By the time I started my MacBook to check, another Slack message came: "Application is not running, pod status says Evicted.” Luckily, the application was in a pre-prod environment, but we want the pod up and running. 

It looks like an effortless task; restart the pod, right? We did, but the pod was evicted again in some time. The application code was simple and shouldn’t be consuming many resources of the K8s cluster. Frankly, before this, we hadn't encountered such an issue, and we weren’t sure about the solution, and there start a saga of pod eviction resolution. 

Eviction is a complex topic with many factors, objects and policies involved. Let’s get started with basics and slowly build the related corresponding concepts. In this first post, I cover the pod phase, status, container status, state, some commands, sample K8 objects to imitate the eviction and QoS class of the pods, and correlating all of them.

What Do You Mean by Evicted?

Is evicted a pod phase or status? How is pod status is related to container status, and what is the container state? Don’t worry; we will try to answer all of these questions. Let’s start with the easiest and most generic one first, the pod phase. It is a simple, high-level summary of where the pod is in its lifecycle. Below are the five possible pod phase values

  • Pending: accepted by the Kubernetes, either not scheduled by the Kube-scheduler or downloading the container image.
  • Running: pod bounded to a node, and all the containers are running.
  • Failed: all containers in the pod have terminated, and at least one container execution leads to failure. Failure means task ends with a non-zero status, or the Kubernetes system killed it.
  • Succeeded: all containers terminated with zero status, and the pod will not restart.
  • Unknown: failed to determine the state of the pod.

From the programmatical perspective, phase is just a String type instance variable in the object PodStatus. The below sketch depicts the relationship between various object types.

Relationships between object types

Let’s explore the next one, pod status  (PodStatus). PodStatus is an aggregated view of all the containers within a pod. The PodStatus object consists of a bunch of array instance variables based on ContainerStatus. Also, it's worth noting some other important instance variables like reason andqosClass. Finally, I want to highlight two important points:

  • The information reported as Pod status depends on the current  ContainerState.
  • The instance variable reason value is a brief CamelCase message indicating details about why the pod is in this state. 

Hold on, did you read the last line thoroughly? Read it once more and keep a note of it. Now, we are moving further to explore container status and state. The ContainerStatus object represents complete information about a container in a pod. Among several instance variables, it’s worth noticing the state one; data type of state is ContainerState. There are only three permissible value of container state,

  • Waiting: the default state of the container. Pulling an image, applying a secret, and several other operations happen in this state.
  • Running: indicates the container is running without any issue.
  • Terminated: container completed the execution and stopped running. Either the container execution is successful, i.e. zero exit code or failed because of some issue. Object ContainerStateTerminated represents the terminated state. This object has exitCode & reason instance variables, which show what happened with the container. Evicted is one of the values of the reason variable. Here is the source code for the Evicted container state.

Command to Get Evicted Pods

Shell
 




x


 
1
kubectl get po -a --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted"))



Simulate Pods Eviction

YAML
 




x
92


 
1
---
2
apiVersion: v1
3
kind: Namespace
4
metadata:
5
  name: evict-example
6
---
7
apiVersion: scheduling.k8s.io/v1
8
kind: PriorityClass
9
metadata:
10
  name: high-priority
11
  namespace: evict-example
12
value: 1000000
13
globalDefault: false
14
description: "This priority class should be used for XYZ service pods only."
15
---
16
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
17
kind: Deployment
18
metadata:
19
  namespace: evict-example
20
  name: pod-with-defined-resources-example
21
spec:
22
  selector:
23
    matchLabels:
24
      app: pod-with-defined-resources-example
25
  replicas: 3 # tells deployment to run 3 pods matching the template
26
  template:
27
    metadata:
28
      labels:
29
        app: pod-with-defined-resources-example
30
    spec:
31
      priorityClassName: high-priority
32
      containers:
33
      - name: pod-with-defined-resources-example
34
        image: abhioncbr/pod-evict:latest
35
        imagePullPolicy: IfNotPresent
36
        resources:
37
          requests:
38
            memory: "570Mi"
39
            cpu: "100m"
40
          limits:
41
            memory: "570Mi"
42
            cpu: "100m"
43
---
44
apiVersion: v1 # for versions before 1.9.0 use apps/v1beta2
45
kind: Pod
46
metadata:
47
  namespace: evict-example
48
  name: pod-oom-killed-example
49
spec:
50
  restartPolicy: Never
51
  containers:      
52
    - name: pod-oom-killed-example
53
      image: abhioncbr/pod-evict:latest
54
      imagePullPolicy: IfNotPresent
55
      resources:
56
        requests:
57
          memory: "10Mi"
58
          cpu: "100m"
59
        limits:
60
          memory: "10Mi"
61
          cpu: "100m"
62
---
63
apiVersion: v1 # for versions before 1.9.0 use apps/v1beta2
64
kind: Pod
65
metadata:
66
  namespace: evict-example
67
  name: pod-evict-burstable-example
68
spec:
69
  restartPolicy: Never
70
  containers:
71
  - name: pod-evict-burstable-example
72
    image: abhioncbr/pod-evict:latest
73
    imagePullPolicy: IfNotPresent
74
    resources:
75
       requests:
76
         memory: "30Mi"
77
         cpu: "100m"
78
       limits:
79
         memory: "1000Mi"
80
         cpu: "100m"
81
---
82
apiVersion: v1 # for versions before 1.9.0 use apps/v1beta2
83
kind: Pod
84
metadata:
85
  namespace: evict-example
86
  name: pod-evict-best-effort-example
87
spec:
88
  restartPolicy: Never
89
  containers:
90
  - name: pod-evict-best-effort-example
91
    image: abhioncbr/pod-evict:latest
92
    imagePullPolicy: IfNotPresent


The code file will create the following Kubernetes objects

  • evict-example namespace.
  • high-priority PriorityClass.
  • pod-with-defined-resources-example Deployment with 3 replicas, each with resource request & limits 570Mi
  • pod-oom-killed-example pod.
  • pod-evict-burstable-example pod.
  • pod-evict-best-effort-example pod.

The Command for Creating and Deleting Objects

Shell
 




xxxxxxxxxx
1


1
kubectl apply -f pod-evict-resources.yaml # for creation of objects. kubectl get pods -n evict-example         # for listing all pods. 
2
kubectl delete all --all -n evict-example # for deletion of objects.



All these resources are created with the perspective of simulating eviction in a node having approx 2GB of memory resources. Please adjust the memory values according to your needs. Also, the image used for the container on the pod is simple Python based abhioncbr/pod-evict

Note: The Python script contains an infinite loop for continuous more memory usage.

Python
 




xxxxxxxxxx
1
16


1
import time
2
 
          
3
some_str = ''
4
while True:
5
 some_str = some_str + ' * hello * ' * 1024000
6
 print(some_str)
7
 
          
8
 # sleep for sometime
9
 print('Before: %s' % time.ctime())
10
 time.sleep(20)
11
 print('After: %s\n' % time.ctime())
12
 
          
13
# sleep again for sometime
14
print('Before: %s' % time.ctime())
15
time.sleep(360)
16
print('After: %s\n' % time.ctime())


How Kubelet Decides to Evict the Pod

Kubelet evicts pods based on the QoS. What the hell is QoS? You may have noticed a variable qosClass in object PodStatus. Is it the same? It is! Okay, let's explore it further to find out what it means and what part it played in pod eviction. Before that, we have to visit the resources request and limit for the pods.

  • Requests: The minimum resources required by the container. If an available node resource is less than the request values, the pod will stay Pending. Requests are at the container level, and if two more containers are present, than the total resources required will be the sum of both container request.
  • Limits: The maximum resources of a node that a pod can utilize. If a container tries to consume more than the mentioned limits, pod gets terminated with error reason OOMKilled or CPU throttled.

Based on the presence, non-presence of the resources and their values, Kubernetes assign three different classes of QoS to the pod. The following are the three classes.

  • Guaranteed: All the containers of a pod have requests and limits, and their values request and limit values are equal too.
  • Burstable: At least one container in a pod with either request or limits values and not satisfying the Guaranteed class condition as Burstable.
  • BestEffort: All containers in a pod without any requests and limits configuration falls into the BestEffort class.

If you check pods created above in the simulation step, all three types of pods are present. You can check by using the following command.

Shell
 




xxxxxxxxxx
1


 
1
kubectl get pods -n evict-example 
2
kubectl describe pod pod-with-defined-resources-XXXXXXX 
3
kubectl describe pod pod-evict-burstable-example 
4
kubectl describe pod pod-evict-best-effort-example



Cool, we made the understanding of resources and limits and how Kubernetes determines the QoS class of a pod. Only one last thing left, how it is related to eviction?

Kubelet evicts pod in series, first the BestEffort one than Burstable one and in last Guaranteed one.

  • BestEffort: pods are the first ones considered for eviction
  • Burstable: second ones to be considered after BestEffort.
  • Guaranteed: least likely to be evicted in-case of eviction.

In our case, our application pod was the Burstable type. We converted it into the Guaranteed type and saved our evening of the long weekend.

Our quest for finding the details about why Kubelet started pod eviction continued for several weeks. I know this post turned to be a long one, and I am concluding it here. In my next post, I will be covering more about soft and hard eviction thresholds signals, pod disruption budget, priority class, resource quotas, limit ranges and more. Thanks for reading and looking for feedback.

Topics:
cloud, cloud native, cloud native and kubernetes, distributed systems, kubectl, kubelet, kubernetes, pod eviction

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}