Please Don’t Evict My Pod: QoS Class
If you've recently recieved a notification that your pod was evicted, you might be wondering why. Take a look at what you can do about it here.
Join the DZone community and get the full member experience.Join For Free
It was the second day of the long weekend; I was watching Money Heist on Netflix (a good one to watch, free recommendation by a human), and in-between, I got the Slack notification on one channel, “Is something wrong with our application?” By the time I started my MacBook to check, another Slack message came: "Application is not running, pod status says Evicted.” Luckily, the application was in a pre-prod environment, but we want the pod up and running.
It looks like an effortless task; restart the pod, right? We did, but the pod was evicted again in some time. The application code was simple and shouldn’t be consuming many resources of the K8s cluster. Frankly, before this, we hadn't encountered such an issue, and we weren’t sure about the solution, and there start a saga of pod eviction resolution.
Eviction is a complex topic with many factors, objects and policies involved. Let’s get started with basics and slowly build the related corresponding concepts. In this first post, I cover the pod phase, status, container status, state, some commands, sample K8 objects to imitate the eviction and QoS class of the pods, and correlating all of them.
What Do You Mean by Evicted?
Is evicted a pod phase or status? How is pod status is related to container status, and what is the container state? Don’t worry; we will try to answer all of these questions. Let’s start with the easiest and most generic one first, the pod phase. It is a simple, high-level summary of where the pod is in its lifecycle. Below are the five possible pod phase values
- Pending: accepted by the Kubernetes, either not scheduled by the Kube-scheduler or downloading the container image.
- Running: pod bounded to a node, and all the containers are running.
- Failed: all containers in the pod have terminated, and at least one container execution leads to failure. Failure means task ends with a non-zero status, or the Kubernetes system killed it.
- Succeeded: all containers terminated with zero status, and the pod will not restart.
- Unknown: failed to determine the state of the pod.
From the programmatical perspective,
phase is just a String type instance variable in the object PodStatus. The below sketch depicts the relationship between various object types.
Let’s explore the next one, pod status (PodStatus). PodStatus is an aggregated view of all the containers within a pod. The PodStatus object consists of a bunch of array instance variables based on ContainerStatus. Also, it's worth noting some other important instance variables like
qosClass. Finally, I want to highlight two important points:
- The information reported as Pod status depends on the current
- The instance variable
reasonvalue is a brief CamelCase message indicating details about why the pod is in this state.
Hold on, did you read the last line thoroughly? Read it once more and keep a note of it. Now, we are moving further to explore container status and state. The ContainerStatus object represents complete information about a container in a pod. Among several instance variables, it’s worth noticing the
state one; data type of
state is ContainerState. There are only three permissible value of container state,
- Waiting: the default state of the container. Pulling an image, applying a secret, and several other operations happen in this state.
- Running: indicates the container is running without any issue.
- Terminated: container completed the execution and stopped running. Either the container execution is successful, i.e. zero exit code or failed because of some issue. Object ContainerStateTerminated represents the terminated state. This object has
reasoninstance variables, which show what happened with the container. Evicted is one of the values of the
reasonvariable. Here is the source code for the Evicted container state.
Command to Get Evicted Pods
Simulate Pods Eviction
The code file will create the following Kubernetes objects
pod-with-defined-resources-exampleDeployment with 3 replicas, each with resource request & limits 570Mi
The Command for Creating and Deleting Objects
All these resources are created with the perspective of simulating eviction in a node having approx 2GB of memory resources. Please adjust the memory values according to your needs. Also, the image used for the container on the pod is simple Python based
Note: The Python script contains an infinite loop for continuous more memory usage.
How Kubelet Decides to Evict the Pod
Kubelet evicts pods based on the
QoS. What the hell is
QoS? You may have noticed a variable
qosClass in object PodStatus. Is it the same? It is! Okay, let's explore it further to find out what it means and what part it played in pod eviction. Before that, we have to visit the resources request and limit for the pods.
- Requests: The minimum resources required by the container. If an available node resource is less than the request values, the pod will stay Pending. Requests are at the container level, and if two more containers are present, than the total resources required will be the sum of both container request.
- Limits: The maximum resources of a node that a pod can utilize. If a container tries to consume more than the mentioned limits, pod gets terminated with error reason
Based on the presence, non-presence of the resources and their values, Kubernetes assign three different classes of QoS to the pod. The following are the three classes.
- Guaranteed: All the containers of a pod have requests and limits, and their values request and limit values are equal too.
- Burstable: At least one container in a pod with either request or limits values and not satisfying the Guaranteed class condition as Burstable.
- BestEffort: All containers in a pod without any requests and limits configuration falls into the BestEffort class.
If you check pods created above in the simulation step, all three types of pods are present. You can check by using the following command.
Cool, we made the understanding of resources and limits and how Kubernetes determines the QoS class of a pod. Only one last thing left, how it is related to eviction?
Kubelet evicts pod in series, first the BestEffort one than Burstable one and in last Guaranteed one.
- BestEffort: pods are the first ones considered for eviction
- Burstable: second ones to be considered after BestEffort.
- Guaranteed: least likely to be evicted in-case of eviction.
In our case, our application pod was the Burstable type. We converted it into the Guaranteed type and saved our evening of the long weekend.
Our quest for finding the details about why Kubelet started pod eviction continued for several weeks. I know this post turned to be a long one, and I am concluding it here. In my next post, I will be covering more about soft and hard eviction thresholds signals, pod disruption budget, priority class, resource quotas, limit ranges and more. Thanks for reading and looking for feedback.
Opinions expressed by DZone contributors are their own.