Kubernetes Namespaces, Resource Quota, and Limits for QoS in Cluster
Check out this post and learn to apply namespaces, resource quotas, and limits for sharing your clusters across different environments in Kubernetes.
Join the DZone community and get the full member experience.
Join For Free
By default, all resources in Kubernetes cluster are created in a default namespace. A pod will run with unbounded CPU and memory requests/limits.
A Kubernetes namespace allows one to partition created resources into a logically named group. Each namespace provides:
- a unique scope for resources to avoid name collisions
- policies to ensure appropriate authority to trusted users
- ability to specify constraints for resource consumption
This allows a Kubernetes cluster to share resources by multiple groups and provide different levels of QoS for each group.
Resources created in one namespace are hidden from other namespaces. Multiple namespaces can be created, each potentially with different constraints.
Default Kubernetes Namespace
By default, each resource created by the user in a Kubernetes cluster runs in a default namespace, called default.
./kubernetes/cluster/kubectl.sh get namespace
NAME LABELS STATUS AGE
default <none> Active 1m
kube-system <none> Active 1mAny pod, service, or replication controller will be created in this namespace. kube-system namespace is reserved for resources created by the Kubernetes cluster.
More details about the namespace can be seen:
./kubernetes/cluster/kubectl.sh describe namespaces default
Name: default
Labels: <none>
Status: Active
No resource quota.
Resource Limits
Type Resource Min Max Request Limit Limit/Request
---- -------- --- --- ------- ----- -------------
Container cpu - - 100m - -This description shows resource quota (if present), as well as resource limit ranges.
So, let’s create a Couchbase replication controller as:
./kubernetes/cluster/kubectl.sh run couchbase --image=arungupta/couchbaseCheck the existing replication controller:
./kubernetes/cluster/kubectl.sh get rc
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
couchbase couchbase arungupta/couchbase run=couchbase 1 5mBy default, only resources in user namespace are shown. Resources in all namespaces can be shown using --all-namespaces option:
./kubernetes/cluster/kubectl.sh get rc --all-namespaces
NAMESPACE CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
default couchbase couchbase arungupta/couchbase run=couchbase 1 5m
kube-system heapster-v11 heapster gcr.io/google_containers/heapster:v0.18.4 k8s-app=heapster,version=v11 1 6m
kube-system kube-dns-v9 etcd gcr.io/google_containers/etcd:2.0.9 k8s-app=kube-dns,version=v9 1 6m
kube2sky gcr.io/google_containers/kube2sky:1.11
skydns gcr.io/google_containers/skydns:2015-10-13-8c72f8c
healthz gcr.io/google_containers/exechealthz:1.0
kube-system kube-ui-v4 kube-ui gcr.io/google_containers/kube-ui:v4 k8s-app=kube-ui,version=v4 1 6m
kube-system l7-lb-controller-v0.5.2 default-http-backend gcr.io/google_containers/defaultbackend:1.0 k8s-app=glbc,version=v0.5.2 1 6m
l7-lb-controller gcr.io/google_containers/glbc:0.5.2
kube-system monitoring-influxdb-grafana-v2 influxdb gcr.io/google_containers/heapster_influxdb:v0.4 k8s-app=influxGrafana,version=v2 1 6m
grafana beta.gcr.io/google_containers/heapster_grafana:v2.1.1As you can see, the arungupta/couchbase image runs in the default namespace. All other resources run in the kube-system namespace.
Let's check the context of this replication controller:
./kubernetes/cluster/kubectl.sh config view couchbase
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://104.197.10.200
name: couchbase-on-kubernetes_kubernetes
contexts:
- context:
cluster: couchbase-on-kubernetes_kubernetes
user: couchbase-on-kubernetes_kubernetes
name: couchbase-on-kubernetes_kubernetes
current-context: couchbase-on-kubernetes_kubernetes
kind: Config
preferences: {}
users:
- name: couchbase-on-kubernetes_kubernetes
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: 1RUrsvA5RDwwRNf0eOvz86elmniOK0oj
- name: couchbase-on-kubernetes_kubernetes-basic-auth
user:
password: cZ9fZSuzIqq5kdnj
username: adminLook for contexts.context.name attribute to see the existing context. This will be manipulated later.
Create a Resource in New Kubernetes Namespace
Let's create a new namespace first. This can be done using the following configuration file:
apiVersion: v1
kind: Namespace
metadata:
name: development
labels:
name: developmentNamespace is created as:
./kubernetes/cluster/kubectl.sh create -f myns.yaml
namespace "development" createdThen querying for all the namespaces gives:
./kubernetes/cluster/kubectl.sh get namespace
NAME LABELS STATUS AGE
default <none> Active 9m
development name=development Active 13s
kube-system <none> Active 8mA new replication controller can be created in this new namespace by using --namespace option:
./kubernetes/cluster/kubectl.sh --namespace=development run couchbase --image=arungupta/couchbase
replicationcontroller "couchbase" createdList of resources in all namespaces looks like:
./kubernetes/cluster/kubectl.sh get rc --all-namespaces
NAMESPACE CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
default couchbase couchbase arungupta/couchbase run=couchbase 1 4m
development couchbase couchbase arungupta/couchbase run=couchbase 1 2m
kube-system heapster-v11 heapster gcr.io/google_containers/heapster:v0.18.4 k8s-app=heapster,version=v11 1 31m
. . .As seen, there are two replication controllers with arungupta/couchbase image–one in default namespace and another in development namespace.
Set Kubernetes Namespace For an Existing Resource
If a resource is already created then it can be assigned a namespace.
On a previously created resource, new context can be set in the namespace:
./kubernetes/cluster/kubectl.sh config set-context dev --namespace=development --cluster=couchbase-on-kubernetes_kubernetes --user=couchbase-on-kubernetes_kubernetes
context "dev" set.Viewing the context now shows:
./kubernetes/cluster/kubectl.sh config view couchbase
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://104.197.10.200
name: couchbase-on-kubernetes_kubernetes
contexts:
- context:
cluster: couchbase-on-kubernetes_kubernetes
user: couchbase-on-kubernetes_kubernetes
name: couchbase-on-kubernetes_kubernetes
- context:
cluster: couchbase-on-kubernetes_kubernetes
namespace: development
user: couchbase-on-kubernetes_kubernetes
name: dev
current-context: couchbase-on-kubernetes_kubernetes
kind: Config
preferences: {}
users:
- name: couchbase-on-kubernetes_kubernetes
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: 1RUrsvA5RDwwRNf0eOvz86elmniOK0oj
- name: couchbase-on-kubernetes_kubernetes-basic-auth
user:
password: cZ9fZSuzIqq5kdnj
username: adminThe second attribute in contexts.context array shows that a new context has been created. It also shows that the current context is still couchbase-on-kubernetes_kubernetes. Since no namespace is specified in that context, it belongs to the default namespace.
Change the context:
./kubernetes/cluster/kubectl.sh config use-context dev
switched to context "dev".See the list of replication controllers:
./kubernetes/cluster/kubectl.sh get rc
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGEObviously, no replication controllers are running in this context. Let's create a new replication controller in this new namespace:
./kubernetes/cluster/kubectl.sh run couchbase --image=arungupta/couchbase
replicationcontroller "couchbase" createdAnd, see the list of replication controllers in all namespaces:
./kubernetes/cluster/kubectl.sh get rc --all-namespaces
NAMESPACE CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
default couchbase couchbase arungupta/couchbase run=couchbase 1 16m
development couchbase couchbase arungupta/couchbase run=couchbase 1 4s
kube-system heapster-v11 heapster gcr.io/google_containers/heapster:v0.18.4 k8s-app=heapster,version=v11 1 17m
. . .Now you can see two arungupta/couchbase replication controllers running in two difference namespaces.
Delete a Kubernetes Resource in Namespace
A resource can be deleted by fully-qualifying the resource name:
./kubernetes/cluster/kubectl.sh --namespace=default delete rc couchbase
replicationcontroller "couchbase" deletedSimilarly, the other replication controller can be deleted as:
./kubernetes/cluster/kubectl.sh --namespace=development delete rc couchbase
replicationcontroller "couchbase" deletedFinally, see the list of all replication controllers in all namespaces:
./kubernetes/cluster/kubectl.sh get rc --all-namespaces
NAMESPACE CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
kube-system heapster-v11 heapster gcr.io/google_containers/heapster:v0.18.4 k8s-app=heapster,version=v11 1 3h
kube-system kube-dns-v9 etcd gcr.io/google_containers/etcd:2.0.9 k8s-app=kube-dns,version=v9 1 3h
. . .This confirms that all user created replication controllers are deleted.
Resource Quota and Limit Using Kubernetes Namespace
Each namespace can be assigned a resource quota.
By default, a pod will run with unbounded CPU and memory requests/limits. Specifying a quota allows one to restrict how much of the cluster resources can be consumed across all pods in a namespace.
A resource quota can be specified using a configuration file:
apiVersion: v1
kind: ResourceQuota
metadata:
name: quota
spec:
hard:
cpu: "20"
memory: 1Gi
pods: "10"
replicationcontrollers: "20"
resourcequotas: "1"
services: "5"The following resources are supported by the quota system:
This resource quota can be created in a namespace:
./kubernetes/cluster/kubectl.sh --namespace=development create -f quota.yaml
resourcequota "quota" createdThe created quota can be seen as:
./kubernetes/cluster/kubectl.sh --namespace=development describe quota
Name: quota
Namespace: development
Resource Used Hard
-------- ---- ----
cpu 0 20
memory 0 1Gi
pods 0 10
replicationcontrollers 0 20
resourcequotas 1 1
services 0 5Now, if you try to create the replication controller that works:
./kubernetes/cluster/kubectl.sh --namespace=development run couchbase --image=arungupta/couchbase
replicationcontroller "couchbase" createdBut, describing the quota again shows:
./kubernetes/cluster/kubectl.sh --namespace=development describe quota
Name: quota
Namespace: development
Resource Used Hard
-------- ---- ----
cpu 0 20
memory 0 1Gi
pods 0 10
replicationcontrollers 1 20
resourcequotas 1 1
services 0 5We expected a new pod to be created as part of this replication controller, but it’s not there. So let's describe our replication controller:
./kubernetes/cluster/kubectl.sh --namespace=development describe rc
Name: couchbase
Namespace: development
Image(s): arungupta/couchbase
Selector: run=couchbase
Labels: run=couchbase
Replicas: 0 current / 1 desired
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
1m 24s 4 {replication-controller } FailedCreate Error creating: Pod "couchbase-" is forbidden: must make a non-zero request for memory since it is tracked by quota.By default, pod consumes all the CPU and memory available. With resource quotas applied, an explicit value must be specified. Alternatively, a default value for the pod can be specified using the following configuration file:
apiVersion: v1
kind: LimitRange
metadata:
name: limits
spec:
limits:
- default:
cpu: 200m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 256Mi
type: ContainerThis restricts the CPU and memory that can be consumed by a pod. Let's apply these limits as:
./kubernetes/cluster/kubectl.sh --namespace=development create -f limits.yaml
limitrange "limits" createdNow, when you describe the replication controller again, it shows:
./kubernetes/cluster/kubectl.sh --namespace=development describe rc
Name: couchbase
Namespace: development
Image(s): arungupta/couchbase
Selector: run=couchbase
Labels: run=couchbase
Replicas: 1 current / 1 desired
Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
8m 2m 14 {replication-controller } FailedCreate Error creating: Pod "couchbase-" is forbidden: must make a non-zero request for memory since it is tracked by quota.
2m 2m 1 {replication-controller } SuccessfulCreate Created pod: couchbase-gzk0lThis shows a successful creation of the pod.
And now, when you describe the quota, it shows correct values as well:
./kubernetes/cluster/kubectl.sh --namespace=development describe quota
Name: quota
Namespace: development
Resource Used Hard
-------- ---- ----
cpu 100m 20
memory 268435456 1Gi
pods 1 10
replicationcontrollers 1 20
resourcequotas 1 1
services 0 5Resource Quota provide more details about how to set/update these values.
Creating another quota gives the following error:
./kubernetes/cluster/kubectl.sh --namespace=development create -f quota.yaml
Error from server: error when creating "quota.yaml": ResourceQuota "quota" is forbidden: limited to 1 resourcequotasSpecifying Limits During Pod Creation
Limits can be specified during pod creation as well:
If memory limit for each pod is restricted to 1g, then a valid pod definition would be:
apiVersion: v1
kind: Pod
metadata:
name: couchbase-pod
spec:
containers:
- name: couchbase
image: couchbase
ports:
- containerPort: 8091
resources:
limits:
cpu: "1"
memory: 512MiThis is because the pod request 0.5G of memory only. And, an invalid pod definition would be:
apiVersion: v1
kind: Pod
metadata:
name: couchbase-pod
spec:
containers:
- name: couchbase
image: couchbase
ports:
- containerPort: 8091
resources:
limits:
cpu: "1"
memory: 2GThis is because the pod requests 2G of memory. Creating such a pod gives the following error:
./kubernetes/cluster/kubectl.sh --namespace=development create -f couchbase-pod.yaml
Error from server: error when creating "couchbase-pod.yaml": Pod "couchbase-pod" is forbidden: unable to admit pod without exceeding quota for resource memory: limited to 1Gi but require 2805306368 to succeedHope you can apply namespaces, resource quotas, and limits for sharing your clusters across different environments.
Published at DZone with permission of Arun Gupta, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments