Kustomize: A Tool for Kubernetes Configuration Management

Kubernetes (k8s) is an increasingly popular container orchestration platform. With that popularity has grown an ecosystem of tools and frameworks. One such tool is Kustomize.

K8s desired states are declaratively described in resource description files such as deployment, services, configmaps, secrets, etc. As we vary the environments from dev to QA to prod these configurations to change. The database may change from an embedded H2 in dev to a MariaDB in QA to Oracle in prod, the amount of allocated memory for a container may change, secrets of user rid and password for the database may also change. Managing all these varying YAML files of deployment, services, etc. per environment can be quite a chore. Configuration management tools maintain consistency across these environments.

Kustomize is a standalone configuration management tool. The basic idea is to create a collection of these configuration files as a ‘base’ configuration and apply ‘overlays’ to change them by the environment. Kustomize has been integrated with kubectl. All this is best understood through a simple example. I will assume that you have a running instance of Docker and Kubernetes. Just to be specific, this example was developed on Docker Desktop with k8s running on WSL2. I will also assume that you have kubectl installed and pointing to this k8s cluster.

Setting Up

First, create a Spring Boot project with Actuator and web dependencies. While there are many ways to do the following steps, we will be very traditional. Then, generate an image of the application:

./mvnw spring-boot:build-image -Dspring-boot.build-image.imageName=spring-k8s/kustomizedemo

 Check that the image has been created:

Start the image and make sure everything works and then stop the container.

docker run -p 8080:8080 --name kustomizedemo  -t spring-k8s/kustomizedemo

curl http://localhost:8080/actuator/health/liveness

{"status":"UP"}

docker stop kustomizedemo

Run the Image With Kustomize

In the project directory, create a directory cms/base.

Now run:

kubectl create deployment kustomizedemo --image spring-k8s/kustomizedemo -o yaml --dry-run=client > cms/base/deployment.yaml

Since the image is available only locally and not in Dockerhub we change the imagePullPolicy to Never. We also set the memory allocation and log level. The modified deployment.yaml look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: kustomizedemo
  name: kustomizedemo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kustomizedemo
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kustomizedemo
    spec:
      containers:
      - image: spring-k8s/kustomizedemo:latest
        name: kustomizedemo
        resources:
          requests:
            cpu: 100m
            memory: "128M"
        env:
        - name: LOG_LEVEL
          value: "DEBUG"
        imagePullPolicy: Never
status: {}

We cannot access this from outside the cluster. So, we need to create a service resource:

kubectl create service clusterip  kustomizedemo  --tcp 80:8080 -o yaml --dry-run=client > cms/base/service.yaml

And test it:

kubectl port-forward svc/gs-spring-boot-k8s 9090:80

curl http://localhost:9090/actuator/health/liveness
{"status":"UP"}

Now we need to create a kustomize file in cms/base:

kustomization.yaml

---
resources:
- services.yaml
- deployment.yaml

Now let us deploy the app to Kubernetes using Kustomize:

Run ‘kubectl get all’ to see that we have pod/kustomizedemo and corresponding services, deployment, and replicaset.

On to Production

It is time to make changes for production. We want to change the replicas to 2, change the cpu allocation, and the log level to INFO. Change to the overlays/production directory. Create a new deployment.yaml file with these changes:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustomizedemo
  namespace: default
spec:
  replicas: 2
  template:
    spec:
      containers:
      - image: spring-k8s/kustomizedemo:latest
        name: kustomizedemo
        resources:
          requests:
            cpu: 250m
            memory: "256M"
        env:
        - name: LOG_LEVEL
          value: "INFO"
        imagePullPolicy: Never

Note that we have only specified the changes in this version.

Now create a kustomization.yaml file:

---
bases:
- ../../base
patchesStrategicMerge:
- deployment.yaml

Now let us deploy it to k8s:

Now do a kubectl get all to that there are have two pods of kustomizedemo running

Check the logs by doing kubectl logs pod/kustomizedemo-<id> and see that only INFO items appear.

Closing Thoughts

Helm is another commonly used tool that provides roughly the same functionality. Kustomize scores points for its sheer familiarity. It uses the same ol’ same ol’ YAML files and it is integrated with kubectl. But the price for that is that it is not a complete tool. It uses Git to hold the base and overlays. Helm has its own repository structure and it should remind users of other template engines like JSTL or Thymeleaf. But it has a learning curve. See the reference for more details.