Managing Secrets Deployment in GitOps Workflow

Introduction

Kubernetes is an open-source system that helps to automate the deployment, autoscaling, and management of container-based applications. It is widely used for building and deploying cloud-native applications on a massive scale, leveraging the elasticity of the cloud. Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Amazon Kubernetes Service (AKS) are the popular managed services for running a production-grade, highly available Kubernetes cluster on AWS, GCP, and Azure without needing to stand up or to maintain the Kubernetes control plane.

There are many ways to spin up Kubernetes clusters either on-prem or on a public cloud (AWS, GCP, and Azure), but in today’s automation world, it is a common practice to use a continuous delivery pipeline to deploy releases to their Kubernetes cluster. As a best practice, all the YAML manifests should be version-controlled in a Git repository.

In this post, I will walk through the approach of sealing secrets using kubeseal to manage the deployment of sensitive information to their Kubernetes clusters so that you can achieve a typical GitOps workflow.

Continuous Delivery Using GitOps

GitOps is a term coined by WeaveWorks in 2017 and is a way to do Kubernetes cluster management and application delivery. GitOps uses Git as a single source of truth to achieve continuous delivery. As illustrated in the architecture below, a Weave Flux agent runs in the Kubernetes cluster and watches the Git repository and image registries continuously. If there are changes to deployment artifacts, they are pushed to this config repository, or a new image is pushed to the image registry by a continuous integration system such as Jenkins, GitLab, GitHub Actions, etc. The Weave Flux agent automatically pulls these changes down and updates the relevant deployment to the cluster.

A Kubernetes cluster has different types of resources, including DaemonSets, ReplicaSets, ConfigMaps, Secrets, etc. A Secret is a resource that helps us to store sensitive information, such as passwords, OAuth tokens, and SSH keys. There are multiple ways to access secrets as part of a deployment. For instance, we can mount the secrets as data volumes or can expose them as environment variables to the containers in a Kubernetes Pod. This way, we can decouple Pod deployment from managing sensitive data needed by the containerized applications within a Pod.

Here, the challenge is to integrate these Secrets in a typical GitOps workflow by storing secrets' YAML templates outside the cluster, in a Git repository. Many times, developers accidentally add sensitive information into their Git repositories.

How to Solve the Problem

Sealed Secrets is an open-source project that helps to address this challenge by encrypting a Secret object so that we can version control them in a private or public repository. We can use kubectl to deploy these encrypted Secrets in a Kubernetes cluster.

How It Actually Works

As per the documentation published over GitHub, Sealed Secrets comprises the following components:

1. A controller, which is deployed to the K8S cluster
2. A command line wrapper called kubeseal
3. And lastly, a customer resource definition (CRD) called SealedSecret

The job of the controller is to look for a private/public key pair across a cluster and generate a new 4096-bit RSA key pair in case it does not exist. The private key is persisted in a Secret object in the same namespace as that of the controller. The public key portion of this is made publicly available to anyone wanting to use Sealed Secrets with this cluster.

Then, kubeseal is used for creating a SealedSecret custom resource definition (CRD) from a Secret resource definition using the public key. Kubeseal can communicate with the controller through the Kubernetes API server and retrieve the public key needed for encrypting a Secret at runtime. The public key may also be downloaded from the controller and saved locally to be used offline.

Installing kubeseal Client

For kubeseal's recent releases and detailed installation instructions, refer to the Kubeseal Releases. To install kubeseal on Linux x86_64 systems:

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.10.0/kubeseal-linux-amd64 -O /usr/bin/kubeseal

chmod 755 /usr/bin/kubeseal

Installing the Custom Controller and CRD for SealedSecret

You can run the command below to install the controller and the SealedSecret custom resource definition in the kube-system namespace. Additionally, it will create a Service Account and RBAC artifacts such as Role/ClusterRole and RoleBinding/ClusterRoleBinding.

xxxxxxxxxx

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.1/controller.yaml

kubectl apply -f controller.yaml

This is the output of the apply command:

You can validate whether the sealed-secrets secret is created or not with the command:

xxxxxxxxxx

kubectl get secrets -n kube-system | grep -i sealed

Actually, the controller searches for a Secret with the label sealedsecrets.bitnami.com/sealed-secrets-key in its namespace, and in case it does not exist, it creates a new one in its namespace and prints the public key portion of the key pair to its output logs. You can check for this secret with the command:

xxxxxxxxxx

Kubectl describe secret sealed-secrets-key7ftkm -n kube-system

You can see the contents of this Secret, which contains the public/private key pair in YAML format, with the following command:

xxxxxxxxxx

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml

Sealing the Secrets

kubeseal encrypts the Secret using the public key that it fetches at runtime from the controller running in the Kubernetes cluster. If a user does not have direct access to the cluster, then a cluster administrator may retrieve the public key from the controller logs and make it accessible to the user.

Suppose you have the database-secrets.yaml file with below contents:

xxxxxxxxxx

apiVersion: v1

kind: Secret

metadata:

  name: database-secrets

  namespace: dev

type: kubernetes.io/basic-auth

stringData:

  password: WyWa2=?

  username: YWRtaW4=

You can use the below command to generate the SealedSecret CRD using public key:

xxxxxxxxxx

kubeseal --format=yaml --cert=public-key-cert.pem < database-secret.yaml > database-sealed-secret.yaml

Next you can apply SealedSecret CRD with command:

xxxxxxxxxx

kubectl apply -f database-sealed-secret.yaml

Once the SealedSecret CRD is created in the cluster, the controller becomes aware of it and unseals the underlying Secret using the private key and deploys it to the same namespace. This is seen by looking at the logs from the controller: