Securing Kubernetes Secrets With Conjur
Kubernetes uses base 64 encoding for storing them, which is not enough. You have to implement a number of security best practices to prevent possible security breaches.
Join the DZone community and get the full member experience.Join For Free
Why Secure Kubernetes Secrets?
Secrets management is one of the important aspects of securing your Kubernetes cluster. Out of the box, Kubernetes uses base 64 encoding for storing them, which is not enough. You have to implement a number of security best practices on top, to prevent possible security breaches. Etcd encryption at rest, access control with RBAC, are a couple of examples of the same. Using secrets management solutions like CyberArk Conjur, not only secures them for Kubernetes but also provides other benefits as we will see in the post.
What Is Conjur?
CyberArk Conjur is a secrets manager. It helps you manage secrets in Kubernetes, as well as across applications, tools, and clouds. It offers Role-Based Access Control (RBAC) with an audit trail to easily track each stored secret. It implements encryption at rest with AES-256-GCM and in transit using mTLS. Additionally, you can manage the access for each secret and can also rotate the secrets automatically.
In this post, we will see how to install Conjur OSS on Kubernetes. We will go through a basic set of Conjur policies and will load them into Conjur. We’ll also see how to run an application in Kubernetes which uses secrets from Conjur by conforming to the defined policies.
- Familiarity with advanced YAML concepts:
You may be already familiar with the way Kubernetes spec files are written in YAML. Although you also need to understand a few more YAML concepts to understand and define Conjur policies, viz. tags, anchors, and aliases. The Conjur website has a quick refresher on this. Alternatively, you can go through the full YAML documentation.
- A working Kubernetes cluster.
- Docker installed locally.
How to Install Conjur
The easiest way to Install Conjur on a Kubernetes cluster is by using the Helm chart. Let's first create a custom values file for the Helm chart.
dataKey is used for encrypting the secrets in the DB. The
ssl.altNames will be used for the SSL configuration of the Conjur service that the Helm chart will create.
Install Conjur OSS on a Kubernetes cluster, with the following commands.
VERSION declared above is the Conjur Helm chart release version. As of writing this post, the latest Conjur OSS Helm chart version is
2.0.3. Refer to Conjur Helm chart releases for the latest Conjur Helm chart available.
Once the helm chart is installed, it creates an admin user. You will need this key for the initial load of the Conjur policies, secrets, etc. You'll also need it in the "break-glass" scenarios. Hence you need to store it in a safe place. You can fetch the same using the commands below.
Verify the installation.
Define Conjur Policies
Conjur policies help define objects in its database in a tree structure. Some examples of the objects defined in the policies are users, roles, secrets, and applications. It also defines rules for role-based access control. While the Conjur documentation defines the policy best practices, we will use one of the Conjur demo repositories to define policies. I've used policies in the demo repository as the base and have further simplified them to understand the basic concepts better. Download and review the simplified policy files from my repository. Note that all the policies need to have a
1_users.ymlfile defines users and roles. It also grants role-based access to a group of users as well as to individual users.
2_app_authn-def.ymlfile defines applications and groups them into a layer for easier access management.
3_cluster-authn-svc-def.ymlfile defines the authenticator service and the SSL certificates for the mTLS communication between Conjur and its clients. In this case, Conjur clients are applications running on Kubernetes. It also defines role-based access to authenticate with the service.
4_app-identity-def.ymlfile connects the authentication identities to application identities.
5_authn-any-policy-branch.ymlpolicy is defined to verify that hosts can authenticate with Conjur from anywhere in the policy branch to retrieve secrets for Kubernetes.
6_app-access.ymldefines secret variables for different applications that will use Conjur as its secrets manager. Note that the variables mentioned here are just secret variable names and not the values.
Generate mTLS Cert and Key
Run the create_mtls_certs.sh shell script to create the mTLS cert and key. Make sure to update the
CONJUR_ACCOUNT in the script with the values appropriate for the Conjur installation.
AUTHENTICATOR_ID is the part that follows
conjur/authn-k8s/ in the
id of the
3_cluster-authn-svc-def.yml policy file.
Load Policies and Secrets
conjur-cli to load policies and data in Conjur. Conjur has a container image pre-packaged with the Conjur CLI. We will run the Conjur client as a pod on the cluster. The policies and certificates will get mounted on the same as configmap volumes.
conjur-cli will load the policies and certificates to the Conjur server from these volumes.
Have all the policy files under the
policy directory and the mTLS cert-key pair in
mtls directory to create configmaps out of it.
Run the Conjur client pod with the above configmaps mounted as volumes. Download the sample pod config from here. Create the pod with downloaded config and exec into it to load values.
Connect to the Conjur server and authenticate as an admin user:
We can start loading policies now. Loading the policy files
2_app-authn-def.yml in Conjur generates API keys for the users and hosts defined in it. The user API keys can be distributed to respective team members, allowing them to authenticate and interact with Conjur. We will use the Kubernetes authenticator instead of host API keys to get the application authenticated with Conjur.
Load mTLS certificate and key.
Load secret values.
You don’t want the client pod to continue running in the cluster, especially because it’s currently logged in to the Conjur server. Hence either log out from the Conjur server with
conjur authn logout or delete the
conjur-client pod. Also, delete the configmaps mounted on the client pod.
Configure and Run the Application
Conjur offers various authenticators for users and hosts. Here we will use the Kubernetes authenticator to get our application host authenticated with the Conjur server. Kubernetes authenticator uses Kubernetes APIs to authenticate resources like Pod, Deployment, etc. Refer Conjur documentation to see the full list of supported Kubernetes resources that can be defined as hosts.
Kubernetes Authenticator Client is one of the two options for using this authentication method. You can run it as
initContainer or as
sidecar for each application. Configure the Conjur URL, account, login, etc., as the env variables on the application and the authenticator container. Login is the full host id as defined in the
2_app-authn-def.yml policy file. You also need to mount the SSL certs for the Conjur service. In this case, we have to use the SSL certificates generated by the Helm chart during Conjur installation. If you have your own SSL certificates configured on the Conjur server, you can use the same. Note that the value of
CONJUR_AUTHN_URL on the authenticator container is slightly different from the
CONJUR_APPLIANCE_URL on the application container.
CONJUR_AUTHN_URL has the authentication service id appended to the
Authentication client authenticates itself to the configured Conjur server URL and provides its identity, i.e., the login id, pod name, namespace, etc. Conjur validates the information provided by the authenticator client against defined policies, as well as, Kubernetes and provides an access token. The access token is valid only for 8 mins. The client container saves the authentication token on an in-memory volume, which is mounted to both the containers viz. application and authenticator.
Summon which is a separate open source utility from CyberArk Conjur, uses this token to fetch the values for secrets. Your application container needs to run
Summon as its main process with a path to a
secrets.yml file that lists all the secret values that it needs to pull from Conjur. Summon runs the application executable as a sub-process and passes the secret values it fetched as env variables. See the command configured on the Dockerfile of the application we're about to run.
!var in the secrets.yml file indicates that the value needs to be injected as an
env variable. It can be replaced with
!file:var in case you want the value to be written to a file. In that case, the env variable name on the left side will contain the file's path where secret content is written. Observe the contents of our
Before running the application, let's first create our application namespace and copy the secret containing Conjur TLS certs to our application namespace.
Use the example application deployment file to run the application. As you saw in the
Dockerfile, my example application is a BusyBox container. It just prints the value of secret
demo-app-vars/password from Conjur every 5 seconds. A typical application should never print secret values in logs; but we'll use it only to demonstrate that the value is available to the application from Conjur. Let's run the same and observe the logs.
Check to see if the application has the secret value from Conjur available as an environment variable.
An additional security feature you get by using Conjur and Summon is that the secret value is only available to the application and not to the entire container. This means, if an attacker were to get access inside the application container, they won't be able to access the secret values by listing the environment variables in the container.
Cleanup all the resources created in this post with the below commands:
In this post, we looked at what Conjur is, its uses, and basic concepts. We also installed Conjur on a Kubernetes cluster and integrated it with a sample application running in Kubernetes. Hope this gives you a good start for using Conjur with Kubernetes.
Published at DZone with permission of Sameer Kulkarni. See the original article here.
Opinions expressed by DZone contributors are their own.