Injecting Secrets: Kubernetes, HashiCorp Vault, and Aqua on Azure

one of the neat features of the aqua security solution is the ability to inject into the environment of a running container so that they never get written to disk. in this article, i’m setting up an aqua installation on azure, using kubernetes as the orchestrator and hashicorp vault as the secrets store, so that i can try this secret injection.

quickstart template for kubernetes

azure have a whole suite of quickstart templates set up that help you get started with some common tasks. there is also a nice walkthrough of setting up kubernetes on azure that’s easy to follow.

i had some issues around not having sufficient privileges in active directory to create a service principal . it turned out that these are different from the privileges you have within the azure subscription, so you may need addition permissions to be added under the azure active directory section of the azure portal.

the azure walkthrough gets you set up with kubectl in your local terminal so that you can do things like kubectl create -f some-file.yaml to deploy the entities described in that file to your kubernetes cluster in azure. it also shows you how to use kubectl proxy so that you can browse the ui for your kubernetes cluster.

once i had kubernetes set up i installed aqua on the cluster — as that’s only available to customers i’ll gloss over the details, suffice it to say it’s simply a matter of deploying aqua components as services and a daemonset through the usual sort of kubernetes yaml files.

installing vault on kubernetes

i used the current default docker image, 0.7.0, by deploying this yaml file:

most of this file is pretty self-explanatory if you’ve ever deployed anything on kubernetes, but a couple of things bear pointing out:

• adding the ipc_lock capability enables mlock for the vault executable. this is used to stop memory being swapped to disk (which we want so that our secrets don’t get written to disk).
• in dev mode, for convenience, you can specify a root token you’ll use for vault as an environment variable.

this sets up vault in its default development mode. at some point, i’d like to set up vault properly so that it can store secrets in azure storage, but that gets quite involved quite quickly — you start having to worry about tls certificates, unsealing the vault, and so on. i’m going to set that aside for another day!

vault status

there may be other ways, but the way i found to get cli access to my vault was to execute a shell in the vault pod as follows (use kubectl get pods to find the pod name first):

to make client requests we’ll need a token, and for the purposes of this non-production deployment let’s just use the vault root token (the same one that’s defined in the vault yaml file above).

we can now run vault commands here, for example, vault mounts , to list the available mount backends for storing secrets.

accessing vault from aqua

setting up a new key store in aqua is really easy because kubernetes and service discovery means i can just refer to my vault service by name rather than needing to find the ip address. of course in a real deployment, this should be https rather than http, but i’m just using vault in dev mode with no tls connections for now.

for this example, i used the same root token for the vault connection token. in production, you’d want to generate another token for this purpose.

the secret backend should be one of the backend mounts where vault stores secrets. i have used the default ‘secret’ generic backend for this example. if this were production we might want to set up an azure storage backend.

storing a secret in aqua

i can now set up secrets within the aqua console, and they’ll get stored in vault.

those same secrets can be seen using the vault cli:

using an aqua secret in a kubernetes deployment

i’ve got a very simple web service demo that uses a postgresql database to store some very simple data. i’m going to pass the database password in as an aqua secret to demonstrate an example.

first, i defined a secret called aqua-password that holds the value of the password. this is a toy example so it is ok for me to tell you that the password is “hello” as you can see in the code that sets up the database (obviously, you should never write a real password into code like that, by the way!).

here’s the whole yaml file for the web service — the secret is passed as an environment variable in the very last line.

the web service works! and as i know it must be reading and writing to the database, it must have the correct password.

but what’s really nice is that from the perspective of everything but the process running my web service, the environment variable actually has the value “{aqua-password}” and not the secret value “hello.” you can see this in the kubernetes ui.

this means the secret value is never written to disk, making it a whole lot safer than storing it directly in an environment variable (which get written to /proc/<id>/environ on the host). if i had set up a storage backend for vault it would be written there, but only in encrypted form.