How to Automate Certificate Issuance to Kubernetes Deployments Using Autocert

In public web browsing, TLS authentication happens in one direction — only servers present their certificates. Serving public web pages without authenticating the client's identity is reasonable, but it isn't for Kubernetes. If other entities are going to be accessing sensitive data on services/clusters, then it makes sense that the identity of such entities should be validated too.

Using TLS everywhere is one of the Kubernetes team's recommendations for hardening cluster security and increasing resilience.

“TLS should be enabled for every component that supports it to prevent traffic sniffing, verify the identity of the server, and (for mutual TLS) verify the identity of the client.”

In public web browsing, TLS authentication happens in one direction — only servers present their certificates. Serving public web pages without authenticating the client's identity is reasonable, but it isn't for Kubernetes. If other entities are going to be accessing sensitive data on services/clusters, then it makes sense that the identity of such entities should be validated too. This is known as mutual TLS (mTLS).

What Is Mutual TLS?

Mutual TLS (mTLS), also known as two-way authentication, is a security process in which entities authenticate each other before communication occurs. This means that in order for a connection to be established, both the client and the server must exchange, verify, and trust each other's certificates. mTLS authentication is a best practice for improving Kubernetes security and ensuring that only authenticated entities communicate with your clusters.

Unlike VPNs and SDNs, implementing mTLS is pretty easy. There's just one hurdle: you need certificates issued by your own certificate authority (CA). Building and operating a CA, issuing certificates, and making sure they're renewed before they expire can be tricky. Autocert and Smallstep Certificate Manager make the whole process easy.

A Little Background on the Tools We’ll Be Using

Autocert

Autocert is an admission webhook that intercepts and patches pod creation requests with some YAML to inject an init container and sidecar that handle obtaining and renewing certificates, respectively. In other words, is an open-source Kubernetes add-on that automatically injects TLS certificates directly into your containers.

Autocert is useful for when you don’t trust the backing datastore where Kubernetes TLS Secrets are stored (often this is etcd). Depending on how your datastore is configured, the data may or may not be encrypted. With Autocert, certificates are injected directly into containers, and private keys are never transmitted across the network and aren't stored in etcd.

To obtain a certificate, you just need to tell autocert your workload's name, using the autocert.step.sm/namepod annotation. Autocert will issue a cert to the pod, make it available in var/run/autocert.step.sm, and keep it renewed. However, it requires a certificate authority to issue certificates. That's where Smallstep Certificate Manager comes in. This guide shows how to configure autocert to use Certificate Manager as the upstream CA.

Smallstep Certificate Manager

Smallstep Certificate Manager is an opinionated, extensible platform for DevSecOps public key infrastructure (PKI) that delivers highly available hosted certificate authorities. It also offers expiry notifications and alerts, a management dashboard, Active Revocation, and other features. With Smallstep Certificate Manager, you can easily issue private TLS and SSH certificates to all your workloads/developers.

Technically, Kubernetes doesn't come with a CA. It has integration points that allow you to use any CA. So, we’ll be using Smallstep Certificate Manager since it’s from the same team that built and maintains Autocert.

Before We Begin

Before we get into the main steps, you need to:

1. Create a free account on Smallstep Certificate Manager
2. Create a Certificate Authority in Certificate Manager that will act as your upstream CA
3. Install step CLI. To interact with Certificate Manager via terminal, you will need the stepCLI command on your local machine. It’s used for many common crypto operations. Check here for a list of all commands.

Setting Up Autocert

1. Bootstrap With Your CA

Bootstrapping with your Authority configures your workstation to trust the authority's root certificate. Run:

step ca bootstrap --ca-url [your CA URL] \\
    --fingerprint [your CA fingerprint] \\
    --install

2. Add a Provisioner

Provisioners are methods used to verify the legitimacy of certificate signing requests and attest to the identity of the requesting service or human. Autocert requires a JWK provisioner. The command below creates a default JWK provisioner called autocert. Run:

step ca provisioner add autocert --create

You’ll be asked to provide a password to encrypt the provisioner private key.

3. Create ConfigMaps and a Secret for Autocert

In Kubernetes, create a namespace for autocert:

kubectl create ns step

Output:

namespace/step created

Use the same password you entered when creating the provisioner to create the Secret.

kubectl -n step create secret generic autocert-password --from-file=password=autocert-password.txt

Output:

secret/autocert-password created

Now create a ConfigMap that includes your (bootstrapped) config dir:

kubectl -n step create configmap config --from-file $(step path)/config`

Output:

configmap/config created

We'll do the same thing for the certs dir, which contains our CA's root certificate:

kubectl -n step create configmap certs --from-file $(step path)/certs

Output:

configmap/certs created

4. Deploy Autocert

Download the Autocert YAML config:

curl -O <https://raw.githubusercontent.com/smallstep/autocert/master/install/02-autocert.yaml>

Edit the caUrl in the autocert-config ConfigMap of the yaml file you just downloaded. Change it from https://ca.step.svc.cluster.local to your Certificate Manager authority URL, e.g. https://autocert.areed.ca.smallstep.com.

Then run:

kubectl apply -f <https://raw.githubusercontent.com/smallstep/autocert/master/install/03-rbac.yaml>

Output:

clusterrole.rbac.authorization.k8s.io/autocert-controller created
clusterrolebinding.rbac.authorization.k8s.io/autocert-controller created

Now let's deploy Autocert. Run:

kubectl apply -f 02-autocert.yaml

Output:

service/autocert created
configmap/autocert-config created
deployment.apps/autocert created

Next, let's deploy the admission webhook. You can see in this block that we include the root CA certificate (in base64) as part of the client configuration for Autocert.

cat <<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: autocert-webhook-config
  labels: {app: autocert}
webhooks:
  - name: autocert.step.sm
    sideEffects: None
    admissionReviewVersions: ["v1beta1"]
    clientConfig:
      service:
        name: autocert
        namespace: step
        path: "/mutate"
      caBundle: $(cat $(step path)/certs/root_ca.crt | base64 | tr -d '\\n')
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    namespaceSelector:
      matchLabels:
        autocert.step.sm: enabled
EOF

Output:

mutatingwebhookconfiguration.admissionregistration.k8s.io/autocert-webhook-config created

Autocert is now added to the cluster and configured. You can run this command to verify the autocert pods are marked ready.

kubectl -n step get deployment/autocert

Use Autocert to Enable mTLS Between Server and Client

Let’s create a test app that will use Autocert. It’s a “Hello World” web server that uses mutual TLS authentication.

Since this deployment is in the default namespace, label it to tell Autocert to issue and renew certificates for new pods with the autocert.step.sm/name annotation:

kubectl label namespace default autocert.step.sm=enabled

Output:

namespace/default labeled

To test things out, we'll create a deployment with the autocert.step.sm/name pod annotation. This example uses the name localhost, since we will be testing from our workstation.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-mtls
  name: hello-mtls
spec:
  selector:
    matchLabels:
      app: hello-mtls
  template:
    metadata:
      annotations:
        autocert.step.sm/name: localhost
      labels:
        app: hello-mtls
    spec:
      containers:
      - image: smallstep/hello-mtls-server-go:latest
        name: hello-mtls
EOF

Output:

deployment.apps/hello-mtls created

For testing, forward localhost:8443 to port 443 on the pod.

kubectl port-forward deploy/hello-mtls 8443:443

Output:

Forwarding from 127.0.0.1:8443 -> 443
Forwarding from [::1]:8443 -> 443

Keep this running in the background during the next steps.

Now issue a client certificate signed by your CA. You’ll need this to authenticate to the “Hello mTLS” test server.

step ca certificate linda.ikechukwu@smallstep.com demo.crt demo.key

Finally, you should be able to verify it's all working:

curl --cacert $(step path)/certs/root_ca.crt \\
     --cert demo.crt --key demo.key \\
     <https://localhost:8443>

Output:

Hello linda.ikechukwu@smallstep.com

Conclusion

It can be challenging to secure Kubernetes while still managing the various moving parts of your cluster. Regardless of the network architecture and policies in place, automating security by design will always make your cluster’s workload safer and more reliable.

Autocert, alongside Smallstep Certificate Manager, makes it easy to automate certificate issuance into a Kubernetes deployment. All you need is a little bit of YAML and a working cluster to begin issuing Kubernetes TLS certificates to your microservices and thwarting nefarious actors.