Using Docker Swarm Secrets to Store and Rotate your SSL Certificates with Nginx

What is Docker Swarm Secrets?

Docker Swarm has an excellent feature out of the box — Docker Swarm secrets. Using it, you can easily keep your sensitive data like credentials, TLS certificates, etc.

In terms of Docker Swarm services, a secret is a blob of data, such as a password, SSH private key, SSL certificate, or another piece of data that should not be transmitted over a network or stored unencrypted in a Dockerfile or in your application’s source code. You can use Docker secrets to centrally manage this data and securely transmit it to only those containers that need access to it.

So, if we want to use it to store our certificates, first we need a certificate. Here we have two options:

• Use a self-signed certificate.
• Buy SSL certificate.

We will use self-signed:

$ mkdir certs && sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./certs/nginx.key -out ./certs/nginx.crt

The command above generates a certificate that expires in 1 year and places it in ./certs/ directory.

Now we have key and crt files, and we can already use them. But besides that, we should always monitor the certificate expiration date. Sure, there are a few ways to do it, but it is out of scope for this topic. Just keep in mind that you can use alerts (Prometheus + Blackbox exporter) of the certificate expiration date to trigger your script, which in its turn updates the secret with the certificate.

Next step, we need to create an Nginx docker service with our certificate. Here is a docker-compose file with a secrets section:

xxxxxxxxxx

version: '3.4'

services:

  ingress_nginx:

    image: ingress_nginx

    build:

      context: .

    ports:

      - "80:80"

      - "443:443"

    networks:

      - network

    deploy:

      mode: global

      restart_policy:

        condition: any

        delay: 5s

      update_config:

        delay: 30s

        parallelism: 1

        failure_action: rollback

    secrets:

      - source: nginx_key

        target: /etc/nginx/nginx.key

      - source: nginx_cert

        target: /etc/nginx/nginx.crt

secrets:

  nginx_key:

    file: ./certs/nginx.key

  nginx_cert:

    file: ./certs/nginx.crt

networks:

  network:

    driver: overlay

You should keep in mind that you cannot update docker secrets on the fly. It means that you should create a dummy secret every time and replace the old secret with a dummy secret. This is an example script of how to update an existing secret:

xxxxxxxxxx

# Create dummy secret

echo "<<< Creating dummy secret >>>" 

docker secret create dummy_key nginx.key

docker secret create dummy_crt nginx.crt

# Delete old certificate and key from docker secret and replace them with dummy

echo "<<< Delete old certificate and key from service and replace them with dummy >>>" 

docker service update \

    --secret-rm ${stack}_nginx_key \

    --secret-rm ${stack}_nginx_cert \

    --secret-add source=dummy_key,target=/etc/nginx/nginx.key \

    --secret-add source=dummy_crt,target=/etc/nginx/nginx.crt \

    ${stack}_ingress_nginx

echo "<<< Delete old certificate from secrets >>>"

docker secret rm  ${stack}_nginx_key

docker secret rm  ${stack}_nginx_cert

# Deploy service with new secrets

echo "<<< Create secret with new certificate and update service >>>"

docker stack deploy --compose-file docker-compose.yml $stack

# Delete dummy secrets

echo "<<< Delete dummy certificate >>>"

docker secret rm dummy_key

docker secret rm dummy_crt

After script execution, we have updated the certificate inside the Nginx container.