Simplifying CockroachDB Kerberos Architecture With a Load Balancer

Today, I'm going to try to simplify our architecture or at least management of Kerberos artifacts as they relate to CockroachDB by introducing a load balancer. Given the presence of LB, we can obfuscate the CockroachDB cluster architecture from Kerberos and ease the management of Kerberos keytabs as well as service principal names.

Articles Covering CockroachDB and Kerberos

I find the topic of Kerberos very interesting and my colleagues commonly refer to me for help with this complex topic. I am by no means an expert at Kerberos, I am however familiar enough with it to be dangerous. That said, I've written multiple articles on the topic which you may find below:

1. CockroachDB With MIT Kerberos and Docker Compose
2. Executing CockroachDB table import via GSSAPI
3. CockroachDB With SQLAlchemy and MIT Kerberos
4. CockroachDB With MIT Kerberos Cert User Authentication
5. CockroachDB with Django and MIT Kerberos
6. CockroachDB With Kerberos and Custom Service Principal Name (SPN)

In my current role, I find new opportunities to learn and grow as an engineer every day. This is the reason I gravitate towards client-facing roles. Having access to many customers and their use cases broadens my understanding and expertise. This series of articles is pushing me to understand security concepts further. 

Today, we're going to focus on clearing one misconception that I may have led my readers on. If you recall my original three articles on CockroachDB and Kerberos, I walked through the steps of creating SPN and keytab entries for every node in the cluster. I was recently conducting internal enablement on CockroachDB and Kerberos and I was faced with a question from a colleague that forced further research into the topic. Turns out it is not necessary to create an SPN and an associated keytab for each node. You can get by with a single SPN and keytab entry if you're using a load balancer. 

The load balancer will authenticate with Kerberos KDC and with the client and that should be enough for the purposes of this conversation. In all the previous articles, I chose not to use a load balancer to keep the architecture simple. Today, I'm going to fill the gaps in our understanding and hopefully close the book on this concept.

1. Copy the cockroach-gssapi directory to your machine.

git clone https://github.com/dbist/cockroach-docker

cd cockroach-docker/cockroach-gssapi

2. Run ./up.sh script to start the environment.

./up.sh
cockroach uses an image, skipping
Building roach-cert
Step 1/15 : FROM cockroachdb/cockroach:v20.1.3 AS generator
 ---> 25bee4f016c4
...
SET CLUSTER SETTING

Time: 8.1053ms

3. Check to make sure all containers are up.

docker-compose ps

cockroach    /cockroach/cockroach.sh st ...   Up      26257/tcp, 8080/tcp
kdc          /start.sh                        Up
lb           /docker-entrypoint.sh hapr ...   Up      0.0.0.0:26257->26257/tcp,
                                                      0.0.0.0:8080->8080/tcp,
                                                      0.0.0.0:8081->8081/tcp
psql         /start.sh                        Up      5432/tcp
roach-cert   /bin/sh -c tail -f /dev/null     Up

We want to make sure we have a load balancer container served by HAProxy.

4. Connect to the psql container and pass the connection string.

We're going to connect to the load balancer lb instead of cockroach node in our connection string.

psql "postgresql://lb:26257/defaultdb?sslmode=verify-full&sslrootcert=/certs/ca.crt" -U tester

psql (9.5.22, server 9.5.0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)
Type "help" for help.

defaultdb=> \q

There are a few things at play here. First, we added postgres/lb@EXAMPLE.COM as our SPN to our KDC and also populated keytab with that information.

So if you look in KDC/Dockerfile, you should see:

RUN kdb5_util create -s -P kpass \
  && kadmin.local -q "addprinc -pw psql tester@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey postgres/cockroach@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey postgres/lb@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey customspn/cockroach@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey customspn/lb@EXAMPLE.COM"

I kept postgres/cockroach@EXAMPLE.COM just so I could connect to the Cockroach node directly but it is no longer necessary. The only SPN and keytab we need is spn/lb@REALM. The start.sh file contains the following:

kadmin.local -q "ktadd -k /keytab/crdb.keytab postgres/lb@EXAMPLE.COM"

# This is an example of overriding postgres SPN
kadmin.local -q "ktadd -k /keytab/crdb.keytab customspn/lb@EXAMPLE.COM"

I'm including a custom SPN just for consistency. So in summary, the only things you need to do are the following:

kadmin.local -q "addprinc -randkey postgres/lb@EXAMPLE.COM
kadmin.local -q "ktadd -k /keytab/crdb.keytab postgres/lb@EXAMPLE.COM

And that's that, now whether you add new nodes or remove old ones, as long as your load balancer is up, you should have a single SPN and keytab entry to manage. Obviously, if you have multiple load balancers, which we highly recommend, you should have those LBs in the KDC database and have appropriate keytabs available. It is hard to emphasize the simplicity of this in a small environment but when you consider hundreds of nodes and just a few load balancer instances, this starts to look valuable.

Finally, let's talk about the caveats, in my roach-cert Dockerfile, I had to add lb to the node cert and you have to do that for each node, as when you connect via the load balancer, the node's cert must have lb hostname as part of SAN.

openssl x509 -in /certs/node.crt -text | grep "X509v3 Subject Alternative Name" -A 1

            X509v3 Subject Alternative Name:
                DNS:cockroach, DNS:lb

And if you have multiple load balancers, they all need to be added to the node certs. Also, in my docker-compose file, you have to populate the psql hosts file with the IP of the load balancer node, not necessarily the cockroach node.

    extra_hosts:
      - "cockroach:172.28.1.4"
      - "lb:172.28.1.5"

I'm including Cockroach anyway but it is not necessary if you only use postgres://lb:26257 as your connection string, having Cockroach there will also allow you to connect with the node's name, not just the load balancer.

Hope this clarifies things for you as it did for me. If you have any feedback, please don't hesitate to reach out.