Using a Custom CockroachDB Image With Docker and Kubernetes

Learn how to build a custom image based on CockroachDB which then can be used in container orchestration of choice.

Artem Ervits

CORE ·

Feb. 26, 22 · Tutorial

Likes (4)

Comment

Save

6.8K Views

Motivation

Cockroach Labs ships new images when a new maintenance release is available, typically on monthly basis. CockroachDB does not rely on the base OS image for any third-party libraries except for geospatial and kerberos packages. That said, OS images may be vulnerable to security exposures depending on their age. Given CockroachDB is written in Golang, replacing the OS can be a trivial task.

Originally, CockroachDB image was shipped with a Debian OS image. CRL then switched to UBI to accommodate a wider scope of use cases including Red Hat OpenShift. UBI images are shipped regularly with CVE patches and given the nature of security vulnerabilities, it is common that the latest and greatest images may or may not have CVEs. Given the preamble, we're going to cover how to replace the base image for CockroachDB and use it in Kubernetes.

High-Level Steps

Build an image
Push this image to a container registry
Verify with Docker
Clean up
Verify with Kubernetes
Clean up

Step by Step Instructions

Build an Image

The current image source code for CockroachDB is located here. I'm going to replace the registry.access.redhat.com/ubi8/ubi-minimal image with ubuntu:22.04 image from Canonical. The good news is that there are not that many changes I need to make. Given CockroachDB used to be based on Debian, we can use the current and the former images as templates for our own custom image. After several changes, I ended up with the following:

    Dockerfile
   
 

   FROM cockroachdb/cockroach:latest-v21.2 AS base

LABEL maintainer="artemervits at gmail dot com"
LABEL version="1.0"
LABEL description="cockroach base image"
LABEL REFRESHED_AT $(date)

FROM ubuntu:22.04

# For deployment, we need the following additionally installed:
# tzdata - for time zone functions; reinstalled to replace the missing
#          files in /usr/share/zoneinfo/
# hostname - used in cockroach k8s manifests
# tar - used by kubectl cp

RUN apt-get update && apt-get install -y \
    tzdata \
    hostname \
    tar  \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir /usr/local/lib/cockroach /cockroach /licenses /docker-entrypoint-initdb.d
COPY --from=base  /cockroach/* /cockroach/
COPY --from=base  /licenses/* /licenses/
COPY --from=base /usr/local/lib/cockroach/* /usr/local/lib/cockroach/

# Set working directory so that relative paths
# are resolved appropriately when passed as args.
WORKDIR /cockroach/

# Include the directory in the path to make it easier to invoke
# commands via Docker
ENV PATH=/cockroach:$PATH

EXPOSE 26257 8080
ENTRYPOINT ["/cockroach/cockroach.sh"]
  

We're using a multi-stage process to generate our custom image.

Once we're satisfied with our result, we can build it.

    Shell
   
   docker build . -t dbist/cockroach:latest-v21.2

Push this Image to a Container Registry

Once it builds, we can push it to our own container registry

    Shell
   
   docker push dbist/cockroach:latest-v21.2

Remember to mark the repo public if you're using Docker Hub. I ran into issues fetching the image originally until I realized what was wrong.

Verify With Docker

Let's run through the Docker tutorial to verify our image is functional.

    Shell
   
   docker network create -d bridge roachnet

docker run -d --name=roach1 --hostname=roach1 --net=roachnet -p 26257:26257 -p 8080:8080  -v "${PWD}/cockroach-data/roach1:/cockroach/cockroach-data"  dbist/cockroach:latest-v21.2 start --insecure --join=roach1,roach2,roach3

docker run -d --name=roach2 --hostname=roach2 --net=roachnet -v "${PWD}/cockroach-data/roach2:/cockroach/cockroach-data" dbist/cockroach:latest-v21.2 start --insecure --join=roach1,roach2,roach3

docker run -d --name=roach3 --hostname=roach3 --net=roachnet -v "${PWD}/cockroach-data/roach3:/cockroach/cockroach-data" dbist/cockroach:latest-v21.2 start --insecure --join=roach1,roach2,roach3

docker exec -it roach1 ./cockroach init --insecure

At this point, let's connect to any of the live containers and verify the OS in use.

    Shell
   
   docker exec -it roach1 bash

    Plain Text
   
   root@roach1:/cockroach# cat /etc/os-release

    Properties files
   
 

   PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
  

At this point, I should be satisfied but we only tested in insecure mode and I want to be absolutely sure I didn't break secure mode. For that, I'm going to use Kubernetes, which will also provide another test scenario.

Let's first clean up after ourselves.

Cleanup

    Shell
   
   docker stop roach1 roach2 roach3 docker container rm roach1 roach2 roach3 docker network remove roachnet

Verify With Kubernetes

I am going to run through the Kubernetes Operator tutorial to be sure our image works.

    Shell
   
   minikube start --driver=hyperkit

We will need 16GB of memory and my local Docker Desktop defaults to 8GB which is insufficient for us. Passing --driver=hyperkid alleviates that problem for me on OSX.

You should see something like:

 Creating hyperkit VM (CPUs=8, Memory=16384MB, Disk=20000MB) ...

Apply the Operator CRD

    Shell
   
   kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.5.0/install/crds.yaml

Apply the operator

```bash
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.5.0/install/operator.yaml

Change the namespace used by the Operator

    Shell
   
   kubectl config set-context --current --namespace=cockroach-operator-system

Check the running pods

    Shell
   
   kubectl get pods

NAME                                          READY   STATUS    RESTARTS   AGE
cockroach-operator-manager-77f4957cdf-sd7gz   1/1     Running   0          46s

Initialize and configure CockroachDB

    Shell
   
   curl -O https://github.com/cockroachdb/cockroach-operator/blob/master/examples/example.yaml

Edit the file with the custom image we built earlier

    Shell
   
 

   # Copyright 2022 The Cockroach Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Generated, do not edit. Please edit this file instead: config/templates/example.yaml.in
#

apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
  # this translates to the name of the statefulset that is created
  name: cockroachdb
spec:
  dataStore:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: "60Gi"
        volumeMode: Filesystem
  resources:
    requests:
      # This is intentionally low to make it work on local kind clusters.
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 2
      memory: 8Gi
  tlsEnabled: true
# You can set either a version of the db or a specific image name
# cockroachDBVersion: v21.2.5
  image:
    name: dbist/cockroach:latest-v21.2
  # nodes refers to the number of crdb pods that are created
  # via the statefulset
  nodes: 3
  additionalLabels:
    crdb: is-cool
  # affinity is a new API field that is behind a feature gate that is
  # disabled by default.  To enable please see the operator.yaml file.

  # The affinity field will accept any podSpec affinity rule.
  # affinity:
  #   podAntiAffinity:
  #      preferredDuringSchedulingIgnoredDuringExecution:
  #      - weight: 100
  #        podAffinityTerm:
  #          labelSelector:
  #            matchExpressions:
  #            - key: app.kubernetes.io/instance
  #              operator: In
  #              values:
  #              - cockroachdb
  #          topologyKey: kubernetes.io/hostname

  # nodeSelectors used to match against
  # nodeSelector:
  #   worker-pool-name: crdb-workers
  

specifically, replace

    YAML
   
    image:
    name: cockroachdb/cockroach:v21.2.5

with

    YAML
   
     image:
    name: dbist/cockroach:latest-v21.2

This is where the Operator is going to fetch the image from a registry, this is why it is important to make sure it is public.

Apply to the cluster

    Shell
   
   kubectl apply -f example.yaml

Watch the pods

    Shell
   
   kubectl get pods --watch

When all pods startup, let's create a secure client we can use to connect to the cluster. You can use the default one from the tutorial or use our custom image to create one. I will do the latter.

    Shell
   
   curl -O https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/examples/client-secure-operator.yaml

Edit the file to replace the image

    YAML
   
 

   # Copyright 2022 The Cockroach Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Generated, do not edit. Please edit this file instead: config/templates/client-secure-operator.yaml.in
#

apiVersion: v1
kind: Pod
metadata:
  name: cockroachdb-client-secure
spec:
  serviceAccountName: cockroachdb-sa
  containers:
  - name: cockroachdb-client-secure
    image: dbist/cockroach:latest-v21.2
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: client-certs
      mountPath: /cockroach/cockroach-certs/
    command:
    - sleep
    - "2147483648" # 2^31
  terminationGracePeriodSeconds: 0
  volumes:
  - name: client-certs
    projected:
        sources:
          - secret:
              name: cockroachdb-node
              items:
                - key: ca.crt
                  path: ca.crt
          - secret:
              name: cockroachdb-root
              items:
                - key: tls.crt
                  path: client.root.crt
                - key: tls.key
                  path: client.root.key
        defaultMode: 256
  

Now we can apply it to the cluster and connect

    Shell
   
   kubectl apply -f client-secure-operator.yaml

    Shell
   
   kubectl exec -it cockroachdb-client-secure -- ./cockroach sql --certs-dir=/cockroach/cockroach-certs --host=cockroachdb-public

    Shell
   
 

   #
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v21.2.5 (x86_64-unknown-linux-gnu, built 2022/02/07 21:01:07, go1.16.6) (same version as client)
# Cluster ID: 372663d8-2e89-4f48-9afd-674e9c5405d5
#
# Enter \? for a brief introduction.
#
root@cockroachdb-public:26257/defaultdb>
  

At this point, we confirmed we can connect to the database and the cluster is up. Let's shell into a pod and confirm OS image is the one we expect.

    Shell
   
   kubectl exec -it cockroachdb-1 -- bash

    Properties files
   
 

   I have no name!@cockroachdb-1:/cockroach$ cat /etc/os-release
PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
  

This wraps it up for us. Hope you found this tutorial useful. We can clean up after ourselves now.

Cleanup

    Shell
   
   minikube stop
minikube delete

Docker (software) Kubernetes CockroachDB shell

Published at DZone with permission of Artem Ervits. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

Trending

Using a Custom CockroachDB Image With Docker and Kubernetes

Learn how to build a custom image based on CockroachDB which then can be used in container orchestration of choice.

Motivation

High-Level Steps

Step by Step Instructions

Build an Image

Push this Image to a Container Registry

Verify With Docker

Cleanup

Verify With Kubernetes

Cleanup

Related

Partner Resources