Simulating a Multi-Region CockroachDB Cluster on Kubernetes With Minikube
Minikube is a great platform to work on Kubernetes locally. It allows you to develop and test your apps on your own computer before you are ready to deploy t...
Join the DZone community and get the full member experience.Join For Free
Minikube is a great platform to work on Kubernetes locally. It allows you to develop and test your apps on your own computer before you are ready to deploy them in a live cluster.
Following are instructions on how to create a 9 nodes CockroachDB cluster on Kubernetes, locally, using Minikube. This is very similar to the deployment we previously run using Docker, so check it out, too.
Below is the high level architecture. We create:
NodePort, one for each region, so we can simulate accessing a particular region.
ClusterIP, to allow all Pods to communicate with each other
Jobsobjects: one to init the cluster and one to run some SQL queries.
NetworkChaosobjects, to simulate the latency between the regions.
It's important to set up the Minikube VM with enough resources to run the cluster and the workload you'll be running on top of it. Everyone's environment is different, so this is for reference only.
Om my laptop, I have allocated 8 CPUs and 24 GB RAM to Minikube. Ensure you have a similar profile; the default 2 CPUs won't be sufficient to run the cluster flawlessly.
Ensure the configuration is saved
All good, you can start Minikube with the new defaults
Ensure Minikube and
kubectl are configured properly
From the above output, my Minikube cluster has address
192.168.64.3. This will be helpful later when it comes to open the Admin UI and connect our SQL client to the CockroachDB cluster.
Please note: The Kubernetes definition files are hosted on Github in my Gist repository. We will use this repo later when we get to apply them.
The first step is to create the NodePort Services objects, one for each region, plus a ClusterIP Service so that all nodes can communicate with each other.
Here is the Service object definition for region
us-west2. Note the
nodePort parameter, which we use to expose the port externally.
Below the intra-node service.
We then need the Pods and their PVCs objects.
Below is the definition of Pod
roach-seattle-1 and its PVC object. Check the
subdomain, and the
command used to start up the object - specifically the
Finally, we have the Job objects.
The first Job object initiates the cluster
The second Job object runs the SQL statements to set the geo coordinates of the regions, so the Map knows where to place them (provided you have an enterprise license).
Now that we understand the required objects, apply the full yaml definition file to create the cluster
After a few minutes, the cluster should be up and running, check the Pods are ready and in a running state:
Simulate the latency between the regions. We set region latency as follows:
- us-west2 <=> us-east4: 60ms
- us-east4 <=> eu-west2: 120ms
- us-west2 <=> eu-west2: 180ms
We will use a special object for this, a
NetworkChaos object which you need to install first.
NetworkChaos object locally, check these instructions for Minikube.
Below is the definition of the NetworkChaos object to set the latency between the
Create the NetworkChaos objects
Confirm the NetworkChaos objects are up
At this point, retrieve from Minikube the URLs of the Services you created.
As expected, each service returns 2 ports, one for the Admin UI, and one for the SQL client access.
In your browser, open the Admin UI using the URL with port 31080, 31180 or 31280. In my case, this is http://192.168.64.3:31080. Check the Network Latency page, confirm you see the latency between regions, but not within a region.
Very good, as expected!
Open a SQL shell. You can download the
cockroachdb binary which includes a built in SQL client or, thanks to CockroachDB's compliance with the PostgreSQL wire protocol, you can use the
You will require an Enterprise license to unlock some of the features described below, like the Map view. You can request a Trial license or, alternatively, just skip the license registration - the deployment will still succeed.
Enter the license registration, if you have one
Confirm you are in the correct region as per port mappings:
Verify you can see the latency between nodes. In this example, we connect to region
eu-west2 and from within the container, we initiate a SQL connection to the database using one of the Seattle nodes. From there, we issue a simple query and verify the latency.
Connect to the node in London
Once in the London container, connect to the CockroachDB cluster at the Seattle node.
At the SQL prompt, ask the Seattle node to tell you its locality and check the
Time it takes to execute. The query is trivial, so the Time will basically reflect the latency.
As expected, we get ~180ms latency, which is what we have setup using the
NetworkChaos objects. You can do the same exercise for the other nodes, too.
Congratulations! The cluster is ready for your development work!
To delete the deployment, simply delete the deployment definition files.
To uninstall the
NetworkChaos object, follow these instructions.
Published at DZone with permission of Fabio Ghirardello. See the original article here.
Opinions expressed by DZone contributors are their own.