How to Simulate a Multi-Region CockroachDB Cluster on localhost With Docker
Simulate a CockroachDB cluster on localhost with Docker! A simulated CockroachDB cluster makes for easy testing, training, and development work.
Join the DZone community and get the full member experience.Join For Free
Simulating a CockroachDB cluster can be an incredibly useful tool for testing, training, and development work. It's an easy process that you can perform directly on localhost using Docker. Today's blog post is a tutorial by Cockroach Labs enterprise architect Fabio Ghirardello on how to simulate a multi-region CockroachDB cluster on localhost.
Following are instructions to simulate the deployment of a 9 nodes CockroachDB cluster across 3 regions on localhost using Docker. This is especially useful for testing, training, and development work.
The instructions assume you are running Linux or macOS, although it should work on Windows using Cygwin, and have Docker installed.
Below is the high level architecture diagram. Each region will host 3 nodes:
Setup: Configure Your Simulated Multi-Region Cockroach Setup With Docker
It's important to set up Docker with enough resources to run the cluster and the workload you'll be running on top of it. Everyone's environment is different, so this is for reference only.
On my laptop, I have allocated 8 CPUs and 24 GB RAM to Docker. Ensure you have a similar profile; the default 2 CPUs won't be sufficient to run the cluster flawlessly.
Create a custom image based on the official CockroachDB image as we need to add package
iproute-tc, required to simulate the latency between the cluster nodes.
Save locally as file 'Dockerfile'.
Build the image with tag name
Verify the image is available to Docker:
Good job! The image is ready.
Build the Cluster
Create the required networks. We create 1 network for each region, plus 1 network for each inter-regional connection.
Each node is associated to its own region network, which will attach to the docker instance
eth0 NIC. We also specify the node IP address with the
--ip flag and the IP addresses of all nodes in its region using the
--add-host flag. This will create an entry in the docker instance
/etc/hosts file, which has precedence over DNS lookups (this will become very important in a bit!)
haproxy.cfg files for the HAProxy in each region.
Next, create the Docker containers:
Now, we'll initialize the cluster:
We then attach each node to the inter-regional networks. These networks will attach to new NICs,
eth2. We then use
tc qdisc to add an arbitrary latency to each new NIC.
Connectivity between nodes in the same region will go through the region network, over
eth0, and connectivity among nodes in different regions via the inter-regional network, over
Note: with the connection to the inter-regional networks, the Docker instance internal DNS gets sometimes scrambled up: issuing, say,
nslookup roach-seattle-1 from host
roach-seattle-2 will resolve to either an IP address from the in-region network or from the inter-regional networks. If the hostname does not resolve to the in-region network IP, traffic will go through
eth2 which has the latency applied, causing in-region connectivity to look very slow. To resolve such a problem we use static IP addresses added to each node's
/etc/hosts file. This makes sure that in-region hostnames resolve to the region IP addresses, forcing the connection to go over
eth0 instead of
How to License and Configure Your Cockroach Cluster
You will require an Enterprise license to unlock some of the features described below, like the Map view. You can request a Trial license or, alternatively, just skip the license registration step - the deployment will still succeed.
Open a SQL shell. You can download the
cockroachdb binary which includes a built in SQL client or, thanks to CockroachDB's compliance with the PostgreSQL wire protocol, you can use the
Run the below SQL statements:
At this point, you should be able to view the CockroachDB Admin UI at http://localhost:8080. Check the map and the latency table:
Congratulations, you are now ready to start your dev work on a simulated CockroachDB multi-region deployment!
When you're done, here's how to stop and remove containers, delete the data volumes, and delete the network bridges:
Published at DZone with permission of Fabio Ghirardello. See the original article here.
Opinions expressed by DZone contributors are their own.