DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Can You Run a MariaDB Cluster on a $150 Kubernetes Lab? I Gave It a Shot
  • How Kubernetes Cluster Sizing Affects Performance and Cost Efficiency in Cloud Deployments
  • The Production-Ready Kubernetes Service Checklist
  • 10 Best Practices for Managing Kubernetes at Scale

Trending

  • DGS GraphQL and Spring Boot
  • Building Resilient Identity Systems: Lessons from Securing Billions of Authentication Requests
  • The Role of AI in Identity and Access Management for Organizations
  • Monolith: The Good, The Bad and The Ugly
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Bootstrapping a Consul Cluster With Cloud-Init-Buddy

Bootstrapping a Consul Cluster With Cloud-Init-Buddy

A look at the scripting processes necessary to bootstrap and integrate a consul cluster into your virtual machine set up. In this tutorial, we'll use Digital Ocean.

By 
David Karapetyan user avatar
David Karapetyan
·
Apr. 06, 18 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
5.9K Views

Join the DZone community and get the full member experience.

Join For Free

Bootstrapping a consul cluster is a non-trivial operation mostly because it requires sequencing the startup of the cluster nodes with the starting of the consul agents. You can't form a cluster until you know the IP addresses of the nodes to pass to the consul agent for registration and cluster formation. So you first need to start the nodes and then distribute the IP addresses to all the nodes that need to be in the cluster. This is where cloud-init-buddy comes in. Cloud-init-buddy provides the glue for the cluster nodes to discover each other and not worry about sequencing startup and setup. Everything happens in the cloud-init script with all the sequencing and discovery automatically handled by the cluster nodes themselves.

The first thing we need to do is create a cloud-init-buddy node. I'm going to use the one provided in the Digital Ocean VM, but you can use whatever cloud provider you feel most comfortable with. The only thing you need is for the VM to support cloud-init.

Here's the cloud-init configuration for setting up the cloud-init-buddy node

#!/bin/bash
while ! (git clone https://github.com/cloudbootup/cloud-init-buddy.git); do
  echo "Looks like network is not ready. Sleeping and retrying."
  sleep 2
done
cd cloud-init-buddy
sudo apt update && sudo apt install --yes ruby
rake setup:initialize
rake flyway:check || rake flyway:install
rake postgres:configure
npm install
./node_modules/.bin/tsc
node utils/generate-certificate.js
tmux new-session -d -s cloud-init-buddy 'node app.js' && sleep 1
node utils/users.js add-user admin \
  $(gpg2 --gen-random 0 64 | shasum -b -a 512 | head -c 32 | tee password)

Wait for the VM to come up. Once it is up, log in to find out the password for the admin user because we need the password for the consul nodes to talk to cloud-init-buddy to coordinate the cluster formation. You can get the admin password by running cat /cloud-init-buddy/password <(echo).

At this point, we have all the pieces and can write the cloud-init scripts for the consul cluster. I'm going to use the values that I got and you will need to substitute your own values for the IP address and admin password.

#!/bin/bash
# Initial parameters for interacting with cloud-init-buddy
export CLOUD_INIT_BUDDY="138.197.199.132"
export PASSWORD="4c6f44cd5b5d549731d9ec1e3d7179e9"
export ENDPOINT="https://admin:${PASSWORD}@${CLOUD_INIT_BUDDY}:8443"
export NODE_COUNT=3
# Directory for us to do work
mkdir /consul
cd /consul
while ! (wget https://releases.hashicorp.com/consul/1.0.6/consul_1.0.6_linux_amd64.zip); do
  echo "Waiting for network to be ready."
  sleep 2
done
apt install unzip
unzip -o *.zip
rm -f *.zip
# What is our IP address? We will need this for registration
export SELF_ADDRESS="$(ip route get 8.8.8.8 | awk '{print $NF; exit}')"
# Initialize the metadata for coordination
export CONSUL_ENDPOINT="${ENDPOINT}/metadata/consul"
curl -k -XPOST "${CONSUL_ENDPOINT}" -d '{"hosts":[]}'
# Wait for all nodes to register
while [[ "$(curl -k "${CONSUL_ENDPOINT}/hosts.length")" -lt "${NODE_COUNT}" ]]; do
  # It is possible some other node reset everything so make sure we re-register
  if ! (curl -k "${CONSUL_ENDPOINT}" | grep "${SELF_ADDRESS}"); then
    curl -k -XPOST "${CONSUL_ENDPOINT}/hosts" -d "\"${SELF_ADDRESS}\""
  fi
  echo "Waiting for other nodes to register."
  sleep 1
done
# Whoever registered first will set a key
if [[ "$(curl -k "${CONSUL_ENDPOINT}/hosts/0")" == "\"${SELF_ADDRESS}\"" ]]; then
  key="$(./consul keygen | head -c 24)"
  curl -k -XPOST -d "{\"key\":\"${key}\"}" "${CONSUL_ENDPOINT}"
fi
# Wait for the key to be set
while ! (curl -k "${CONSUL_ENDPOINT}.keys" | grep "key"); do
  echo "Waiting for key to be set"
  sleep 1
done
# Everyone is registered and there is a key so we can form the cluster.
# In a production setting this would be an actual systemd unit file 
# and you would not use public facing IP addresses, i.e. you'd run
# the nodes in a private address space
nohup ./consul agent -ui -syslog -server -bootstrap-expect "${NODE_COUNT}" \
  -data-dir "/consul" \
  -bind "${SELF_ADDRESS}" \
  -advertise "${SELF_ADDRESS}" \
  -encrypt "$(curl -k "${CONSUL_ENDPOINT}/key" | tr -d '"')" \
  -retry-join "$(curl -k "${CONSUL_ENDPOINT}/hosts/0" | tr -d '"')" &

It looks a little ugly but it gets the job done.

Cloud-init-buddy also provides an endpoint for storing and retrieving tar packages but I didn't use that feature here. It comes in handy when you want to include other assets on the consul nodes or if you want to use something other than bash for bootstrapping the node. The package could include a Ruby or Python script and the cloud-init script would just be reduced to downloading the package, unpacking it, and executing the necessary script from the package.

cluster

Published at DZone with permission of David Karapetyan, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Can You Run a MariaDB Cluster on a $150 Kubernetes Lab? I Gave It a Shot
  • How Kubernetes Cluster Sizing Affects Performance and Cost Efficiency in Cloud Deployments
  • The Production-Ready Kubernetes Service Checklist
  • 10 Best Practices for Managing Kubernetes at Scale

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: