DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Software Design and Architecture
  3. Containers
  4. Set up Docker Swarm Cluster Using Consul

Set up Docker Swarm Cluster Using Consul

Learn how to create a Consul discovery service that sets up Docker Swarm clusters.

Arun Gupta user avatar by
Arun Gupta
·
Dec. 02, 15 · Tutorial
Like (5)
Save
Tweet
Share
26.45K Views

Join the DZone community and get the full member experience.

Join For Free

Docker Swarm is native clustering for Docker. It allows you create and access to a pool of Docker hosts using the full suite of Docker tools. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts.

Docker Swarm LogoDocker Swarm has a Manager, a pre-defined Docker Host, and is a single point for all administration. Swarm manager orchestrates and schedules containers on the entire cluster, and can be configured in High Availability. The containers are deployed on Nodes that are additional Docker Hosts.

Swarm talks to a hosted Discovery Service that maintains a list of IPs in your cluster. For development, its easy to use the default discovery service hosted on Docker Hub. Complete instructions for that are available in Install and Create Docker Swarm. This blog will show how to setup Docker Swarm Cluster using Consul.

Lets get started!

Create Consul Discovery Service

  1. Create a Machine that will host discovery service:
    docker-machine create -d=virtualbox consul-machine
    Running pre-create checks...
    Creating machine...
    Waiting for machine to be running, this may take a few minutes...
    Machine is running, waiting for SSH to be available...
    Detecting operating system of created instance...
    Provisioning created instance...
    Copying certs to the local machine directory...
    Copying certs to the remote machine...
    Setting Docker configuration on the remote daemon...
    To see how to connect Docker to this machine, run: docker-machine env consul-machine
  2. Connect to this Machine:
    eval $(docker-machine env consul-machine)
  3. Run Consul service using the following Compose file:
    myconsul:
      image: progrium/consul
      restart: always
      hostname: consul
      ports:
        - 8500:8500
      command: "-server -bootstrap"
    This file is also available at github.com/arun-gupta/docker-images/tree/master/consul. The service is started as:
    docker-compose up -d
    Pulling myconsul (progrium/consul:latest)...
    latest: Pulling from progrium/consul
    3b4d28ce80e4: Pull complete
    e5ab901dcf2d: Pull complete
    30ad296c0ea0: Pull complete
    3dba40dec256: Pull complete
    f2ef4387b95e: Pull complete
    53bc8dcc4791: Pull complete
    75ed0b50ba1d: Pull complete
    17c3a7ed5521: Pull complete
    8aca9e0ecf68: Pull complete
    4d1828359d36: Pull complete
    46ed7df7f742: Pull complete
    b5e8ce623ef8: Pull complete
    049dca6ef253: Pull complete
    bdb608bc4555: Pull complete
    8b3d489cfb73: Pull complete
    c74500bbce24: Pull complete
    9f3e605442f6: Pull complete
    d9125e9e799b: Pull complete
    Digest: sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274
    Status: Downloaded newer image for progrium/consul:latest
    Creating consul_myconsul_1
    Started container can be verified as:
    docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    f05d8dd11e7f progrium/consul "/bin/start -server -" 30 seconds ago Up 29 seconds 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 0.0.0.0:8500->8500/tcp, 8301-8302/udp consul_myconsul_1

Create Docker Swarm Cluster Using Consul

Swarm is fully integrated with the machine, and so is the easiest way to get started.

  1. Create a Swarm Master using the Consul discovery service:
    docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" --engine-opt="cluster-advertise=eth1:2376" swarm-master
    Running pre-create checks...
    Creating machine...
    Waiting for machine to be running, this may take a few minutes...
    Machine is running, waiting for SSH to be available...
    Detecting operating system of created instance...
    Provisioning created instance...
    Copying certs to the local machine directory...
    Copying certs to the remote machine...
    Setting Docker configuration on the remote daemon...
    Configuring swarm...
    To see how to connect Docker to this machine, run: docker-machine env swarm-master

    Three options here:
    • --swarm-discovery defines address of the discovery service.
    • --cluster-advertise advertises the machine on the network.
    • --cluster-store designates a distributed k/v storage backend for the cluster.
    In addition, --swarm configures the machine with Swarm and --swarm-master configures the created Machine to be Swarm master.
  2. Connect to this newly created master and find some information about it:
    eval "$(docker-machine env swarm-master)"
    docker info
    This will show the output as:
    docker info
    Containers: 2
    Images: 8
    Server Version: 1.9.0
    Storage Driver: aufs
     Root Dir: /mnt/sda1/var/lib/docker/aufs
     Backing Filesystem: tmpfs
     Dirs: 12
     Dirperm1 Supported: true
    Execution Driver: native-0.2
    Logging Driver: json-file
    Kernel Version: 4.1.12-boot2docker
    Operating System: Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov  3 19:49:22 UTC 2015
    CPUs: 1
    Total Memory: 996.2 MiB
    Name: swarm-master
    ID: 2EDA:WPOD:YVWO:GGLZ:ZUHY:WCBU:ZERW:OWBE:6MPQ:IPXN:BS2V:QCSI
    Debug mode (server): true
     File Descriptors: 30
     Goroutines: 67
     System Time: 2015-11-29T02:08:10.636829121Z
     EventsListeners: 1
     Init SHA1: 
     Init Path: /usr/local/bin/docker
     Docker Root Dir: /mnt/sda1/var/lib/docker
    Username: arungupta
    Registry: https://index.docker.io/v1/
    Labels:
     provider=virtualbox
    Cluster store: consul://192.168.99.109:8500
    Cluster advertise: 192.168.99.111:2376
  3. Create a new machine to be part of this Swarm cluster:
    docker-machine create -d virtualbox --swarm --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" --engine-opt="cluster-advertise=eth1:2376" swarm-node-01
    Running pre-create checks...
    Creating machine...
    Waiting for machine to be running, this may take a few minutes...
    Machine is running, waiting for SSH to be available...
    Detecting operating system of created instance...
    Provisioning created instance...
    Copying certs to the local machine directory...
    Copying certs to the remote machine...
    Setting Docker configuration on the remote daemon...
    Configuring swarm...
    To see how to connect Docker to this machine, run: docker-machine env swarm-node-01
    Machine talks to the Discovery Service using --swarm-discovery.
  4. Create a second node in this cluster:
    docker-machine create -d virtualbox --swarm --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" --engine-opt="cluster-advertise=eth1:2376" swarm-node-02
    Running pre-create checks...
    Creating machine...
    Waiting for machine to be running, this may take a few minutes...
    Machine is running, waiting for SSH to be available...
    Detecting operating system of created instance...
    Provisioning created instance...
    Copying certs to the local machine directory...
    Copying certs to the remote machine...
    Setting Docker configuration on the remote daemon...
    Configuring swarm...
    To see how to connect Docker to this machine, run: docker-machine env swarm-node-02
  5. List all the created machines:
    docker-machine ls 
    NAME             ACTIVE   DRIVER       STATE     URL                         SWARM
    consul-machine   -        virtualbox   Running   tcp://192.168.99.109:2376   
    default          -        virtualbox   Running   tcp://192.168.99.100:2376   
    swarm-master     *        virtualbox   Running   tcp://192.168.99.111:2376   swarm-master (master)
    swarm-node-01    -        virtualbox   Running   tcp://192.168.99.112:2376   swarm-master
    swarm-node-02    -        virtualbox   Running   tcp://192.168.99.113:2376   swarm-master
    The machines that are part of the cluster have cluster’s name in the SWARM column, blank otherwise. For example, “default” and “consul-machine” are standalone machines where as all other machines are part of the “swarm-master” cluster. The Swarm master is also identified by (master) in the SWARM column.
  6. Connect to the Swarm cluster and find some information about it:
    eval "$(docker-machine env --swarm swarm-master)"
    docker info
    The main difference here is --swarm when finding information about Swarm cluster as opposed to a single machine.This shows the output as:
    docker info
    Containers: 4
    Images: 3
    Role: primary
    Strategy: spread
    Filters: health, port, dependency, affinity, constraint
    Nodes: 3
     swarm-master: 192.168.99.111:2376
      └ Containers: 2
      └ Reserved CPUs: 0 / 1
      └ Reserved Memory: 0 B / 1.021 GiB
      └ Labels: executiondriver=native-0.2, kernelversion=4.1.12-boot2docker, operatingsystem=Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov  3 19:49:22 UTC 2015, provider=virtualbox, storagedriver=aufs
     swarm-node-01: 192.168.99.112:2376
      └ Containers: 1
      └ Reserved CPUs: 0 / 1
      └ Reserved Memory: 0 B / 1.021 GiB
      └ Labels: executiondriver=native-0.2, kernelversion=4.1.12-boot2docker, operatingsystem=Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov  3 19:49:22 UTC 2015, provider=virtualbox, storagedriver=aufs
     swarm-node-02: 192.168.99.113:2376
      └ Containers: 1
      └ Reserved CPUs: 0 / 1
      └ Reserved Memory: 0 B / 1.021 GiB
      └ Labels: executiondriver=native-0.2, kernelversion=4.1.12-boot2docker, operatingsystem=Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov  3 19:49:22 UTC 2015, provider=virtualbox, storagedriver=aufs
    CPUs: 3
    Total Memory: 3.064 GiB
    Name: 10edc606d097
    There are three nodes – one Swarm master and two Swarm _worker_ nodes. There is a total of four containers running in this cluster – one Swarm agent on master and each node, and there is an additional swarm-agent-master running on the master. This can be verified by connecting to the master and listing all the containers.
  7. List nodes in the cluster with the following command:
    docker run swarm list consul://$(docker-machine ip consul-machine):8500
    192.168.99.111:2376
    192.168.99.112:2376
    192.168.99.113:2376
Docker (software) clustering

Published at DZone with permission of Arun Gupta, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Public Cloud-to-Cloud Repatriation Trend
  • Promises, Thenables, and Lazy-Evaluation: What, Why, How
  • Simulate Network Latency and Packet Drop In Linux
  • A Beginner's Guide to Back-End Development

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: