Container Orchestration for IoT devices and Distributed Systems

DZone 's Guide to

Container Orchestration for IoT devices and Distributed Systems

Let's take a closer look at container orchestration for your IoT devices.

· IoT Zone ·
Free Resource

Let’s say you started a new job as a developer at a company that created a new smart speaker (think Amazon Echo or Google home). The said device gained a lot of success and you quickly find yourself with a million clients, each with a single device at their home. Sounds great, right? Now, the only problem you have is how you handle deployments to a million devices located all across the world?

  • You could go the way most old-school vendors do by releasing a package for the end user to download and install themselves on the company website, but in this day and age, this will quickly lose your customers to the competition who doesn’t have such high maintenance needs.
  • You could create a self-updating system built into your codebase, but that will require a lot of maintenance and man-hours from the development team, and even then, it will likely lead to problems and failures down the road.
  • You could containerize the codebase, create on each smart speaker a single server Kubernetes cluster, and create a huge federated cluster out of all of them (as Kubernetes doesn’t support this scale nor latency tolerant workers, this is required), but that will lead to huge costs on all the resources wasted only to run all said clusters.
  • You could use Nebula Container Orchestrator — which was designed to solve exactly this kind of orchestration needs.

As you may have guessed from the title, I want to discuss the last option from the list.

Nebula Container Orchestrator aims to help devs and ops treat IoT devices just like distributed Dockerized apps. Its aims are to act as Docker orchestrator for IoT devices as well as for distributed services, such as CDN or edge computing, that can span thousands (possibly even millions) of devices worldwide, and it does it all while being open-source and completely free.

Different Requirements Lead to Different Orchestrators

When you think about it, a distributed orchestrator has the following requirements:

  • It needs to be latency tolerant — if the IoT devices are distributed, then each will connect to the orchestrator through the Internet at a connection that might not always be stable or fast.
  • It needs to scale out to handle thousands (and even hundreds of thousands) of IoT devices — massive scale deployments are quickly becoming more and more common.
  • It needs to run on multiple architectures — a lot of IoT devices uses ARM boards.
  • It needs to be self-healing — you don’t want to have to run across town to reset a device every time there is a little glitch do you?
  • The code needs to be coupled to the hardware — if your company manufacture the smart speaker in the example mentioned above and a smart fridge, you will need to ensure coupling of the code to the device it’s intended to run on (no packing different apps into the same devices in the IoT use case).

This is quite different from the big three orchestrators (Kubernetes, Mesos, and Swarm), which are designed to pack as many different apps\microservices onto the same servers in a single (or relatively few) data centers, and as a result, none of them provide truly latency tolerant connection, and the scalability of Swarm and Kubernetes is limited to a few thousand workers.

Nebula was designed with a fanout message queue (RabbitMQ) as the Central component. This ensures that changes to any of the applications managed by Nebula are pushed to all managed devices at the same time and ensures proper ordering of changes. All the data is stored in MongoDB, which is the single point of truth for the system. You can read more about Nebula architecture. But, as you can tell, it’s based around a worker container on each device that is in charge of starting\stopping\changing the other containers running on that device and a RESTful API layer. This is, in part, due to the design of each component, which can be scaled out. As such, Nebula can grow as much as you require it.

Nebula Features

As it was designed from the ground up to support distributed systems, Nebula has a few neat features that allow it to control distributed IoT systems:

  • Designed to scale out on all of its components (IoT devices, an API layer, RabbitMQ, and Mongo all scale out)
  • Able to manage tens of thousands of IoT devices
  • Latency tolerant — even if a device goes offline, it will be re-synced when it gets back online
  • Dynamically add/remove managed devices
  • Fast and easy code deployments, single API call with the new container image tag, and it will be pushed to all devices of that app.
  • Simple install — RabbitMQ, MongoDB, and a stateless API are all it takes for the management layer, and a single container with some envvars on each IoT device you want to manage takes care of the worker layer.
  • Single API endpoint to manage all devices
  • Allows control of multiple devices with the same Nebula orchestrator (multiple apps)
  • Not limited to IoT, also useful for other types of distributed systems
  • API, Python SDK, and CLI control available

Nebula Example

The following command will install a Nebula cluster for you to play on and will create an example app. This example requires having Docker, curl, and docker-compose installed:

curl -L "https://raw.githubusercontent.com/nebula-orchestrator/docs/master/examples/hello-world/start_example_nebula_cluster.sh" -o start_example_nebula_cluster.sh && sudo sh start_example_nebula_cluster.sh

But, let’s go over what this command does to better understand the process:

  1. The scripts downloads and runs a docker-compose.yml file, which creates:
a) A MongoDB container  —  the backend DB where Nebula apps current state is saved.
b) A RabbitMQ container  —  the message queue that provides the commiuncation pipe to the worker devices.
c) A manager container  —  A RESTful API endpoint; this is where the admin manages Nebula from.
d) A worker container  —  this normally runs on the IoT devices; only one is needed on each device, but as this is just an example, it runs on the same server as the management layer components runs on.

1. It’s worth mentioning the “APP_NAME = example” environment variable set on the worker container. This APP_NAME variable controls what Nebula apps will be connected to the device.

2. The script then waits for the API to become available.

3. Once the API is available, the scripts send the following command:

curl -X POST \ \

-H ‘authorization: Basic bmVidWxhOm5lYnVsYQ==’ \

-H ‘cache-control: no-cache’ \

-H ‘content-type: application/json’ \

-d ‘{

“starting_ports”: [{“81”:”80"}],

“containers_per”: {“server”: 1},

“env_vars”: {},

“docker_image” : “nginx”,

“running”: true,

“volumes”: [“/tmp:/tmp/1”, “/var/tmp/:/var/tmp/1:ro”],

“networks”: [“nebula”],

“privileged”: false,

“devices”: []


This command creates an app named example and configures it to run a Nginx container to listen on port 81. As you can see, it can also control other parameters usually passed to the Docker run command, such as envvars, networks, or volume mounts.

4. After the app is created on the Nebula API, the worker container will pick it up and start a Nginx container on the server. You can run “docker logs worker” to see the Nginx container being downloaded before it starts (this might take a bit longer if you're on a slow connection). And after it’s completed, you can access http://<server_exterior_fqdn>:81/ on your browser to see it running

Now that we have a working Nebula system running, we can start playing around with it to see it’s true strengths.

We can run:

sudo docker run -d — restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock — env RABBIT_HOST=<server_exterior_fqdn> — env RABBIT_VHOST=/ — env RABBIT_USER=nebula — env RABBIT_PASSWORD=nebula — env APP_NAME=example — name nebula-worker nebulaorchestrator/worker

We can run this on other devices and see them start the same example Nginx container on them as well. There is no limit on how many devices we can run this command on.

We can change the container image on all devices with a single API call. Let’s, for example, replace the container image to Apache to simulate that:

curl -X PUT \ \

-H ‘authorization: Basic bmVidWxhOm5lYnVsYQ==’ \

-H ‘cache-control: no-cache’ \

-H ‘content-type: application/json’ \

-d ‘{

“docker_image”: “httpd:alpine”


Similarly, we can also update any parameter of the app such as env_vars, privileged permissions, volume mounts, etc. — the full list of API endpoints is available here.

Hopefully, this little guide allowed you to see the need of an IoT Docker orchestrator and it’s use cases. Should you find yourself interested in reading more about it, you can visit Nebula Container Orchestrator or skip right ahead to the documentation at https://nebula.readthedocs.io.

containerization, containers, distrubted systems, docker, docker orchestration, internet of things, iot, nebula

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}