Containers was definitely one of major topics during the OpenStack Summit in Vancouver.
There many announcements and projects that aim to leverage containers in various ways. The most notable one was the new Magnum project, which aims to make Docker and Kubernetes first class citizens in OpenStack.
Having said that, it's not too hard to see that the container space is becoming very crowded and container orchestration specifically is now a hot area and, as such, very confusing. My session during the summit was aimed to map the orchestration landscape for those who feel confused.
Apparently comparing the various orchestration tools was much more challenging than what I had originally anticipated. Part of the reason was that every orchestration framework took a fairly different approach depending on the underlying assumption by which it was built.
In this post I wanted to highlight the main takeaway from this exercise.
I'll start with some basic definitions to set the stage - specifically, what is orchestration and then what was the method we used to compare the different approaches to orchestration. I will end by giving some guidelines on how to choose the right orchestration for the job.
Orchestration is a means with which to automate manual processes. A good analogy for this is the role of Robots in the car manufacturing pipeline.
Similar to these robots, Orchestration is programmed to take an IT management task done manually and automate it.
The common characteristics behind most orchestrations is that they often use some sort of DSL or script to define the application deployment process. The orchestration engine uses this definition as an input and translates it into a set of tasks that interact with the underlying infrastructure which allow it to spawn VM's, configure, and install apps.
Different assumptions led to different approaches. For example, if you're assuming that your world is based only on containers, your approach would be similar to that of Docker. If you're specifically designing for Micro-Services then Kubernetes may be the right choice. However, if your looking for orchestration of a specific cloud infrastructure, OpenStack Heat or Amazon CloudFormation would be the right approach. Similarly, if your environment is a mix of containers and non containerized workloads, and you may be using a mix of OpenStack and other clouds, then you may need an orchestration tool that isn't bound to a specific infrastructure or container technology. This is where TOSCA/Cloudify and Terraform could be the right choice.
For the orchestration roundup we ended up choosing 5 different orchestration tools which cover three main categories:
- Container-centric- Docker, Kubernetes
- Pure-Play - TOSCA/Cloudify, Terraform
- Infrastructure-centric - OpenStack Heat
As the scope of this exercise was too broad, we decided deliberately to exclude PaaS solutions like OpenShift and CloudFoundry as well as configuration management tools such as Chef, Puppet, Ansible and SaltStack. We also decided not to cover all the possible orchestration tools but rather pick a selected sample per category with the assumption that they would serve as a good representation of all the rest.
Finally, we decided not to take a deep dive into each technology but rather focus on measuring the different approaches and compare them on a semi technical/semi conceptual level.
The method of comparison
We chose a NodeJS + MongoDB application, which is a fairly popular stack, and measured what it takes to automate the process of installation, configuration and management of this application in each tool starting from setting up the infrastructure (i.e. compute, storage, network) to the point where we have a full production-ready deployment.
We also measured how each tool handles the following set of common characteristics:
- Dependency Management - The ability to provision and configure the elements in a stack in the right order.
- Reproducible - The ability to wipe the entire deployment, start it all over again, and get the exact same results as the previous run.
- Cloneable - The ability to create multiple instances of the same application between QA, DEV, Production.
- Recoverable - The ability to recover from failure during the deployment or provisioning process.
What's interesting is that the steps needed to orchestrate even a relatively simple stack proved to require lots of steps that are often overlooked.
You can see the full details of that comparison on the slides and recording of the session from the summit:
We started the exercise by comparing the different approaches to orchestration. However, the reality is that each tool has its own sweet-spots and weaknesses, and therefore it is common to see a combination of the tools as a way to make the best out of each tool.
Strong evidence of this was the announcement of the Magnum project which uses Heat to create the OpenStack infrastructure for running Kubernetes or Swarm. In this case Heat is used where its best suited, which is to setup the OpenStack infrastructure and Kubernetes and Swarm to run containers on top of OpenStack.
Similarly Cloudify/TOSCA also announced support for containers, which will allow portability between Docker, CoreOS and other container technologies. The next step would be to serve as a complementary orchestrator to that of Kubernetes and Swarm, allowing for a way to setup the environment for containers on clouds other than OpenStack as well as by allowing management of mixed workloads that are not bounded to containers only.
Which Orchestration to choose?
The bottom line of this entire exercise was to help users map the orchestration landscape. In a nutshell, I would say that if you're looking to orchestrating for Docker, then you should look at Docker Swarm. If you're looking for orchestration of Micro-Services in a way that you are not bounded to Docker, you should look to use Kubernetes.
If you are looking for orchestration that is strictly bounded to OpenStack, a good starting point would be OpenStack Heat. On the other hand, if you're looking for orchestrating a mixed environment, you should be looking at TOSCA/Cloudify.
As I mentioned in the section above, the reality is that in most enterprises we'll see a mix of containers and non-container workload in which case it makes sense to look at the combinations of orchestration tools rather than looking for a single solution to cover them all.
You can find a more detailed response to this question on a previous interview on the subject here.
Final Worlds - The only constant is change!
I've been a regular visitor at almost every OpenStack Summit event. I've also been covering the user-survey from the summit regularly. If there's been one thing that I can see clearly, year after year, it's that in each survey the top list of popular tools tends to vary quite dramatically between Puppet, Chef and later Ansible and SaltStack. Now containers and Docker.
I'm also seeing the growth in popularity of TOSCA specifically in the enterprise and Carrier/NFV world. So, in my personal opinion, when we approach the orchestration and management challenge we need to assume that we'll continue to see this sort of dynamic where new tools and disruption are becoming a constant reality.
A good indication of this was the recent move by Google who teamed with CoreOS to create an alternative stack to that of Docker as was recently announced here.
So rather than continually jumping to the next big thing every time a new tool pops up, we can use an integration approach to allow us to plug-in those new technologies as they come along, and this has been the approach that we've taken with the Cloudify project.
It wouldn't have been possible to cover all these vast amounts of information without the help of my friends at GigaSpaces who jumped on the challenge and spent hours and nights making sure that we are giving fair and honest coverage of each tool in the market.
Special thanks goes to Uri Cohen for covering the Test-Application and Terraform, and Dan Kiliman for covering Docker-Swarm and Kubernetes. I would also like to thank Idan Moyal and Yoram Weinreb who helped me by covering TOSCA and Heat.
Further reading ..