Diamanti's Vision for Containers

Mark Balch, VP of Products at Diamanti shared his thoughts on the state of the orchestration and deployment of containers.

How is your company involved in the orchestration and deployment of containers?

Diamanti enables enterprise users to orchestrate the complete container deployment process that goes beyond the compute cluster to automate critical networking and persistent storage services. Diamanti has upstreamed Kubernetes APIs for storage plugins (FlexVolume) and scheduler extensions to support smart, real-time placement of containers and their infrastructure requirements.

What do you see as the most important elements of orchestrating and deploying containers?

As the community has made it easy to package applications within containers and deploy those containers onto compute nodes, critical elements remaining to be adopted by mainstream users include orchestrating networking and persistent storage services with predictable outcomes. The increased density and rapid lifecycle of containers compared with virtual machines makes automation of the complete application-infrastructure stack mandatory to achieve reliable, predictable container application deployment.

Which programming languages, frameworks, and tools do you, or your company use, to orchestrate and deploy containers?

Diamanti is an active member of the Docker and Kubernetes ecosystem and leads the Kubernetes storage SIG. We use programming languages such as Go, while also leveraging popular modern software components including Linux, Docker, Kubernetes, FlexVolume, and Container Network Interface.

How has the orchestration and deployment of containers changed application development?

As organizations prioritize shipping new products and features faster, developers are favoring containers over VMs because containers allow them to scale applications and deploy resources much faster than what traditional VMs on public and private clouds can support. Provisioning and starting a VM takes minutes under the best of circumstances, while containers barely require milliseconds.

What kind of security techniques and tools do you find most effective for orchestrating and deploying containers?

Securing container orchestration interfaces and the container runtime is important to provide a dependable platform. Standard Linux technologies including SELinux (access control), seccomp (system call restriction), and AppArmor (security profiles) can be configured via orchestration to improve the isolation and security of containers.

What are some real-world problems being solved by the orchestration and deployment of containers?

Tremendous benefits in reducing application deployment time and improving application quality and test coverage can be attained by orchestrating the deployment of entire applications - both stateless and stateful components - with containers. Two examples are data analytics pipelines and CI/CD.

Data analytics pipelines often consist of several stages including ingestion, persistence, and processing. Though each stage may scale independently of one another, it should do so rapidly to meet real-time data requirements. Orchestrating each stage’s container deployment in addition to its’ networking and persistent storage resources ensures the pipeline will adjust to meet demand without unacceptable delay or excessive costs due to over-provisioning for peak capacity.

CI/CD is inherently automated to build, test, and deploy code. Yet, most CI/CD approaches require compromises and manual setup of stateful elements such as databases and messaging queues. These manual processes result in limited test coverage and mismatched development and production environments. Orchestrating the full application stack including stateless and stateful elements increases test coverage and eliminates errors and scaling problems caused by manual configuration of databases.

What are the most common issues you see affecting the orchestration and deployment of containers?

Lacking support for fully automated network and storage infrastructure that complements mainstream container orchestration is a common issue across our industry today. Most users are forced to bridge gaps between agile, automated container deployment and manual, complex infrastructure deployment.

Do you have any concerns regarding the current state of orchestrating and deploying containers?

Open-source APIs that extend container orchestration software (e.g., Kubernetes, Mesos, Docker Swarm) to infrastructure services (e.g., networking and storage) are at a nascent stage and not widely understood. Each open source orchestration project is at a different level of maturity, though community activity is actively narrowing the gap. Kubernetes is furthest ahead with CNI (Container Network Interface) and FlexVolume/CSI (storage). More vendors and uses are adopting these open APIs and the community is adding richer capabilities to address more use cases including network segmentation and storage services including data protection and encryption.

What’s the future for containers from your point of view - where do the greatest opportunities lie?

Containers are the next mainstream application development and deployment platform, ultimately replacing virtual machines as the defacto package. While still an emerging technology, containers are being rapidly adopted by developers, which creates an unstoppable demand on IT and operations teams to natively support containers to more rapidly scale digital services and achieve lower overall deployment costs.

What do developers need to keep in mind when working on orchestrating and deploying containers?

Developers should seek common tools and interfaces with operations to maximize application deployment velocity and reduce dissimilar processes that lead to application downtime. Common tooling like Kubernetes from development to testing to production enables developers to express their application requirements once and ensure those requirements are adhered to throughout the application lifecycle. Taking advantage of orchestration APIs like CNI and FlexVolume/CSI allow developers to define a broader set of application requirements including networking and storage services, which enables consistency and automation with deploying into production.

Is there anything you’d like to know about what developers are doing with regards to the orchestration and deployment of containers?

The industry-at-large would benefit from better understanding developers’ ability to influence production deployment of their container applications to achieve more consistent outcomes in test and production environments through orchestration. This increased understanding will encourage maturation of APIs optimized for both developers and IT operators.

What have I failed to ask you that you think we need to consider with regards to containers?

Are we nearing the time when the industry should standardize on a single, open-source container orchestration system or will better innovation come through competition across several orchestration systems?

Given the open-source nature of the Kubernetes ecosystem and the fact that many commercial vendors are contributing value-added capabilities to the project, there is a high degree of competition built into Kubernetes that is constructive. It remains to be seen if there is a productive outcome to replicating equivalent capabilities on projects like Mesos and Docker Swarm, each of which is supported by primarily one commercial vendor.