Containers Are the New Build Artifact

This article is featured in the new DZone Guide to Orchestrating and Deploying Containers. Get your free copy for more insightful articles, industry statistics, and more!

We can all acknowledge that containers are gaining traction and rapidly becoming a
common development tool. Containers make it easy to download and run pre-built images of the software components we use every day in our development and test environments
(and maybe even in production environments if we are really cutting edge). But how well-
integrated are containers in our own build and deployment systems? It is challenging to ensure consistency between our development and test environments. And the local development environments our developers are running can be even harder to verify to ensure the correct versions of our applications and services. And most importantly, we have to ensure that the testing is being performed against the exact same code and environment that will eventually be deployed to production.

If we think back to when we were first developing centralized build systems, we dealt with many of these same issues. We wanted to ensure everyone was using the same X.Y.Z version of an artifact, and that what was compiled into the artifact was exactly what was checked into our source control repository. Centralized build systems solved many of these problems and made sure that only code from source control repositories was compiled into artifacts. The artifacts were then published to the central artifact repositories everyone pulled from. This solution removed the need for individual developers to compile all artifacts on their local machines, share them via a central file share, or email them to their co-workers. In addition, our developers only needed to build and test the components of the system they were directly working on.

The DevOps movement of the past decade then started to address this for the rest of the environments our artifacts were running in. These systems would be responsible for everything else running in those environments beyond the artifacts we are compiling. This controlled the version of Java running on our systems, the Java Cryptography Extension (JCE) policy files, the proper configuration files, etc. It helped make sure our artifacts had the correct version of all the other dependencies installed and set up correctly in their runtime environment.

Maybe you are already using a configuration management tool like Puppet, Chef, Ansible, etc. to set up and maintain these dependencies on long-running hosts. These tools are terrific and have greatly improved our ability to set up and maintain long-running environments consistently. They have also accomplished a major goal of the Infrastructure as Code movement — ensuring that all of our configuration is checked into source control and managed by a release process just like our build artifacts (it is all checked in, right?). But now we have a configuration tool that is likely separate from our deployment process. Which means, now we have to manage both a configuration management system and the deployment process.

What if we evolved our thinking of what the build artifact is? How about if we bundle the build artifact with the runtime environment and everything it needs to run as a stand-alone deployment artifact? Container images are a perfect solution for this — we can bundle our artifacts and all of the other dependencies into a single image. Our build systems can create container images and push them to a central container registry. Docker has their own open-source private registry implementation, and many of the existing artifact repositories like Artifactory and Nexus have added the ability to manage Docker images. Once an image is published to a container registry, we can then share and deploy the exact same image to every environment — local dev, QA, and production. Need to support multiple versions of a runtime component like the Java Runtime Environment (JRE)? We can build different container images for each of our supported versions of Java. In Smartsheet’s case, we have set up multiple base images for the different versions of Java we currently run. To update from Java 8.0.121 to Java 8.0.131, we switch our default image and generate new container images from that specific version.

But hold on a second, you say. You are telling me to build a single container image that can be deployed to any environment? How do I set up environment-specific attributes like database connection settings, secrets, etc? Externalizing your configuration is already an important aspect of building a proper Twelve-Factor App. Containers provide a number of options for managing the external configuration of the container image — we can mount files or directories from the host, use environment variables, access external configuration stores (database, Consul/Vault), use the new Docker secrets feature, etc.

Storing data (log files, database data, etc.) is an obvious challenge many quickly encounter when they first start running containers. I’m sure you can find many articles focused entirely on this topic. Simple solutions include mounting directories from the host file system into the container to store persistent data. As an example, a MySQL server could mount the /var/lib/mysql directory (where the data files are stored — NOT the executable files) from a host folder. This still allows you to change the version of MySQL you are running by stopping the container with the old version and then starting the new version with the same data directory.

Services with stateful data like databases are often a lower priority for migrating to containers. Eventually, the tools and ecosystem will better evolve to support them. We are starting to see this with tools like Kubernetes that expose persistent data primitives across clusters in a cloud environment like Google Container Engine. Our technologies will also evolve to better support containers with cloud-native distributed data stores like CockroachDB.

If you agree that containers could be a useful deployment artifact, you can start by generating them as part of your build pipeline. Containers can easily be built with command line tools that are easy to integrate into existing build systems. At Smartsheet, we use a Jenkins build environment that compiles artifacts and then executes downstream jobs, which generate container images with the build artifact. Each artifact is compiled using an array of base container images. This generates a container image for each supported version of Java, Tomcat, and other services we want to be able to deploy it with.

These containers are then tagged with the version of the build artifact, plus information about the base image, to allow deployment of each container image. We also tag images with easy defaults, so if a user wants to run the current version of a container, they can use the image   <private registry>/servicefoo  . For a specific version of a build artifact with the default version of Java, one would specify the image   <private registry>/servicefoo:1.5.1  . For a specific version with a non-default version of Java, they would use the image   <private registry>/servicefoo:1.5.1-java8u121  .

In the words of a co-worker, containers are just like a Go application — a statically linked image containing everything we need to execute at runtime. Our container registry provides a library of images that we can deploy to any local dev, QA, or production environment. Just provide a
Linux kernel with a container runtime, and our application will have everything it needs to run. To walk through a very simple example, take a look at the Docker Container Java Hello World GitHub example. After all, containers are the new build artifact.

This article is featured in the new DZone Guide to Orchestrating and Deploying Containers. Get your free copy for more insightful articles, industry statistics, and more!