10 Tips for Building and Managing Containers

With Kubernetes, you can scale your business automatically, and on-demand with little to no downtime, optimizing IT costs and increasing the reliability of your systems.

Containers are at the center of an application running in Kubernetes. When you create Kubernetes workloads (the rules for scheduling, scaling and upgrading an application), you start with a container image that runs your service or Kubernetes workload. After the image is tested and integrated with the rest of the application's code base, it is generally pushed to a container registry. But before we get to that point, there are many best practices to keep in mind when you are writing your services and containerizing them.

1. Keep Up With the Latest Kubernetes Patterns

With new features being released continuously, Kubernetes usage patterns can change. To make sure that your cluster follows the currently established patterns of Kubernetes usage, we encourage you to follow and periodically read the official Kubernetes documentation as well as the changes introduced into each Kubernetes release in the release notes.

2. Save Time and Curate Base Images

Creating application containers to use with your Kubernetes cluster involves building a Docker base image that some or all application containers are based upon. Using a base image makes it possible to reuse image configurations since many apps will share dependencies, libraries, and configuration.

Docker Hub and Google Container Registry all have thousands of ready-to-use base images available for download. You can save a lot of time by using these pre-configured base images for your apps.

3. Don't Trust Arbitrary Base Images; Always Use a Vulnerability Scanner

Although it's convenient to use a pre-built image, be safe and ensure that you've run some sort of vulnerability scan on it first.

Some developers will take a basic image from DockerHub that somebody else created — because at first glance it has the package that they need — and then push the arbitrarily-chosen container to production.

There's a lot wrong with this: you could be using the wrong version of code that has exploits, has a bug in it, or worse, it could have malware bundled in on purpose — you just don't know.

To mitigate this, you can run a static analysis like Snyk or Twistlock and incorporate it into your CI/CD pipeline and scan all of your containers for vulnerabilities.

As a general rule, if you do find a vulnerability in your base image, you should rebuild it and not just patch it. Containers are meant to be immutable, and as a result, the best practice is to rebuild the image with the included patches and redeploy the image.

4. Optimize Base Images

Start with the leanest, most viable base image and then build your packages on top of that. In this way, you will know exactly what is inside of your container.

Smaller base images also reduce any overhead. Your app may only be 5 MB, but if you just add an off-the-shelf image like Node.js and include the entire library, you could end up with 600MB of extra libraries that you don't need.

Other advantages of smaller images include the following:

• Faster builds
• Less storage
• Image pulls are faster
• Potential less attack surface

5. Use Only One Process Per Container

Related to keeping base images small is to only have one process per container. A container has the same lifecycle as the app it hosts, which means that each of your containers should only contain a single parent process.

According to Google Cloud, it's a common mistake to treat containers like virtual machines which can run multiple processes simultaneously. While it is possible to use containers in this way, it doesn't take advantage of Kubernetes's self-healing properties.

As a rule, both the container and the app should start at the same time; likewise, when the app stops, so should the container. If you have multiple processes in one container, you might have a case where states of the application are mixed, which can result in Kubernetes being unable to determine if a container is healthy or not.

6. Properly Handle Linux Signals

Linux signals are used to control the lifecycle of processes inside of a container. In order to link the lifecycle of your app to the container, you need to ensure that your app properly handles Linux signals.

Linux signals such as SIGTERM, SIGKILL and SIGINIT are used within the kernel to terminate processes. But Linux inside containers executes these common signals differently. They don't function as expected by default, resulting in errors and interrupt writes.

A way to overcome this is to use a specialized init system, like Linux Tini that is suited to containers. This tool correctly registers signal handlers (such as PIDs) so that Linux signals will work properly for containerized applications and gracefully shut down orphaned and zombie processes in order to recoup the memory.

7. Take Advantage of The Docker Build Cache Order

Container images are built in a series of layers using instructions found in a template or a Dockerfile. The layers and the order in which they are built are typically cached by the container platform. Docker, for example, has a build cache that allows layer reuse. This cache makes your builds much faster, but you'll only be able to use it if all prior layers used in a previous build are present.

For example, let's say you have a build file with step X, step Y, and step Z, and you make a change to step Z. In this case, the build file reuses steps X and Y from the cache since those layers were present before the layer you changed. This saves you some time by accelerating the build. But if you only changed X, the cache will not include the layers and any steps after it.

While this is convenient behavior and it can save you time, you have to ensure that all of your layers are up to date and are not being drawn from an older out-of-date cache.

8. Use a Package Manager Like Helm

As an unofficial package manager for Kubernetes, Helm is another option for sourcing and updating common workloads and containers to run on your cluster. For custom applications, Helm uses a chart to declare dependencies, and provides tools for rolling upgrades and rollbacks.

You can take advantage of existing base images for generic services that you would want to provide inside your Kubernetes cluster, like databases or web servers, or you can create custom base images for your in-house applications. Creating your own charts will make deployments simpler with less overhead and duplication of work for your development team.

For a tutorial on how Helm works, see, "Managing Helm Releases the GitOps Way."

9. Use Tags and Semantic Versioning

As a general rule, you shouldn't use :latest tag. This is pretty obvious for most developers, but if you don't add a custom tag for your container, it will always try to pull the latest one from the repository. And that latest container may or may not contain the changes you think it has.

When creating custom images, use image tagging and semantic versioning to keep track of changes to your Docker containers. When they run in Kubernetes, image tags are how you communicate which version of the image you want to run on your Kubernetes cluster. For optimal results with Kubernetes, you should pick a Docker image versioning scheme with both the production workload and the development flow in mind.

10. Manage Secrets Wisely

In many cases when building Docker images, you need to give the applications running inside the container access to sensitive data. Those could include API tokens, private keys, and database connection strings.

Embedding those secrets inside the containers is not a secure solution, even if you are keeping the images private. Pushing an unencrypted secret as part of a Docker image exposes you to a myriad of additional security risks, including the security of your network and your image registry. The Docker architecture itself is not optimized for allowing unencrypted sensitive data inside the containers.

Instead, the easiest way to securely store the secrets outside your containers is to use Kubernetes Secrets objects.

Kubernetes offers a Secrets abstraction which allows you to store the secrets outside of the Docker images or the pod definitions. You can expose Kubernetes secrets as mounted volumes inside the containers or as environment variables. When you update the secret values in Kubernetes Secrets, it's enough to rotate the pods of your service to start using the new credentials. There are other options available for storing secrets, such as Hashicorp Vault and Bitnami sealed secrets.