Configuring Containerized Services

I’m assuming you’ve already tried to run some example of a multi-container application. Let’s say we have an application composed of the following:

• Web service
• Database
• Key-value store
• Worker

Let’s focus on the database now. Container images for databases usually come with an easy way to configure them via environment variables. The great thing about this approach is how easy to use it is, e.g. let’s take our RHSCL PostgreSQL 9.5 container image. If you try to run it just like that, it refuses to run and instead guides you how you should run the container:

$ docker run registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
You must either specify the following environment variables:
  POSTGRESQL_USER (regex: '^[a-zA-Z_][a-zA-Z0-9_]*$')
  POSTGRESQL_PASSWORD (regex: '^[a-zA-Z0-9_~!@#$%^&*()-=<>,.?;:|]+$')
  POSTGRESQL_DATABASE (regex: '^[a-zA-Z_][a-zA-Z0-9_]*$')
Or the following environment variable:
  POSTGRESQL_ADMIN_PASSWORD (regex: '^[a-zA-Z0-9_~!@#$%^&*()-=<>,.?;:|]+$')
Or both.
Optional settings:
  POSTGRESQL_MAX_CONNECTIONS (default: 100)
  POSTGRESQL_MAX_PREPARED_TRANSACTIONS (default: 0)
  POSTGRESQL_SHARED_BUFFERS (default: 32MB)

For more information see /usr/share/container-scripts/postgresql/README.md
within the container or visit https://github.com/openshift/postgresql.

Neat! So let’s pass the environment variables:

$ docker run --rm \
    -e POSTGRESQL_USER=test_user \
    -e POSTGRESQL_PASSWORD=secret \
    -e POSTGRESQL_DATABASE=test_database \
    --name=pg \
    registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/userdata/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

output shortened

LOG: redirecting log output to logging collector process
HINT: Future log output will appear in directory "pg_log".

And it’s running!

If we want to start configuring the service, we may do so using a bunch of environment variables, such as POSTGRESQL_SHARED_BUFFERS. And this is where we may hit the wall: How do we configure other parameters?

The first thing to note is that it would be awesome if images clearly documented how you can change the configuration of the containerized service running inside.

In the meantime, let’s have a look at the best ways to configure a containerized service.

Overlaying Configuration

In this method, you should create a new container image by applying your configuration changes on top of an existing container image. Let’s try this with the PostgreSQL container image mentioned before.

The first thing to do is to copy the existing configuration file template,openshift-custom-postgresql.conf.template from the running container:

$ docker cp pg:/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template ./openshift-custom-postgresql.conf.template

We should edit the template file now and produce our custom PostgreSQL container image.

# Custom OpenShift configuration.
#
# NOTE: This file is rewritten every time the container is started!
# Changes to this file will be overwritten.
#

# Listen on all interfaces.
listen_addresses = '*'

# Determines the maximum number of concurrent connections to the database server. Default: 100
max_connections = ${POSTGRESQL_MAX_CONNECTIONS}

# Allow each connection to use a prepared transaction.
max_prepared_transactions = ${POSTGRESQL_MAX_PREPARED_TRANSACTIONS}

# Sets the amount of memory the database server uses for shared memory buffers. Default: 32MB
shared_buffers = ${POSTGRESQL_SHARED_BUFFERS}

# Sets the planner's assumption about the effective size of the disk cache that is available to a single query.
effective_cache_size = ${POSTGRESQL_EFFECTIVE_CACHE_SIZE}

# Let’s increase work_mem so most operations are performed in memory
work_mem = 64MB

We added the last two lines to the template.

Now is the time to build the image. We will use this Dockerfile: (We update labels so it’s clear that the newly built image is not the official RHSCL image, rather a variant based on it):

FROM registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
LABEL name="tomastomecek/postgresql" \
      vendor="Tomas Tomecek"
COPY ./openshift-custom-postgresql.conf.template /usr/share/container-scripts/postgresql/

Let’s build…

$ docker build --tag=tomastomecek/postgresql .

And run…

$ docker run --rm \
    -e POSTGRESQL_USER=pg_test \
    -e POSTGRESQL_PASSWORD=secret \
    -e POSTGRESQL_DATABASE=test_db \
    --name pg tomastomecek/postgresql

And check if the work_mem attribute was changed:

$ docker exec pg bash -c 'psql --command "show work_mem;"'
work_mem
----------
64MB
(1 row)

It’s updated!

We can even go one step further and change the value dynamically. Let’s update our template from

work_mem = 64MB

to:

work_mem = ${POSTGRESQL_WORK_MEM}

We need to rebuild our image first:

$ docker build --tag=tomastomecek/postgresql .

And then let’s set the work_mem attribute to 128MB via the environment variable:

$ docker run --rm \
    -e POSTGRESQL_USER=pg_test \
    -e POSTGRESQL_PASSWORD=secret \
    -e POSTGRESQL_DATABASE=test_db \
    -e POSTGRESQL_WORK_MEM=128MB \
    --name pg tomastomecek/postgresql

We were able to define a new environment variable because the container’s startup script is using the envsubst command. Obviously, this is just an implementation detail and it should be clearly documented how one can define new variables (if possible).

Let’s verify now that work_mem is set to 128MB.

$ docker exec pg bash -c 'psql --command "show work_mem;"'
work_mem
----------
128MB
(1 row)

• Portable — the container will work the same way in any environment.
• Easy to test and audit.

Cons

• Building and distributing a large number of new images can be complicated.
• Requires image to be built – which needs additional automation (git repository for Dockerfile, build a pipeline, image naming conventions, registry).
• It’s tricky to figure out without documentation — which may lead to undefined behavior.

Injecting Configuration

In OpenShift, we can take advantage of ConfigMaps and inject configuration inside pods using them. Christoph Görn wrote a blog post on best practices for configuration inside OpenShift.

Let’s do the example from the “Overlaying Configuration” section above without building a new image.

Instead of building a new image, we will bind mount the template inside the RHSCL PostgreSQL container.

$ docker run --rm \
    -e POSTGRESQL_USER=pg_test \
    -e POSTGRESQL_PASSWORD=secret \
    -e POSTGRESQL_DATABASE=test_db \
    -v openshift-custom-postgresql.conf.template:/usr/share/container-scripts/postgresql/ \
    -e POSTGRESQL_WORK_MEM=128MB \
    --name pg \
    registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest

This was a lot easier. All we had to do was to:

1. Get the template file.
2. Update it.
3. Mount the template file inside the container.

Let’s verify the work_mem attribute is set correctly.

$ docker exec pg bash -c 'psql --command "show work_mem;"'
work_mem
----------
128MB
(1 row)

Pros

• Decoupling configuration from the immutable image — you can configure the containerized service independently.
• No need to create new images.

Cons

• The container is no longer portable and requires configuration.

Conclusion

As you can see, there is no silver bullet when it comes to the configuration of containerized services. There are multiple options and it’s up to you to pick the one that suits you best.