Stopping Docker Containers Gracefully

This was my playground for testing, but after I started to work on enterprise-level applications deployment using Docker I found out that there were a lot of things I was doing wrong. One of them was how I started an application inside Docker.

Almost all my dockerfiles have some bash script at the end to do some minor changes inside containers and to finally start the application. I usually add this script to CMD instruction in the Dockerfile. I though there isn’t anything wrong with that, except that Docker stopped working as it should.

The problem is that when you run bash script it will get PID 1 and your application is a child process with PPID 1. Bash will not forward SIGTERM signal to your application when you run docker stop. Instead, the container will be killed after 10 seconds of timeout which is the default with the docker stop command. This timeout can be adjusted.

There is an easy way to handle this with the “exec” command inside a bash script. It will replace the shell without creating a new process and your application will get PID 1. Let’s test both scenarios first.

For testing purposes I will use this simple redis dockerfile:

FROM ubuntu:trusty

ENV DEBIAN_FRONTEND noninteractive

RUN \
  apt-get update && \
  apt-get -y install \
          software-properties-common && \
  add-apt-repository -y ppa:chris-lea/redis-server && \
  apt-get update && \
  apt-get -y install \
          redis-server && \
  rm -rf /var/lib/apt/lists/*

COPY start.sh start.sh

EXPOSE 6379
RUN rm /usr/sbin/policy-rc.d
CMD ["/start.sh"]

And here is the start.sh script which will change some recommended kernel settings for redis container (docker must be started with privileged set to true):

#!/usr/bin/env bash
# Disable THP Support in kernel
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# TCP backlog setting (defaults to 128)
sysctl -w net.core.somaxconn=16384
#-------------------------------------------------------------------------------
/usr/bin/redis-server

Now let’s build and run this container:

docker build -t my/redis .
docker run -d --privileged --name test my/redis

Then check what is running inside the redis container:

docker exec test ps -ef
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  0 13:20 ?        00:00:00 bash /start.sh
  root         6     1  0 13:20 ?        00:00:00 /usr/bin/redis-server *:6379

Ok, so this is a problem. Now let’s try to stop this docker container:

docker stop test

After 10 seconds the container is killed, not gracefully stopped. We can see this if we check the log. Last message will be that redis is ready to accept connections:

docker logs test

To get this working we need to change the last line in start.sh script and to rebuild the image. We are adding exec before /usr/bin/redis-server command:

#!/usr/bin/env bash
# Disable THP Support in kernel
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# TCP backlog setting (defaults to 128)
sysctl -w net.core.somaxconn=16384
#-------------------------------------------------------------------------------
exec /usr/bin/redis-server

Build, start and run ps -ef command again:

docker exec test ps -ef
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  1 13:24 ?        00:00:00 /usr/bin/redis-server *:6379

As you can see now redis is running as PID 1, and docker stop will work just fine. Let's try it first and check the docker log again. You should see this message in the redis log:

Received SIGTERM scheduling shutdown...

This is how I’m using exec with postgres and tomcat containers where processes are not running with root user:

exec sudo -E -u tomcat7 $CATALINA_HOME/bin/catalina.sh run
exec su postgres -c "${POSTGRES_BIN} -D ${PGDATA} -c config_file=${CONF}"

Here processes will not be running with PID 1 because of sudo or su, however Docker stop works perfectly in both cases. The reason for this is that sudo and su commands will relay SIGTERM signal to child processes.