Docker – Clean Up After Yourself!

We started dockerizing some of our applications recently, and I got to say, that I am in love with Docker! It’s an awesome piece of engineering, and on AWS EC2, it made our lives much easier. However, one problem that we came across when we used Docker is the insane disk usage of it. We run Docker on Amazon Linux, and we have a build server that builds docker images as part of our build pipeline. Once built, the image is then pushed to our servers through an Ansible playbook. Probably I will blog more about it when I get a chance.

What we noticed was that over time, docker seem to eat up the disk space of the host. A quick df -h showed that /var/lib/docker was growing to the point where it pretty much covered the entire disk. So we looked around for a solution and came across the following.

1. Make sure that exited containers are deleted.

When a docker container exists, the container is not deleted automatically. You can see all of the containers with ‘docker ps -a’ command. To clean up the exited containers, the command to use is as follows.

docker rm -v $(docker ps -a -q -f status=exited)

This will remove the exited containers. The -v flag is there to remove any containers that will no longer be needed. If there are no such containers you will see an error message like below.

docker: “rm” requires a minimum of 1 argument. See ‘docker rm –help’.

It just means that there’s nothing to delete and you are good to go.

2. Remove unwanted ‘dangling’ images.  

Docker keeps all of the images that you have used in the disk, even if those are not actively running. This is done so that it has the necessary images in the local ‘cache’. This is awesome because when you want to pull an image that depends on those, or when you are building an image, all of these are locally available. The bad news is that this eats up disk space!

The command to remove these unwanted images is:

docker rmi $(docker images -f "dangling=true" -q)

Again, you might get an error message that the rmi commands needs an argument if you don’t have any such images.

3. Still not enough space? What is this ‘vfs’ directory?

If your docker directory still takes up space, that probably means that you have unwanted volumes filling up in your disk. The -v flag we passed to rm command usually takes care of this. But sometimes, if your way of shutting down containers do not auto remove the containers, the vfs directory will grow pretty fast. We can reclaim this space by deleting the unwanted volumes. To do this, there’s a docker image that you can use!

Here’s how to run it.

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

If you want to be safe and see what it’s going to delete, just pass –dry-run as an argument.

When that program runs, it will do the needful to delete all unwanted volumes and you should have your disk space back.

4. That’s all good. Do I have to do this everytime?

Then next question we had was while all that is good, we had to manually run it whenever our servers get filled up. So we decided to automate it. And it is a breeze with crontabs. Just drop all of the above commands into a file under /etc/cron.daily/ directory. We created a file named docker-clean in that directory with execute rights. The file contains the following.

docker rm -v $(docker ps -a -q -f status=exited)

docker rmi $(docker images -f "dangling=true" -q)

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

Linux will run this job every day automatically and clean up after Docker. I personally think that this should be baked into the docker daemon as a housekeeping feature. Nevertheless, kudos to docker team for building such an awesome tool.