Setting Up A Docker Registry As Pull Through Cache

In our last blog post, Peter talked about how to Simulate A Docker Hub Outage, where he gave some context on what a pull-through cache (aka registry mirror) is and why Shipyard needs to mirror parts of Docker Hub.

Peter described the implementation as straightforward. While he’s not exactly wrong, I wanted to share some gotchas and tell you some caveats that I wish I knew before implementing this.

Quick Context

Here at Shipyard, we need a place to store the images that we build and pull, which is why we maintain our own internal registries.

I am assuming you already have your own registry up and running if you want to follow along or try out some of the techniques I describe. If not here is a good place to start.

Setting Up The Pull Through Cache

The Docker page on Mirroring Docker Hub is very clear on how to set up a pull through cache. There are only two necessary steps:

Step 1 - Configure The Docker Daemon

Docker allows you to pass the registry-mirrors as a flag when starting the Docker daemon or as a key/value on the daemon JSON config file. I add the flag to our Terraform since we use that to deploy to whichever cloud our customers might be on.

Step 2 - Configure The Registry

I went ahead and added the proxy section to our registry config and off to the races!

proxy:
  remoteurl: https://registry-1.docker.io
  username: XXXXX
  password: XXXXX

Now the hard part, testing to make sure this is actually working.

Curling the registry directly

# Before pulling
bash-5.0# curl 10.0.16.16:5000/v2/_catalog
{"repositories":[]}

# After pulling images
bash-5.0# curl 10.0.16.16:5000/v2/_catalog
{"repositories":["library/postgres","library/redis"]}

Checking the registry’s filesystem:

bash-5.0# ls docker/registry/v2/repositories/library/
postgres  python    redis

That means that next time we try pulling this image, it will pull from our own registry instead of Docker Hubs. Mission accomplished, so I thought…

When it was time to push images to our registry, we got a frustrating unsupported response:

bash-5.0# docker push 10.0.16.16:5000/rogerioshieh/hello-world:test
The push refers to repository [10.0.16.16:5000/rogerioshieh/hello-world]
e07ee1baac5f: Retrying in 1 second
unsupported

Why? 

The Issue: Pushing To Pull-Through Cache Registry

With the unsupported response, we dug through the Docker docs and in the proxy option docs found:

Pushing to a registry configured as a pull-through cache is unsupported. 

The Resolution: Set Up A Second Registry As Pull-Through Cache

Our resolution was to set up a second registry, one where we pull images from and another where we push images to.

The docker daemon routes those requests accordingly and since they both share the same mounted volumes in our Kubernetes environment, this guarantees that pushed images can also be pulled.

The implementation here varies depending on how your registry is set up. For us, since we’re in a Kubernetes environment, the solution at a high-level was to:

1. use essentially the same registry manifest for the registry mirror, with the only difference being the presence of a RUN_AS_MIRROR flag
2. in the registry, use the flag to either start it as a normal registry or as a mirror registry

Voilà, after those changes we confirmed that pulling an image will store it in our registry, and that pushing an image also will store it in our registry.

How About Images In Private Registries…

Most of our customers use private registries (DockerHub, Quay, ECR, GCR, etc.) to store, well, their private images. Security is a first class citizen at Shipyard so obviously we had some questions and concerns to test/figure out:

Can We Still Pull Private Repositories Using A Registry Mirror?

Yes! Even though we do not use your credentials when setting up our registry mirror, we are still able to pull private images by authenticating before the pull.

One thing I have learned with security and infrastructure: always find multiple ways to test!

Testing Locally

I pushed a local image to Docker Hub and made it a private repository on my Docker Hub account:

From my registry pod, I tried authenticating and then pulling the private image via the command line. Just to make sure the private repository wasn’t stored locally, I also logged out and tried pulling again, which fails as expected.

bash-5.0# docker pull rogerioshipyard/postgres:9.6-alpine
Error response from daemon: pull access denied for rogerioshipyard/postgres, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

bash-5.0# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: rogerioshipyard
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
bash-5.0# docker pull rogerioshipyard/postgres:9.6-alpine
9.6-alpine: Pulling from rogerioshipyard/postgres
59bf1c3509f3: Pull complete
c50e01d57241: Pull complete
a0646b0f1ead: Pull complete
912227b294ee: Pull complete
696b75eaa88e: Pull complete
0d83746bb80b: Pull complete
3a431a2253cd: Pull complete
24ff05c04760: Pull complete
587cf5e11ca1: Pull complete
Digest: sha256:84e6f6c787244669a874be441f44a64256a7f1d08d49505bd03cfc3c687b6cfd
Status: Downloaded newer image for rogerioshipyard/postgres:9.6-alpine

bash-5.0# docker logout
Removing login credentials for https://index.docker.io/v1/

bash-5.0# docker pull rogerioshipyard/postgres:9.6-alpine
Error response from daemon: pull access denied for rogerioshipyard/postgres, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Testing On Shipyard

To functionally test this on Shipyard I needed a good E2E test…so I deployed my favorite flask react sample app.

I modified the starter app to use the private image I had set up:

#Compose File Example
postgres:
    image: 'rogerioshipyard/postgres:9.6-alpine'

I added the credentials to my Docker account on Shipyard:

Lastly, I deployed my sample app following Shipyard’s quickstart instructions, and confirmed it worked!

One More Check On These Private Registries…

Docker recognizes private images as such and does not store them in our registry mirror…but I needed to triple check…so I went ahead and used the ole disk size before and after pulling a private image trick:

# Get disk size of registry
bash-5.0# du -s /mnt/nfs/registry
144256    /mnt/nfs/registry

# Pull private image
bash-5.0# docker pull rogerioshipyard/ubuntu
Using default tag: latest
latest: Pulling from rogerioshipyard/ubuntu
7b1a6ab2e44d: Pull complete
Digest: sha256:7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17
Status: Downloaded newer image for rogerioshipyard/ubuntu:latest

# Certify disk usage is the same
bash-5.0# du -s /mnt/nfs/registry
144256    /mnt/nfs/registry

Note: If Docker Hub is inaccessible, we unfortunately won’t be able to pull the private repositories. This is expected behavior, since we depend on Docker Hub to authenticate the request but something to keep in mind.