Cloning Code In Containers

One common situation that I run into on different projects is how to tell my Docker container to clone or pull code automatically. In this article, we will see the various ways of getting the code to the Docker container.

There are multiple ways of getting code into the Docker container. In this post, we will see how we can clone a Github public and private repo using oauth tokens and with ssh keys.

1. Using COPY or ADD Command

Docker provides both COPY and ADD instructions for adding files/code to the Docker image. A simple Dockerfile will look something like this:

FROM ubuntu
MAINTAINER "jagadish Manchala"
COPY . /home/sampleTest
WORKDIR /home/sampleTest

The source code needs to be cloned to the current directory where the Dockerfile exists. Once we build the Docker image, the code is copied to the /home/sampleTest directory and will be available once we run the container. ADD and COPY both work similarly. ADD has the extra capability of fetching remote URLs and extracting tarballs. COPY and ADD techniques are most commonly used in production to build production-ready Docker images

docker run -d --name my-source-project -v "$PWD":/tmp gitdocker

This command starts a container attaching the current working directory mounted as a volume to the container on /tmp.

This is a commonly used feature when developing code. We can use a favorite editor to edit the code locally and then attach the directory as a volume to the running container and check the changes.

2. Using the Volume Feature 

The second method is the most commonly used way of sharing local content with the container. Using volumes we can mount a local directory to the container at runtime. The running container can share the files with the host machine.

We need to checkout the latest source code to a directory and mount the directory as a volume to the running container. This can be done as,  docker run -d --name my-source-project -v "$PWD":/tmp gitdocker 

This command start a container attaching the current working directory mounted as a volume to the container on /tmp.  This is commonly used feature when developing code. We can use a favourite editor to edit the code locally and then attach the directory as a volume to the running container and check the changes

3. Using Git Clone Directly

The third method is to run the git clone command directly. Docker provides the RUN  instruction to execute the commands directly inside the container while building the image. We can use the RUN  instruction with the git clone   command in the Dockerfile as below,

FROM ubuntu
MAINTAINER "jagadish Manchala"
#Install git
RUN apt-get update \        
     apt-get install -y git
RUN mkdir /home/sampleTest \      
           cd /home/sampleTest \        
           git clone https://github.com/jagadish12/SampleTest.git
#Set working directory
WORKDIR /home/sampleTest

Though this creates a security risk we can pull the public Github repo directly into the Docker image while building. The code will be available when we start the container.

4. Clone Code Using Github OAuth Tokens

We can securely pull the code from Github using the Dockerfile RUN  instruction. We will be using a Github OAuth token to access our repositories in an automated fashion in the Dockerfile

Create Github OAuth Token

Using personal access tokens allows us to skip providing the username and password while making the request. This is a recommended approach to automating the automation of code. This is also an easier way then working with ssh Keys.

In order to generate the OAuth token, go to your Github -> Settings -> Applications -> Generate New Token. Github will ask for the access levels for the token. Choose the Repo level access permissions and generate the token. Save the token.

Using the Token

Using the token is very easy. All we have to do is to provide the Github URL along with the token generated. From the Dockerfile, we can use the RUN  instruction along with the Git clone:

RUN mkdir -p /home/app
RUN git clone -b master https://7dd7c748e8c0f376920911952fffc7210e03b6db:x-oauth-basic@github.com/jagadish12/SampleTest.git /myapp/

The syntax for the git clone is, git clone -b https://:x-oauth-basic@github.com/jagadish12/SampleTest.git /myapp/

All we have to do is to pass the token along with the Github URL. This way we can clone the private repo using secure way.

5. Using SSH Keys

Docker from version 18 provides us a way to use ssh keys for cloning the code while building the image itself. There are a few things that need to be done before using this feature.

The first thing that needs to be done is enabling the Buildkit backend. Buildkit is an opt-in feature in Docker version 18 that can be enabled with an environment variable DOCKER_BUILDKIT=1   before running the Docker build. Run the export DOCKER_BUILDKIT=1 command to set the variable.

The second thing that we need to do to use the buildkit is to enable the support by adding an experimental syntax in Dockerfile. We need to define the syntax directive as the first line of the Dockerfile. This feature is not enabled in the stable version of the Dockerfile so we need to use the experimental release by adding the syntax line as the first line in the Dockerfile. e.g. docker/dockerfile:experimentalor docker/dockerfile/1.0.0-experimental.
# syntax=docker/dockerfile:1.0.0-experimental

Before running the build for the above Docker image, we need to make sure to configure our host machine with Github.
1. Run the ssh-keygen command and create the id_rsa and id_rsa.pub files.
2. Go to Github -> Repository -> Deploy keys. Create a new key and paste the id_rsa.pub file contents in there and give it a name. Make sure you can clone the repo using the ssh way.
3. The dockerfile looks something like below:

jagadishm@[/Volumes/Work/build]: cat Dockerfile 
# syntax=docker/dockerfile:1.0.0-experimental

FROM centos AS build

# install git 
RUN yum install -y git

RUN mkdir -m 700 /root/.ssh; \
  touch -m 600 /root/.ssh/known_hosts; \
  ssh-keyscan github.com > /root/.ssh/known_hosts

RUN --mount=type=ssh,id=github git clone git@github.com:jagadish12/simple-java-maven-app.git

We are aware of the most instructions. The most important one is the:

RUN --mount=type=ssh,id=github git clone git@github.com:jagadish12/simple-java-maven-app.git

With the new build kit feature we can use mounts in the RUN  instruction of the Dockerfile. In the above RUN  instruction we are using the mount type as ssh. In the above case, I have defined an id “GitHub” which will be used during the build. The value to the Github id will the private key file from our local machine. We will be creating the build image by running the

In the command we are running the build by passing the --ssh and passing the location of the private key file to the Github id that we used in the Dockerfile. The ssh key that is mounted is exposed to the single command that defined the mount, not to the other parts of the build. The ssh key mounted will not be available even in the image metadata. If we run the “docker <Image> history” ,we will not be seeing any details regarding the key mounted. 

docker build --ssh github=/Users/jagadishmanchala/.ssh/id_rsa -t buildtest.

When we run this,

jagadishm@[/Volumes/Work/build]: docker build --ssh github=/Users/jagadishmanchala/.ssh/id_rsa -t buildtest .

[+] Building 7.1s (12/12) FINISHED                                                                                                
 => [internal] load build definition from Dockerfile  0.1s 
 => transferring dockerfile: 617B  0.0s                                                         
 => [internal] load .dockerignore 0.1s
 => => transferring context: 2B 0.0s
 => resolve image config for docker.io/docker/dockerfile:1.0.0-experimental 2.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1.0.0-experimental@sha256:d2d402b6fa1dae752f8c688d72066a912d7042cc172  0.0s
 => [internal] load .dockerignore   0.0s
 => => transferring context: 2B    0.0s
 => [internal] load build definition from Dockerfile  0.1s
 => => transferring dockerfile: 617B   0.0s
 => [internal] load metadata for docker.io/library/centos:latest   0.0s
 => [1/4] FROM docker.io/library/centos  0.0s
 => CACHED [2/4] RUN yum install -y git 0.0s
 => CACHED [3/4] RUN mkdir -m 700 /root/.ssh;   touch -m 600 /root/.ssh/known_hosts; ssh-keyscan github.com > /root/.ssh/  0.0s
 => [4/4] RUN --mount=type=ssh,id=github git clone git@github.com:jagadish12/simple-java-maven-app.git                       3.5s
 => exporting to image  0.1s
 => => exporting layers 0.1s
 => => writing image sha256:7a7c7b6731eccde3e5aa9bc5b186174cb88f19d383524fb67a5cae11fed95888                                 0.0s
 => => naming to docker.io/library/buildtest    0.0s

Once the image is built, we can run the container and see if the repository is cloned.

jagadishm@[/Volumes/Work/build]: docker run -it buildtest /bin/bash
[root@536ee37212ad /]# ls
anaconda-post.log  dev  home  lib64  mnt  proc  run   simple-java-maven-app  sys  usr
bin                etc  lib   media  opt  root  sbin  srv                    tmp  var
[root@536ee37212ad /]# exit

We can see that the project simple-java-maven-app is cloned successfully.

Note: The buildkit feature also supports forwarding ssh connections. So for cloning the command with a private key, the request is then forwarded to the host machine where the key is already configured. This key should be available and need to be added using the “ssh-add” command. Once the key is available to the agent, the Docker build will be using the agent to clone the command. This way we don't need to pass the keys to the builder also.