Clusters of Docker Containers in AWS With GitLab

This is meant to be a quick how-to reference for people who are new to Docker deployment in AWS.

Before starting, have a certificate that allows admin access to AWS. This is the *.pem that the web browser offers to download when a key-pair in EC2 is created.

Familiarity with the concepts of AWS/ECS, Docker, and Pipelines is assumed.

It's also assumed that the app is already in a Docker image.

Without further ado, let’s dive right in.

AWS: Create a Load Balancer

The load balancer will provide a single URL to reach the cluster where containers can be added or removed. This is a manual step that is only necessary to be done once.

Navigate to Services > EC2 > Target Groups

Click on Create Target Group

All containers deployed under this Load Balancer will be associated directly with this Target Group.

Fill in this form below. The important thing to note here is the health check settings section: the health check endpoint must actually exist, otherwise, the container will be considered unhealthy and ECS will keep killing and creating it forever.

Create the actual Load Balancer

Navigate to Services > EC2 > Load Balancers.

Fill in the name and port where the Load Balancer will listen to and...

...select the availability zones.

On the second step of the wizard, there will be a warning about the lack of HTTPS. Just skip it.

Select the default security group.

In Target Group, select Existing Target Group and in Name, type MyTargetGroup, which was created above.

Go to the end of the wizard and confirm/save the new Load Balancer.

The address to access the application is under DNS name.

AWS: ECS Initial State

Navigate to Services > EC2 Container Service.

The page should look either this…

…or this:

At this point, it is expected to have no containers running.

GitLab: Configure the Pipelines

The docker-compose.yml referred to in the .gitlab-ci.yml below:

version: '2'

services:
  api:
    image: "registry.gitlab.com/my_username/my_project_name/api:latest"
    ports:
      - "80:8080"

Create the .gitlab-ci.yml File

The environment variables seen below can be configured in Settings > CI/CD Pipelines.

image: docker:latest
services:
  - docker:dind

stages:
  - build
  - dockerise
  - deploy

api-build:
  stage: build
  image: maven:3-jdk-8-alpine
  script:
    - cd api
    - mvn clean verify
##### 'artifacts' is the way artifacts can be passed around to the next pipelines #####
  artifacts:
    paths:
      - api/target/api.jar

##### This simply builds and pushes the Docker image #####
api-containerisation:
  stage: dockerise
  script:
    - docker version
    - docker build -t registry.gitlab.com/my_username/my_project_name/api:$CI_JOB_ID api/
    - docker build -t registry.gitlab.com/my_username/my_project_name/api:latest api/
##### $DOCKER_LOGIN_KEY can be obtained in GitLab as a Personal Access Token #####    
    - docker login -u my_username -p $DOCKER_LOGIN_KEY registry.gitlab.com
    - docker images
    - docker push registry.gitlab.com/my_username/my_project_name/api:$CI_JOB_ID
    - docker push registry.gitlab.com/my_username/my_project_name/api:latest

CI_deploy:
  stage: deploy
  image: python:3-alpine
  variables:
    AWS_DEFAULT_REGION: "us-east-2"
  before_script:
##### Install the AWS ECS-CLI #####
    - apk add --update curl
    - curl -o /usr/local/bin/ecs-cli https://s3.amazonaws.com/amazon-ecs-cli/ecs-cli-linux-amd64-latest
    - chmod +x /usr/local/bin/ecs-cli
  script:
##### Configure ECS-CLI, run the container and scale to 2 #####
    - ecs-cli configure --region $AWS_DEFAULT_REGION --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --cluster CI-MyCluster-API
    - ecs-cli up --keypair $AWS_KEY_PAIR --capability-iam --size 2 --instance-type t2.micro --vpc vpc-xxxxxxx --subnets subnet-123abc,subnet-321cba
##### This docker-compose.yml is the one described above #####
    - ecs-cli compose --file api/docker-compose.yml service up --target-group-arn $PROD_TARGET_GROUP_ARN --container-name api --container-port 8080 --role ecsServiceRole
    - ecs-cli compose --file api/docker-compose.yml service scale 2
  environment:
    name: ci
##### This is the URL seen under 'DNS name' when the LB was created #####
    url: $PROD_LOAD_BALANCER_URL
  only:
    - master

When the file above is committed to master in GitLab, the pipelines should be automatically triggered:
…and by clicking on the pipeline, the console should show the logs. The example below is the CI_deploy logs.

Notice the last message from the CLI: "ECS Service has reached a stable state." In the .gitlab-ci.yml, the requested count was 2.

AWS: ECS Final State

The dashboard will now show some stats about the new cluster.

Testing the Cluster

Notice the request is distributed evenly across the available containers in the cluster (in this case 2) by hitting the Load Balancer URL several times.