{{announcement.body}}
{{announcement.title}}

The Results of A Comparison Between Five CI/CD Services

DZone 's Guide to

The Results of A Comparison Between Five CI/CD Services

Check out how the deployment of a Dockerized Node app is handled by five different CI/CD services.

· DevOps Zone ·
Free Resource

colored stones

What CI/CD service is right for you?

Introduction

In this article I will outline the results of a comparison between five different CI/CD services, namely:

For this comparison, I will use each service to run tests and deploy a Dockerized Node application to AWS Elastic Beanstalk. I will detail how to configure the CI/CD process with each service and I will provide some performance statistics that compare the services.

You may also enjoy:  Choosing the Right Continuous Integration Tool

Test Project and Process

The repository hosting the project tested and deployed by the various services can be found on Github. The application is a small Node/Express application that is Dockerised and has a single test. Clone this application and push it up to your own Github repository as we will use it frequently throughout this article.

The process followed for the test for each service is as follows:

Testing process
Testing process

Results

The results can be seen in the table below. I encourage you to follow the instructions that follow these results to replicate these yourself. A few notes:

  • The configuration time is subjective as it depends on your familiarity with each tool. For transparency, I set up pipelines with Buddy as you can see in my other blog posts. That said, the ability to setup pipelines visually drastically decreases the configuration time.

  • The run times are from when I first ran these pipelines and can be seen in screenshots later on in this post. I welcome suggestions on how to improve my run times. The run times are all from using the default machines that these services provide. These can, of course, be configured and a pricing comparison is out of the scope of this article.Comparison chart

    Comparison chart

From my use case, Buddy is the clear winner. As I am not a DevOps specialist, setting up CI/CD pipelines visually really appealed to me and I find that it was extremely easy to configure. Buddy also had the quickest runtime, but I suspect that people will soon make suggestions on how I can improve my existing pipelines. Buddy is, however, not as well-known as the other services and you should, therefore, examine their offering carefully before using their service over their competitors.

In second place, I would recommend AWS CodePipeline as it was surprisingly easy to configure, which has not always been my experience with AWS. CodePipeline is deeply integrated with other AWS services which made it immensely powerful. If you are already using AWS heavily, I would thoroughly recommend this service.

I personally found that Gitlab was the hardest to use out of all services as it wasn’t as well documented. Travis and CircleCI were very well documented and, if I wasn’t using AWS or couldn’t use a smaller company like Buddy I would definitely consider these services.

Please comment if you have any suggestions — I would love to regularly update this article with further optimisations. If you disagree with my findings, please let me know. Finally, please try replicate these results yourself and share your experiences using these tools in your own company.

Methodology

Creating AWS User

For this article, we will need to create an AWS User so that the CI/CD services can programatically access Elastic Beanstalk. Navigate to the IAM service in IAM:

IAM service
IAM service

Click Users.

Users
Users

Click Add user:

Add user
Add user

Name your user whatever you wish. In this example, I named mine CIComparisonBlogUser and gave them Programmatic access before clicking Next.

Adding user
Adding user

We then need to add a policy for this user. Click Attach existing policies directly and search for AWSElasticBeanstalkFullAccess, attach the policy and click Next.

Attach policy
Attach policy

Review your user and save the Access key ID and Secret access key in a secure location. These credentials will be used when setting up each CI service.

Creating an Elastic Beanstalk App

We will need to create an application on Elastic Beanstalk where all the services can deploy to. Navigate to AWS Elastic Beanstalk where you should see the following starter screen:

Starter screen
Starter screen

Click Get started:

Get started
Get started

Name your application (I named mine CI Comparison Blog), select Docker as the platform and choose Sample application which we will use for now. Click Create application and go make some coffee as it will take some time to spin up:

Create application
Create application
Create application complete
Create application complete

Click the provided URL to view the sample application:

Sample application
Sample application

Creating Docker Hub Repository

The final step needed for all services is the creation of a Docker Hub repository. Head over to https://hub.docker.com/, log in, and you should see the following splash screen:

Docker Hub repository
Docker Hub repository

Click Repositories and click Create Repository:

Create Repository
Create Repository

Name your repository whatever you wish (I named mine ci-comparison-blog) and click Create.

Click Create
Click Create

You should see your created repository. The key part to remember is your equivalent of andrewbestbier/ci-comparison-blog:

Create repo
Create repo

Setting Up a CircleCI Pipeline

To setup a CircleCI pipeline, head over to https://circleci.com/ and click Log in with Github (You will need to authorize CircleCI through Github).

Circle CI page
Circle CI page

Next click Settings and Projects and find your Github repository (mine is called ci-comparison-blog) before clicking the settings cog on the right:

Setting
Setting

Click Environment variables and add your AWS and Docker Hub credentials ( AWS_ACCESS_KEY_ID ,  AWS_SECRET_ACCESS_KEY ,  DOCKER_USER and  DOCKER_PASS ). These will be used later on:

Environmental variables
Environmental variables

Next click Add Projects on the sidebar, search for your Github repository and click Set Up Project:

Set up project
Set up project

You will then see a screen providing instructions on how to set your project up. The most important instruction is “Create a folder named .circleci and add a file config.yml ”.

Create folder
Create folder


If you look in the Github repository, you can see that I have already created this file:

version: 2
jobs:
  test:
    working_directory: ~/app
    docker:
      - image: circleci/node:latest # (1)
    steps:
      - checkout
      - run:
          name: Update npm
          command: 'sudo npm install -g npm@latest'
      - restore_cache: # (2)
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: Install npm dependencies
          command: npm install
      - save_cache: 
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - ./node_modules
      - run:
          name: Run tests # (3)
          command: 'npm run test'
  docker-deploy-image:
    working_directory: ~/app
    machine:
      docker_layer_caching: true # (4)
    steps:
      - checkout
      - run: | # (5)  
          docker build -t andrewbestbier/ci-comparison-blog .  
          docker login -u $DOCKER_USER -p $DOCKER_PASS
          docker push andrewbestbier/ci-comparison-blog
  deploy-aws:
    working_directory: ~/app
    docker:
      - image: circleci/python:latest
    steps:
      - checkout
      - run: # (6)
          name: Installing deployment dependencies
          working_directory: /
          command: 'sudo pip install awsebcli --upgrade'
      - run: # (7)
          name: Deploying application to Elastic Beanstalk
          command: eb deploy
workflows:
  version: 2
  build-test-and-deploy:
    jobs:
      - test
      - docker-deploy-image:
          requires:
            - test
      - deploy-aws:
          requires:
            - docker-deploy-image


Let’s break down what’s happening during this build’s execution:

  1. The first job, test is run with a Node Docker container.
  2. The node_modules are restored if they exist in the cache, otherwise, they are installed.
  3. The tests are run.
  4. Docker layer caching is enabled to speed up image building performance.
  5. Next, the Docker image is built and pushed to Docker Hub with the  DOCKER_USER  and  DOCKER_PASS environment variables. Remember to change the repository to the one you created.
  6. The Elastic Beanstalk CLI tool is installed.
  7. The CLI is used to deploy the app to Elastic Beanstalk with eb deploy . This command works as we have already been authenticated invisibly by CircleCI with the  AWS_ACCESS_KEY_ID, and  AWS_SECRET_ACCESS_KEy  environment variables.

You may be wondering how eb deploy knows where to deploy to? To specify a location, you will need to modify the .elasticbeanstalk/config.yml file in your Github repository to align with the sample Elastic Beanstalk application you created:

branch-defaults:
  master:
    environment: CiComparisonBlog-env
environment-defaults:
  CiComparisonBlog-env:
    branch: null
    repository: null
global:
  application_name: CI Comparison Blog
  default_ec2_keyname: null
  default_platform: arn:aws:elasticbeanstalk:eu-west-2::platform/Docker running on
    64bit Amazon Linux/2.12.17
  default_region: eu-west-2
  include_git_submodules: true
  instance_profile: null
  platform_name: null
  platform_version: null
  profile: null
  sc: git
  workspace_type: Application


To find your equivalent values, see the following screenshot and the AWS documentation here.

Recent events
Recent events

We are finally ready to go so click Start Building in the CircleCI dashboard and, with a little luck, your project should test, build and deploy successfully:

Test, build, and deploy with CircleCI
Test, build, and deploy with CircleCI

Gitlab

First, head over to https://about.gitlab.com/ and sign in.

If this is your first time using Gitlab, you will see the following screen. Click Create a project:

Homepage
Homepage

Next click the CI/CD for external repo tab and connect to Github.

CI/CD for external repo
CI/CD for external repo

You will be prompted for a personal access token from Github with repo access. To do so follow this short guide.

Authenticate
Authenticate

Next, connect your repository:

Connect to repository
Connect to repository

You should then see a dashboard with your project details:

Project details
Project details

The next step is to add the environment variables. Click Settings on the side bar and then click CI/CD on the popup.

Setting
Setting

Scroll down and add your AWS and Docker Hub credentials ( AWS_ACCESS_KEY_ID,  AWS_SECRET_ACCESS_KEY ,  DOCKER_USER and  DOCKER_PASS).

The next step is to .gitlab-ci.ymlfile to the root of your directory. If you look in the Github repository, you will see the following file:

image: node:latest # (1)

stages:
  - build
  - test
  - docker-deploy-image
  - aws-deploy

cache:
  paths:
    - node_modules/ # (2)

install_dependencies:
  stage: build
  script:
    - npm install # (3)
  artifacts:
    paths:
      - node_modules/

testing:
  stage: test
  script: npm test # (4)

docker-deploy-image:
  stage: docker-deploy-image
  image: docker:dind
  services:
    - docker:dind
  script:
    - echo "$DOCKER_PASS" | docker login -u "$DOCKER_USER" --password-stdin # (5)
    - docker build -t andrewbestbier/ci-comparison-blog .
    - docker push andrewbestbier/ci-comparison-blog

aws-deploy:
  image: 'python:latest'
  stage: aws-deploy
  before_script:
    - 'pip install awsebcli --upgrade' # (6)
  script:
    - eb deploy CiComparisonBlog-env


Let’s break down what’s happening during this build’s execution:

  1. The base image is a Node Docker container.
  2. The  node_modules cached.
  3. The packages are installed.
  4. The tests are run.
  5. Next, the Docker image is built and pushed to Docker Hub with the  DOCKER_USER  and  DOCKER_PASS  environment variables. Remember to change the repository to the one you created.
  6. The Elastic Beanstalk CLI tool is installed.
  7. The CLI is used to deploy the app to Elastic Beanstalk with eb deploy . This command works as we have already been authenticated invisibly by Gitlab with the  AWS_ACCESS_KEY_ID,  AWS_SECRET_ACCESS_KEY environment variables. Note how we have to specify the environment name in this command.

We are finally ready to go, so push up a small irrelevant commit and, with a little luck, your project should test, build and deploy successfully:

Test, build, and deploy
Test, build, and deploy

Travis CI

First, head over to https://travis-ci.com and sign in with Github:

Travis CI
Travis CI

You should then see a dashboard showing your connected projects. I have blocked out other personal projects I have. Click the small + button to add a new project:

Add a new project
Add a new project

Next, search for your Github repository and click the Settings button:

Settings
Settings

Scroll down and, like the other services, add your AWS and Docker Hub credentials ( AWS_ACCESS_KEY_ID ,  AWS_SECRET_ACCESS_KEY ,  DOCKER_USER and  DOCKER_PASS ).

The next step is to .travis.ymlfile to the root of your directory. If you look in the Github repository, you will see the following file:


language: node_js # (1)
node_js:
  - 'node'
services:
  - docker # (2)
jobs:
  include:
    - stage: test
      script:
        - npm install # (3)
        - npm test # (4)
    - stage: docker-deploy-image # (5)
      script: 
        - echo "$DOCKER_PASS" | docker login -u "$DOCKER_USER" --password-stdin 
        - docker build -t andrewbestbier/ci-comparison-blog .
        - docker push andrewbestbier/ci-comparison-blog
    - stage: deploy
      script: skip
      deploy: # (6)
        provider: elasticbeanstalk
        access_key_id: $AWS_ACCESS_KEY_ID
        secret_access_key: $AWS_SECRET_ACCESS_KEY
        region: 'eu-west-2'
        app: 'CI Comparison Blog'
        env: 'CiComparisonBlog-env'
        bucket_name: 'elasticbeanstalk-eu-west-2-094505317841'
        bucket_path: 'CI Comparison Blog'


Let’s break down what’s happening during this build’s execution:

  1. The base image is a Node Docker container.
  2. The Docker service is used so that Docker commands can later be used
  3. The packages are installed (Travis caches npm modules by default)
  4. The tests are run
  5. Next, the Docker image is built and pushed to Docker Hub with the  DOCKER_USER  and  DOCKER_PASS  environment variables.
  6. This step is quite different to other services as Travis offers its own Elastic Beanstalk deployment method (rather than using the CLI). One simply needs to specify the Elastic Beanstalk configuration details in the format Travis specifies and they take care of the rest.

We are finally ready to go so push up a small irrelevant commit and, with a little luck, your project should test, build and deploy successfully:

Test, build, and deploy with Travis CI
Test, build, and deploy with Travis CI

Buddy

Head over to Buddy and sign in with Github. Then click Create new project:

Create new project
Create new project

Buddy should intelligently work out that our project is an Express application. Click Add a new pipeline:

Add a new pipeline
Add a new pipeline

Specify a pipeline name (I picked CI Blog Post), select trigger On push and click Add a new pipeline.

Adding new pipeline
Adding new pipeline

We can then create a first action to install our dependencies and run the tests. Search for the Node.js action and click it.

Search Node.js
Search Node.js

You are then prompted for some bash commands to run. In our case, we wish to run yarn install and yarn test . Then click Add this action:

Adding action
Adding action

Next we want to build a Docker image from our source code. Search for the Build Image action:

Build Image action
Build Image action

Buddy automatically detects your Dockerfile so just click Add this action:

Add this action
Add this action

Next, we want to push the image we just build up to Docker Hub. Search for the Push Image action:

Push Image
Push Image

Buddy automatically uses the image built in the previous action. Provide your Docker Hub username, password, repository, and tags before clicking Add this action:

Providing Docker Hub credentials
Providing Docker Hub credentials

Finally, we want to deploy our files to Elastic Beanstalk so search for this action:

Search for deploy action
Search for deploy action

A modal will appear requesting your AWS user Access Key and Secret Access Key:

Requesting access key and secret access key
Requesting access key and secret access key

Select the AWS Region and your application should automatically appear. Then click Add this action:

AWS Region
AWS Region

Your pipeline should look like this:

Pipeline
Pipeline

Click Run pipeline and it should test, build and deploy your application successfully:

Run, build, and deploy Buddy
Run, build, and deploy Buddy" title="Run, build, and deploy Buddy

If you wish, you can also set up this pipeline with a buddy.yml file in your root directory:

- pipeline: "CI Blog Post"
  trigger_mode: "ON_EVERY_PUSH"
  ref_name: "master"
  ref_type: "BRANCH"
  trigger_condition: "ALWAYS"
  actions:
  - action: "Execute: yarn test"
    type: "BUILD"
    working_directory: "/buddy/ci-comparison-blog"
    docker_image_name: "library/node"
    docker_image_tag: "10"
    execute_commands:
    - "yarn install"
    - "yarn test"
    setup_commands:
    - "npm install -g gulp grunt-cli"
    mount_filesystem_path: "/buddy/ci-comparison-blog"
    shell: "BASH"
    trigger_condition: "ALWAYS"
  - action: "Build Docker image"
    type: "DOCKERFILE"
    dockerfile_path: "Dockerfile"
    trigger_condition: "ALWAYS"
  - action: "Push Docker image"
    type: "DOCKER_PUSH"
    login: "andrewbestbier"
    password: "secure!KmF0va9L3z4s450LWVlvNdHi1+6Z6+45vQbkHS4bWFo="
    docker_image_tag: "latest"
    repository: "andrewbestbier/ci-comparison-blog"
    trigger_condition: "ALWAYS"
  - action: "Upload files to Elastic Beanstalk/CI Comparison Blog"
    type: "ELASTIC_BEANSTALK"
    application_name: "CI Comparison Blog"
    environment: "e-zpfbesiqpa"
    environment_name: "CiComparisonBlog-env"
    region: "eu-west-2"
    trigger_condition: "ALWAYS"
    integration_id: 65587


AWS CodePipeline

Login to AWS and navigate to the CodePipeline service. Next click Create pipeline:

Create pipeline
Create pipeline

Next name your pipeline (I named mine CI-comparison) before clicking Next:

Naming pipeline
Naming pipeline

Next, you will need to add a pipeline source. Select Github, follow the login prompts, connect your repository, select the master branch and click Next:

Pipeline source
Pipeline source

We then need to add a build stage. Select AWS CodeBuild as a build provider and click Create project. This will open a new tab where you configure the build.

New project
New project

Name your build (I named mine CI-comparison) and scroll down the page.

Create build project
Create build project

Select Managed image, Ubuntu operating system, Standard runtime, and the 2.0 image. Also enable the Privileged flag as we wish to build Docker images before scrolling down further:

Configuration
Configuration

Next add the  DOCKER_USER and  DOCKER_PASS environment variables and

Environment variables.
Environment variables

Select use buildspec file as we will be using a buildspec.yml file in our repositories root directory to specify the build steps. Finally, click Continue to CodePipeline.

Build step specify
Build step specify

The tab will close and you will be taken back to your CodePipeline configuration. Click Next:

Add build stage
Add build stage

Finally, we need to add a deploy stage. Select AWS Elastic Beanstalk as the deploy provider and find your application before clicking Next:

Deploy provider
Deploy provider

Review your pipeline and click Create pipeline:

Create pipeline
Create pipeline

Now view the buildspec.yml file in the root of our directory:

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 10 # (1)
    commands:
      - echo "$DOCKER_PASS" | docker login -u "$DOCKER_USER" --password-stdin  # (2)
  pre_build:
    commands:
      - npm install # (3)
      - npm test # (4)
  build:
    commands:
      - docker build -t andrewbestbier/ci-comparison-blog . # (5)
  post_build:
    commands:
      - docker push andrewbestbier/ci-comparison-blog


Let’s break down what’s happening during this build’s execution:

  1. The Node version is specified.
  2. The build logs into our Docker account with the environment variables.
  3. The packages are installed.
  4. The tests are run.
  5. Next, the Docker image is built and pushed to Docker Hub.


Finally, click Release change and it should test, build and deploy your application successfully.

Test, build, and deploy AWS
Test, build, and deploy with AWS

Further Reading

The Complete CI/CD Collection [Tutorials]

What CI/CD Tool Should I Use?

Topics:
aws ,continious deployment ,continious integration ,travis ,circleci ,gitlab ,codepipeline ,buddyci ,tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}