Automated Pipelines Using CircleCI

This topic was taken as a session as part of the #ThursdayDevops series we have here at Francium Tech. You can find the information about sessions in this Github repo.

We won't go into the details of the full DevOps process (this might be for another post), but we will start something simple: developing an automated pipeline for deployments. Before going into an explanation of CircleCI, we have to review the basics of continuous integration and continuous deployment.

Continuous Integration and Continuous Deployment

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

Continuous deployment (CD) is a strategy for software releases wherein any code commit that passes the automated testing phase is automatically released into the production environment, making changes that are visible to the software’s users.

This itself is a vast topic, you can read more about it here and here. Here is a sample pipeline setup for an automated build,

Here are a few points to be noted,

• For each and every check-in into a version control system, this pipeline would be triggered.
• The confidence of the build would increase as it goes through each stage.
• We can have an approval process for higher-level environments and have automated deployment to a development environment (remember this is an example; your requirements might vary).

Now let's take a look at setting up this pipeline with CircleCI.

CircleCI

CircleCI is a hosted CI server which integrates with Github very well.

Setting Up The Project

When you sign into CircleCI using a Github account, it will ask permission to read your projects. If you give permission, you can go to CircleCI and get all your projects listed there. Click on "Set Up Project" and proceed.

Helper Utilities

CircleCI nees us to create the configuration file at this path, PROJECT_ROOT/.circleci/config.yml. So create that file at the root of your project.

If you want to test the CircleCI configuration in local you can use these commands,

•  circleci config validate: Validates the yaml content of the CircleCI configuration

•  circleci local execute — job linting: Executes the job in local

If you want to setup the CircleCI CLI in local, please read this post.

Jobs

In the above sample pipeline diagram, each and every box is a job. It runs the actual set of steps given inside a machine. The machine that is running the steps is called an executor. There are multiple executors (Docker, Shell, VM), but we will use Docker Executor and create a "Hello World" example.

version: 2
jobs:
        build:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Hello World
                        command: echo "Hello World"

Here we declared that

• We are going to use CircleCI version 2
• Created one job called build. If we don’t have workflows (which we will talk about next) then that default job name should be build
• We used Docker executor with busybox as the image
• The list of steps to run inside the Busybox container

Workflows

But this example is not sufficient to build a complete pipeline. To create multiple jobs and link them together, we will need to create workflows.

version: 2
jobs:
        unit_test:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Unit Test
                        command: echo "Going to run Unit Test"
        linting:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Lint Test
                        command: echo "Going to run Lint Test"

workflows:
        version: 2
        sample_pipeline:
                jobs:
                        - unit_test
                        - linting

We created the workflow named sample_pipeline and gave 2 jobs to it.

• These two jobs, unit_test and linting would be running in parallel and one doesn’t depend on the output of other.
• Also each job would gets its own executor. In our case, both the jobs use Docker Executor.

Pipeline Setup

Here we will create multiple jobs with dependencies between them,

version: 2
jobs:
        unit_test:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Unit Test
                        command: echo "Going to run Unit Test"
        linting:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Lint Test
                        command: echo "Going to run Lint Test"
        build_artifact:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Build Artifact
                        command: echo "Going to run the build artifact"

        deploy_to_dev:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Deploy to Dev
                        command: echo "Going to deploy to dev"
        deploy_to_qa:
                docker:
                - image: busybox:latest
                steps:
                - run:
                        name: Deploy to QA
                        command: echo "Going to deploy to QA"
                - run:
                        name: Step no 2
                        command: echo "Going to deploy to QA again"
workflows:
        version: 2
        sample_pipeline:
                jobs:
                        - linting
                        - unit_test                                
                        - build_artifact:
                                requires:
                                - unit_test
                                - linting
                        - deploy_to_dev:
                                requires:
                                - build_artifact
                        - deploy_to_qa:
                                requires:
                                - deploy_to_dev

There are a few points to note here:

• Both linting and unit_test jobs would still run parallel
• Fan In: the build_artifact job would be triggered only when both unit_testandlinting are successful, so both linting & unit_test fan in to thebuild_artifact .
• A job can have multiple steps, as given in  deploy_to_qa. These steps are always executed in sequential order and always uses the same container.

Approval Process

The only missing part here is the approval process. If we don’t have that, then the whole pipeline would be running for each and every check-in, which we don’t want to have (again, this depends).

...
workflows:
        version: 2
        sample_pipeline:
                jobs:
                        - linting
                        - unit_test
                        - build_artifact:
                                requires:
                                - unit_test
                                - linting
                        - deploy_to_dev:
                                requires:
                                - build_artifact
                        - hold_for_approval:
                                type: approval
                                requires:
                                - deploy_to_dev
                        - deploy_to_qa:
                                requires:
                                - hold_for_approval

For readability, I have pasted only the workflows and left out the jobs part. Again, you can check the entire configuration here. Some points to be noted,

• hold_for_approval is a new job with type marked as approval.
• deploy_to_qa would be triggered only when hold_for_approval is done

Once everything is completed, this would be the pipeline output.

This post was originally published in Francium Tech Blogs.