Creating a DevOps Toolbox for Gitlab CI

So let’s talk about what we are going to do. Because Gitlab CI doesn’t have plugins or even a curl, the YAML can be a pain to write for several reasons: escape of special characters, double and single quotes, concatenation, etc.

So in this article, we are going to present a simple toolbox idea (I was inspired by DPL which I also encourage you to do). It’s a pretty simple idea that can help you a lot with complex pipelines and re-use of code inside the pipelines. Instead of writing everything inside your .gitlab-ci.yml, we are going to create a toolbox to perform actions.

You may also enjoy:  Reusable Code: The Good, the Bad, and the Ugly

The Problem

If we want to create a release inside of our Gitlab and we are using Community edition. We would have to do something like this:

xxxxxxxxxx

create_release:

  stage: deploy

  image: docker:latest

  tags:

    - docker_runner

  script:

    - curl --header 'Content-Type: application/json' --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" --data '{ "name": "New release", "tag_name": "v0.3", "description": "Super nice release", "milestones": ["v1.0", "v1.0-rc"], "assets": { "links": [{ "name": "hoge", "url": "https://google.com" }] }}' --request POST https://gitlab.example.com/api/v4/projects/24/releases

  only:

    - master

    - merge_request

Yeah, that’s not pretty right? And guess what? It won’t work without some effort to escape and make the quotes right, this is going to be a pain to fix and another huge problem in that all of your pipelines would need to copy and paste this horrible YAML.

Creating the Solution

So how are going to make this more readable and easier for us? You are going to need:

• Python 3.7
• Docker
• Appropriate libraries

Let’s start with basics, how do we create a new release with Python? Yep using the Python-Gitlab wrapper, so let’s start!

Before you start to code, you must set up your environment and do a:

pip install python

Then creates a new Python script called create_release.py, which will be create the Gitlab object and then support function to get the project ID. This will be useful for our release creation.

Ok, this code has a lot of problems, such as hardcoded information, re-usability, etc. But let’s see how this is going to look if we use it on our .gitlab-ci-yml:

Click!

Click! Click is a package that allows you to create CLI from Python scripts.

Click implementation will allow us to use parameters/options on our script in a very easy way. For this function, we will be receiving the Project ID directly from a pre-defined Gitlab-CI variable and the tag from git. For this article I won’t be covering the CI flow or how to generate the tag; I will just assume that you have your ways of creating and defining tags.

So let’s get started:

This is a sample of Click implementation:

This should do the trick. The only difference from the example is that we are using arguments instead of options. Arguments are required.

Ok now our CLI is implemented, but how it looks on our .gitlab-ci.yml?

But wait: we’re going to re-use this and right now, this Python script needs to be inside all of our repositories.

There’s a final catch! 

Toolbox Creation!

Remember that one of the requisites for this was Docker? Yeah, that’s right: we are going to create our runner that will serve as a DevOps toolbox. It will have all of our scripts at /bin/ so we can just invoke them!

So you will need a brand new Docker file to serve as a runner; I’m using the Python official image.

I would advise you to create a new repository with your runner and the Python scripts; the structure will be something like this:

Now for the Dockerfile keep it simple:

x

FROM python:latest

COPY scripts/ /usr/local/bin/

RUN apt update -y

Build the image, push it to your preferred registry, and now you can simply do this in your initial .gitlab-ci.yml:

So, what changed?

• We have moved the  create_release Python script to our DevOps tools repository and removed it from the project that we were working on.
• Now we assign this specific stage to be run inside our "devops_toolbox" image, which contains our "create_release" Python script. When you do this, we can invoke it directly because it's on the path.
• Script tag with our "create_release" Python without the "extension."

That’s a lot better and that will be re-usable from your container, so every stage that you need to execute it using the tag from your custom runner.

This is just an idea, you can create multiple binaries to do multiple actions, slack hooks, upload artifacts to Nexus/Artifactory, deploy binaries to your services, DPL, etc, then you will have a fully DevOps toolbox docker image to support your Continuous Integration and Delivery.

Summary

• Instead of having duplicate scripts inside your .gitlab-ci.yml, we created a Python script inside our repo.
• To make it more readable, we modified the Python script with Click!, turning it into a CLI.
• To make it even better and re-usable, we embedded it into a custom Docker image that will serve us as stage runner inside the Gitlab-CI.

Things that I didn't cover that you should look into:

• Python tests
• Error treatment; remember if your API returns 404, the script may return exit code 0, and you must treat that.
• The entire CI/CD flow

That's it for now, hope it helps anyone using Gitlab CI.