{{announcement.body}}
{{announcement.title}}

Creating a DevOps Toolbox for Gitlab CI

DZone 's Guide to

Creating a DevOps Toolbox for Gitlab CI

After running into a bit of trouble with working with Gitlab and Gitlab CI, this author demonstrates how to reuse your CI YAML with Docker.

· DevOps Zone ·
Free Resource

toolbox

Build a DevOps toolbox using Circle CI


I love Gitlab, really do, and when I started to work with Gitlab and GitlabCI in 2016, every project that we have to use it with, I struggle with some basic things. These are the things that we are going to cover, things like lack of plugins and re-usability of your CI YAML.

Gitlab DevOps Booth @ AWS Re:Invent 2019

Gitlab DevOps Booth @ AWS Re:Invent 2019

So let’s talk about what we are going to do. Because Gitlab CI doesn’t have plugins or even a curl, the YAML can be a pain to write for several reasons: escape of special characters, double and single quotes, concatenation, etc.

So in this article, we are going to present a simple toolbox idea (I was inspired by DPL which I also encourage you to do). It’s a pretty simple idea that can help you a lot with complex pipelines and re-use of code inside the pipelines. Instead of writing everything inside your .gitlab-ci.yml, we are going to create a toolbox to perform actions.

You may also enjoy:  Reusable Code: The Good, the Bad, and the Ugly

The Problem

If we want to create a release inside of our Gitlab and we are using Community edition. We would have to do something like this:

YAML




xxxxxxxxxx
1
10


 
1
create_release:
2
  stage: deploy
3
  image: docker:latest
4
  tags:
5
    - docker_runner
6
  script:
7
    - curl --header 'Content-Type: application/json' --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" --data '{ "name": "New release", "tag_name": "v0.3", "description": "Super nice release", "milestones": ["v1.0", "v1.0-rc"], "assets": { "links": [{ "name": "hoge", "url": "https://google.com" }] }}' --request POST https://gitlab.example.com/api/v4/projects/24/releases
8
  only:
9
    - master
10
    - merge_request


Yeah, that’s not pretty right? And guess what? It won’t work without some effort to escape and make the quotes right, this is going to be a pain to fix and another huge problem in that all of your pipelines would need to copy and paste this horrible YAML.

Creating the Solution

So how are going to make this more readable and easier for us? You are going to need:

  • Python 3.7
  • Docker
  • Appropriate libraries

Let’s start with basics, how do we create a new release with Python? Yep using the Python-Gitlab wrapper, so let’s start!

Before you start to code, you must set up your environment and do a:

Shell




x


 
1
pip install python



Then creates a new Python script called create_release.py, which will be create the Gitlab object and then support function to get the project ID. This will be useful for our release creation.

Python




xxxxxxxxxx
1


 
1
import os
2
import gitlab
3
 
          
4
#Creates a Gitlab Connection object, you must create your token on Gitlab
5
gl = gitlab.Gitlab('https://gitlab.example.com/gitlab', private_token=os.environ['your_token'])
6
 
          
7
#Creates a function to get the project internal ID from the CI.
8
def get_project(project_id):
9
   return gl.projects.get(project_id)



Now that we have our support function, let’s create the create release function.

Python




xxxxxxxxxx
1
22


 
1
import os
2
import gitlab
3
 
          
4
#Creates a Gitlab Connection object
5
gl = gitlab.Gitlab('https://gitlab.example.com/gitlab', private_token=os.environ['your_token'])
6
 
          
7
#Creates a function to get the project internal ID from the CI.
8
def get_project(project_id):
9
   return gl.projects.get(project_id)
10
  
11
#Creates the create_release function
12
def create_release(project_id, version):
13
   #Gets the Project ID to be used later and stores to a variable.
14
   project = get_project(project_id)
15
   #defines the information to go to the release.
16
   release_name = "My initial release"
17
   description = "My awesome release notes"
18
   #Creates the new release.
19
   release = project.releases.create({'name': release_name, 'tag_name': version, 'description': description })
20
   print(f'Created a new release with the tag: {tag_name} and project ID is: {project_id}')
21
#Creates the release!
22
create_release("24","v1.0")



Ok, this code has a lot of problems, such as hardcoded information, re-usability, etc. But let’s see how this is going to look if we use it on our .gitlab-ci-yml:

YAML




xxxxxxxxxx
1
12


1
before_script:
2
   - pip install python-gitlab
3
create_release:
4
  stage: deploy
5
  image: docker:latest
6
  tags:
7
    - docker_runner
8
  script:
9
    - python create_release.py
10
  only:
11
    - master
12
    - merge_request



Ok, that’s so much better, right? Let’s make it even better on the next step.

Click!

Click! Click is a package that allows you to create CLI from Python scripts.

Click implementation will allow us to use parameters/options on our script in a very easy way. For this function, we will be receiving the Project ID directly from a pre-defined Gitlab-CI variable and the tag from git. For this article I won’t be covering the CI flow or how to generate the tag; I will just assume that you have your ways of creating and defining tags.

So let’s get started:

This is a sample of Click implementation:

Python




xxxxxxxxxx
1
11


 
1
import click
2
@click.command()
3
@click.option('--count', default=1, help='Number of greetings.')
4
@click.option('--name', prompt='Your name',
5
              help='The person to greet.')
6
def hello(count, name):
7
    """Simple program that greets NAME for a total of COUNT times."""
8
    for x in range(count):
9
        click.echo('Hello %s!' % name)
10
if __name__ == '__main__':
11
    hello()



Alright so let’s use this example to make our   create_release function a CLI command:

Python




x


1
#!/usr/bin/env python
2
 
          
3
import os
4
import gitlab
5
import click
6
 
          
7
#Creates a Gitlab Connection object
8
gl = gitlab.Gitlab('https://gitlab.example.com/gitlab', private_token=os.environ['your_token'])
9
 
          
10
#Creates a function to get the project internal ID from the CI.
11
def get_project(project_id):
12
   return gl.projects.get(project_id)
13
  
14
#Creates the create_release function and decorate with Click.
15
@click.command()
16
@click.argument('project_id')
17
@click.argument('version')
18
def create_release(project_id, version):
19
   #Gets the Project ID to be used later and stores to a variable.
20
   project = get_project(project_id)
21
  
22
   #defines the information to go to the release.
23
   release_name = "My initial release"
24
   description = "My awesome release notes"
25
    
26
   #Creates the new release.
27
   release = project.releases.create({'name': release_name, 'tag_name': version, 'description': description })
28
   print(f'Created a new release with the tag: {tag_name} and project ID is: {project_id}')
29
  
30
if __name__ == '__main__':
31
    create_release()


This should do the trick. The only difference from the example is that we are using arguments instead of options. Arguments are required.

Ok now our CLI is implemented, but how it looks on our .gitlab-ci.yml?

YAML




xxxxxxxxxx
1
12


 
1
before_script:
2
   - pip install python-gitlab click
3
create_release:
4
  stage: deploy
5
  image: docker:latest
6
  tags:
7
    - docker_runner
8
  script:
9
    - ./create_release.py ${CI_PROJECT_ID} v1.0
10
  only:
11
    - master
12
    - merge_request



Alright, that’s so much better. Now we have a command-line Python script and we are using predefined variables to create our release!

But wait: we’re going to re-use this and right now, this Python script needs to be inside all of our repositories.

There’s a final catch! 

Toolbox Creation!

Remember that one of the requisites for this was Docker? Yeah, that’s right: we are going to create our runner that will serve as a DevOps toolbox. It will have all of our scripts at /bin/ so we can just invoke them!

So you will need a brand new Docker file to serve as a runner; I’m using the Python official image.

I would advise you to create a new repository with your runner and the Python scripts; the structure will be something like this:

DevOps Toolbox

DevOps Toolbox

Yes, you should have CI for your runner image as well.

Now for the Dockerfile keep it simple:

Dockerfile




x


1
FROM python:latest
2
COPY scripts/ /usr/local/bin/
3
RUN apt update -y


Build the image, push it to your preferred registry, and now you can simply do this in your initial .gitlab-ci.yml:

YAML




x


 
1
before_script:
2
   - pip install python-gitlab click
3
create_release:
4
  stage: deploy
5
  image: devops_toolbox:latest
6
  tags:
7
    - docker_runner
8
  script:
9
    - create-release ${CI_PROJECT_ID} v1.0
10
  only:
11
    - master
12
    - merge_request



So, what changed?

  • We have moved the  create_release Python script to our DevOps tools repository and removed it from the project that we were working on.
  • Now we assign this specific stage to be run inside our "devops_toolbox" image, which contains our "create_release" Python script. When you do this, we can invoke it directly because it's on the path.
  • Script tag with our "create_release" Python without the "extension."

That’s a lot better and that will be re-usable from your container, so every stage that you need to execute it using the tag from your custom runner.

This is just an idea, you can create multiple binaries to do multiple actions, slack hooks, upload artifacts to Nexus/Artifactory, deploy binaries to your services, DPL, etc, then you will have a fully DevOps toolbox docker image to support your Continuous Integration and Delivery.

Summary

  • Instead of having duplicate scripts inside your .gitlab-ci.yml, we created a Python script inside our repo.
  • To make it more readable, we modified the Python script with Click!, turning it into a CLI.
  • To make it even better and re-usable, we embedded it into a custom Docker image that will serve us as stage runner inside the Gitlab-CI.

Things that I didn't cover that you should look into:

  • Python tests
  • Error treatment; remember if your API returns 404, the script may return exit code 0, and you must treat that.
  • The entire CI/CD flow

That's it for now, hope it helps anyone using Gitlab CI.

Further Reading

In Defense of YAML

IoT DevOps Hands-On (Day 3): GitLab CI/CD and Friends 

Topics:
devops ,gitlab ,ci ,cd ,continuous integration ,python ,docker

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}