Creating a DevOps Toolbox for Gitlab CI
Creating a DevOps Toolbox for Gitlab CI
After running into a bit of trouble with working with Gitlab and Gitlab CI, this author demonstrates how to reuse your CI YAML with Docker.
Join the DZone community and get the full member experience.Join For Free
I love Gitlab, really do, and when I started to work with Gitlab and GitlabCI in 2016, every project that we have to use it with, I struggle with some basic things. These are the things that we are going to cover, things like lack of plugins and re-usability of your CI YAML.
So let’s talk about what we are going to do. Because Gitlab CI doesn’t have plugins or even a curl, the YAML can be a pain to write for several reasons: escape of special characters, double and single quotes, concatenation, etc.
So in this article, we are going to present a simple toolbox idea (I was inspired by DPL which I also encourage you to do). It’s a pretty simple idea that can help you a lot with complex pipelines and re-use of code inside the pipelines. Instead of writing everything inside your .gitlab-ci.yml, we are going to create a toolbox to perform actions.
You may also enjoy: Reusable Code: The Good, the Bad, and the Ugly
If we want to create a release inside of our Gitlab and we are using Community edition. We would have to do something like this:
Yeah, that’s not pretty right? And guess what? It won’t work without some effort to escape and make the quotes right, this is going to be a pain to fix and another huge problem in that all of your pipelines would need to copy and paste this horrible YAML.
Creating the Solution
So how are going to make this more readable and easier for us? You are going to need:
- Python 3.7
- Appropriate libraries
Let’s start with basics, how do we create a new release with Python? Yep using the Python-Gitlab wrapper, so let’s start!
Before you start to code, you must set up your environment and do a:
Then creates a new Python script called create_release.py, which will be create the Gitlab object and then support function to get the project ID. This will be useful for our release creation.
Ok, this code has a lot of problems, such as hardcoded information, re-usability, etc. But let’s see how this is going to look if we use it on our .gitlab-ci-yml:
Click! Click is a package that allows you to create CLI from Python scripts.
Click implementation will allow us to use parameters/options on our script in a very easy way. For this function, we will be receiving the Project ID directly from a pre-defined Gitlab-CI variable and the tag from git. For this article I won’t be covering the CI flow or how to generate the tag; I will just assume that you have your ways of creating and defining tags.
So let’s get started:
This is a sample of Click implementation:
create_releasefunction a CLI command:
This should do the trick. The only difference from the example is that we are using arguments instead of options. Arguments are required.
Ok now our CLI is implemented, but how it looks on our .gitlab-ci.yml?
But wait: we’re going to re-use this and right now, this Python script needs to be inside all of our repositories.
There’s a final catch!
Remember that one of the requisites for this was Docker? Yeah, that’s right: we are going to create our runner that will serve as a DevOps toolbox. It will have all of our scripts at /bin/ so we can just invoke them!
So you will need a brand new Docker file to serve as a runner; I’m using the Python official image.
I would advise you to create a new repository with your runner and the Python scripts; the structure will be something like this:
Now for the Dockerfile keep it simple:
Build the image, push it to your preferred registry, and now you can simply do this in your initial .gitlab-ci.yml:
So, what changed?
- We have moved the
create_releasePython script to our DevOps tools repository and removed it from the project that we were working on.
- Now we assign this specific stage to be run inside our "devops_toolbox" image, which contains our "create_release" Python script. When you do this, we can invoke it directly because it's on the path.
- Script tag with our "create_release" Python without the "extension."
That’s a lot better and that will be re-usable from your container, so every stage that you need to execute it using the tag from your custom runner.
This is just an idea, you can create multiple binaries to do multiple actions, slack hooks, upload artifacts to Nexus/Artifactory, deploy binaries to your services, DPL, etc, then you will have a fully DevOps toolbox docker image to support your Continuous Integration and Delivery.
- Instead of having duplicate scripts inside your .gitlab-ci.yml, we created a Python script inside our repo.
- To make it more readable, we modified the Python script with Click!, turning it into a CLI.
- To make it even better and re-usable, we embedded it into a custom Docker image that will serve us as stage runner inside the Gitlab-CI.
Things that I didn't cover that you should look into:
- Python tests
- Error treatment; remember if your API returns 404, the script may return exit code 0, and you must treat that.
- The entire CI/CD flow
That's it for now, hope it helps anyone using Gitlab CI.
In Defense of YAML
Opinions expressed by DZone contributors are their own.