IoT DevOps Hands-On (Day 3): GitLab CI/CD and Friends

In this article, we continue our journey into the world of DevOps and IoT with GitLab CI/CD. That's right, GitLab has its own CI/CD solution and, actually, it's been used by many developers. Until recently, Gitlab CI/CD was only an end-to-end solution within the GitLab platform, but it seems that now they added support for external repositories as well.  So, let's see how Gitlab CI/CD can be used in DevOps for IoT development.

No, wait! I think it's also a good time to share with you some interesting findings of the remaining CI/CD tools, because I realized that they do not seem much different, in terms of features and design. Actually, they look very similar. For that reason, in this article, we will also say a few words about the rest of the CI/CD tools that are available.

Background

GitLab has integrated CI/CD pipelines to build, test, deploy, and monitor your code. You can run builds on multiple platforms like Windows, Linux, and Mac. Support for many languages is provided, like Java, Ruby, PHP, and C (yes, great for all the web developers out there but still nothing for the embedded world).

Like in the other CI/CD solutions, a special file called  .gitlab-ci.yml, should be added in the root directory of our repository to configure GitLab and trigger the execution of the defined jobs.  In GitLab, you can have a pipeline with multiple stages, e.g. build, test and deploy. Each stage can have a number of jobs that will run in parallel. The jobs of the next defined stage will run after the jobs from the previous stage have finished. I think this approach with the dedicated groups of jobs seems much better than Travis CI that we tested in the previous article.

But, let's see how GitLab CI/CD can help us in an IoT device development workflow. For example, how can we use it for functional testing in an IoT device?

GitLab CI/CD Setup

Since GitHub integration has been added recently in GitLab CI/CD, we don't need to move our project to a GitLab repository. That said, please note that this will be provided as a paid service from GitLab. Currently, they offer this integration for about 1 year for free.

Fortunately, the quick start guide is very helpful and I was really pleased to see that someone is actually caring about their documentation, great job guys!

To connect with GitHub, we need to go to https://github.com/settings/tokens/new and generate a new access token with repo and admin:repo_hook enabled. Then we create a new CI/CD for external repo project and we select GitHub.

Next, we can paste the previously generated access token and we will be able to get a list of all our GitHub repositories with the option to "connect" them with GitLab.

Now, everything is synchronized and every change of our GitHub repository will be reflected in the GitLab project. What is missing is to add the .gitlab-ci.yml file and start playing with it.

GitLab Playground 

In .gitlab-ci.yml, you can define one or more stages. Βy default, there are three stages: build, test, and deploy. We are free to define our job names, etc. An example to see the format of stages and jobs is shown below:

stages:
  - build
  - test
  - deploy

job 1:
  stage: build
  script: make build dependencies

job 2:
  stage: build
  script: make build artifacts

job 3:
  stage: test
  script: make test

job 4:
  stage: deploy
  script: make deploy

In the above example, the execution order is job1 and job2 (parallel), job3, job4. 

Now, based on the quick start guide, let's try to convert the .travis.yml file that we created in our previous article to a .gitlab-ci.yml file. 

variables:
    GIT_SUBMODULE_STRATEGY: none

before_script:
    - sed -i 's/git@github.com:/https:\/\/github.com\//' .gitmodules
    - git submodule update --init --recursive
    - apt-get update && apt-get install -y make libarchive-zip-perl git gcc-multilib vim

stages:
    - build
    - test
    - deploy

build:
    stage: build
    script:
        - pwd
        - git clone https://github.com/cpipilas/firmware.git firmware
        - cd firmware
        - git checkout release/stable
        - cd ..
        - wget https://launchpad.net/gcc-arm-embedded/4.9/4.9-2015-q3-update/+download/gcc-arm-none-eabi-4_9-2015q3-20150921-linux.tar.bz2
        - tar xjf gcc-arm*.tar.bz2
        - export PATH=$PATH:/builds/cpipilas/vehicle-tracking/gcc-arm-none-eabi-4_9-2015q3/bin
        - arm-none-eabi-gcc --version
        - mkdir particleGeoLoc && cp -Rf vehicle_tracking.cpp libs particleGeoLoc
        - cd firmware/main
        - make all PLATFORM=photon APPDIR=../../particleGeoLoc
        - cd ../../
    artifacts:
        paths:
            - firmware/modules/particleGeoLoc/target
            - firmware/modules/particleGeoLoc/target/particleGeoLoc.bin

.ota_update:
    stage: test
    dependencies:
        - build
    script:
        # ota update
        - curl -X PUT "https://api.particle.io/v1/devices/25001c001347343438323536?access_token=9f9d7874f34b93011193c8f09f6cfb3c09161851" -F file=@firmware/modules/particleGeoLoc/target/particleGeoLoc.bin -F file_type=binary
        # ping device
        - curl https://api.particle.io/v1/devices/25001c001347343438323536/ping -X PUT -d access_token=9f9d7874f34b93011193c8f09f6cfb3c09161851

As you can see, they look almost identical! Also, note that one job can be ignored by simply setting a dash in front of the job name — e.g ".ota_update:" above will be ignored. One more interesting thing is that each job can have dependencies from other jobs, which is very helpful if one wants to have a meaningful workflow. Last, we can define the artifacts from the build stage and after running the jobs they will be available for download from the web interface. Cool!

Since we created the required gitlab-ci.yml file, we are ready to go as soon as we push this file in our repository. Then, we can check the status of our pipelines and jobs from the GitLab CI/CD web interface.

On the bad side, if you're using a shared runner (virtual runtime environment), you may encounter big delays (up to 10 minutes) until you get an available runner. Dedicated runners are available, but they require extra configuration. For example, one option is to install runners on a Kubernetes cluster.

In general, I believe GitLab has done a better job in terms of flexibility compared with Travis CI for example.

GitLab CI/CD for IoT? No Thanks!

When it comes to IoT device development, we end up having the exact same problems with Travis CI that we covered in our previous article. There is no automation in terms of setting up the build environment for an IoT hardware platform, and there is no "what's next" after getting a binary from the build system. What this means is that we are out of luck when we try to write some unit tests for an IoT device. And it becomes more challenging when we start talking about functional testing. How can I check if my smart bulb actually lights up or goes from red to green? I guess you agree that we cannot rely on the result of an HTTP request, right? 

GitLab Friends?

Except for Travis CI and GitLab CI/CD, there are plenty of popular CI/CD tools and services out there like TeamCity, CircleCI, Bamboo, Codefresh, Codeship, and the list is endless. Although, for an embedded developer that is trying to create and/or maintain his precious IoT hardware, all these tools seem very similar in terms of the end result. They all provide a way to build a binary for the hardware by manually adding all the required commands to the build job of each CI/CD tool. They all provide a way to run some tests, usually limited to the communication API of the device with the rest of the world. 

Is this what we need? Of course not. What we need is to somehow add the actual hardware in the tests. That is what will make the biggest difference. Fully automated functional tests increase productivity, lead to robust products, reduce time to market, and save money. That's good news for Execs, but, even if you are the IoT device developer, automated tests make you sleep better as well — no more horror themes with flying devices in your dreams!

What Goes Around Comes Around

Nobody expects that a generic CI/CD tool or service will cover everything but it's obvious that they try hard for this to become more and more generic and to have new features and areas of support. But this race to perfection and completeness make them bad players for the specifics and, in our case, for the IoT device specifics. When we add the hardware to the equation, then everything is going south for the majority of the available CI/CD tools. It's where you'll be alone, trying to find ingenious solutions for testing, but that you know that in the end, you have to put your hands on the hardware and test it. You know that you have to create your unit and functional tests and test them one by one, manually. 

What's Next?

So far, we talked about some of the most popular CI/CD tools and whether they fit well in IoT device development. The results were not very encouraging. In our final series of articles, we are going to cover Spanner CI, a very promising solution for Continuous Integration, specifically for IoT device development. Stay tuned!