Hands-On IoT DevOps: Day 1

Let's be honest, there is no match between the number of people writing about IoT or DevOps compared to the actual product developers. There is nothing wrong with that, but I always like to see how things work, rather than letting others convince me how cool is that new technology or product. That's why I decided to create a series of hands-on articles with my experiences in IoT device development using DevOps. Let's start!

First Things First: The Rise of IoT

About 10 years ago, IoT made the first steps towards a broader audience. The idea of having a tiny sensor attached to a microcontroller that has some kind of connectivity gave birth to the first IoT devices. That was huge — it was possible to have a small thing that can collect measurements from anything and transmit them wirelessly to a remote server. And that small thing could be either completely autonomous or attached to an existing device (e.g a home appliance) and make it smart. As you may expect, during the last 10 years, this led to the creation of numerous startups and developers working in the IoT ecosystem. Nowadays, most of the big players have also joined the IoT rush.

Why IoT DevOps? An Unexpected Challenge

Some things are never meant to happen, and that's what everybody was thinking back in the old days about the need for version control software at the device level — at the firmware. After all, who needs to version control a bunch of .c files that are developed by a single person? Strangely enough, things changed. There was a need for products with loads of new features, and the firmware increased in size and complexity. The firmware was now controlled by developer teams, each dedicated to one specific component. The need for collaboration systems was born with tools like CVS and SVN to help developers change their way of thinking about development.

When the first IoT devices were introduced, the same thing happened again. The device firmware was relatively simple, and everybody was happy with SVN, Git, and others. Gradually, there was an increased need for powerful IoT devices, and that started a lot of discussions regarding the quality of the delivered firmware, mainly in terms of stability and security. New tools had to be used or developed from scratch.

And then, two major events took place. The first was the fact that most of the IoT devices were mature enough to properly support Over-The-Air (OTA) updates. In other words, how cool is to update your smart thermostat from your mobile without the need to rip it off the wall, plug a JTAG cable, and flash a new firmware? The second event was the huge impact of DevOps best practices in web development. I mean, look how easy is to write a simple app and with a click of a button. This app will be deployed somewhere in the cloud, ready to be used.

These two events will now change the way IoT device development works. We can make IoT DevOps a reality. Now we have the tools to build, the tools to tes,t and the tools to deploy any connected device from anywhere in the world. Or do we? Let's see, and let's put those tools to the test!

Building the Story, Defining the Requirements

Now that we have identified the need for DevOps in IoT device development, we will dig into some of the most popular CI/CD tools and services available today and define the required criteria to see whether they can fit in IoT as well as they fit in web development, for example.

To achieve this, we will pretend that we are developing an actual IoT product using DevOps best practices that each tool or service provides. This will give us an answer if we are ready to use DevOps in IoT or there is a need for something new.

Let's assume that the IoT product to be developed is a vehicle tracking device solution that operates in a city, like on a bus for example, and sends the location plus some extra data to the city bus operator. As part of the product development, we have to develop a device-level application and push a number of commits per day. For every commit, we need to have:

• Build automation, so that our code can provide a valid binary for the device.
• Test automation, to make sure that we didn't break anything and that the application is behaving as expected in a production-like environment.
• Automatic deployment, so that we can update a number of devices, anywhere in the world.
• History, logging, and monitoring to keep track of everything.

From the above requirements, testing and deployment should apply to thousands, if not millions of IoT devices.

Selecting the Thing

Of course, the most important component to start with is the actual Thing, the IoT device. To make an IoT product, you need a good base, not only a proof of concept. An IoT product requires solutions with a small footprint, decent power consumption figures, reliability, and performance, all in one small package. For that reason, the very popular Arduino and Raspberry Pi boards were both out of the game.

I decided to go with the Particle Photon. The Photon is a tiny (and cheap!) development kit that includes a Wi-Fi module from USI with a powerful Cortex-M3 MCU that can store and run easily your device-level applications, has plenty of I/O to interface with all your sensors, supports OTA updates, and the provided SDK contains a lot of application examples to start with. Moreover, they provide a cloud service, which is good for device management and integrations with other web services.

Note that the wireless modules (Wi-Fi, BLE, etc.) are very popular and ideal for making an IoT product. Many big vendors provide similar solutions, like Marvell, Qualcomm, Microchip, Cypress, and so on. We can talk about IoT hardware in another article. What we discuss here applies more or less to all the above solutions.

Who Ordered IoT DevOps? The Application Firmware!

Like many others, Particle divides the firmware into parts. To simplify things, let's say that the firmware is divided into two parts, the core and the application firmware. The reason behind this is to have a modular approach that enables developers to work only on the application level and not mess with the low-level core functionality. Also, this is a good way to provide OTA updates only to the application part of the firmware.

For this series of articles, we will focus only on the application firmware. We will then check a number of DevOps tools and services and see if IoT DevOps can really work.

Since we want to build a vehicle tracking product and we have a Wi-Fi module, we will develop application firmware that retrieves location information using the Geolocation API from Google Maps with very good accuracy, about 30 meters or better. Very cool to get this level of accuracy without GPS, right? Fortunately, Particle has developed a location library that we can include in our application. More on this later.

Getting Ready

Do not expect any special workspace environment setup here, this is what we try to avoid by using DevOps best practices. There are only a few things that we should take care of:

1. Set Up the IoT Device

More or less, each IoT device comes preloaded with the core firmware and a demo application. This is true for Photon as well, so we only need to set up the Wi-Fi credentials to get the device online and then register it to the cloud. That will enable us to talk to the device with the provided REST API.

Particle has detailed instructions about the Photon setup. In short, we put the device in Access Point mode and, using a built-in web server, we set the Wi-Fi credentials to connect to our Wi-Fi router.

2. Set Up Google Maps Geolocation

There is also a nice tutorial on how you can get a Google API key and then use it for the Google Maps integration. No need to read the complete tutorial, check only the section referred to here to get the Google key for Particle's cloud.

3. Create the Application Firmware

We first need to create a repository for the application firmware. I used a fresh GitHub account to create a repository named 'vehicle-tracking' for our application firmware and added a main source file, which is empty for now. We will deal with this later.

Next, we fork the Particle 'Google Maps Device Locator' library that we mentioned before and integrate it as a submodule to our application repository under the folder libs/geolocator.

Now that we have our library, we can add some code to our main source file, vehicle_tracking.cpp:

// vehicle tracking application

#include "application.h"
#include "libs/geolocator/src/google-maps-device-locator.h"

PRODUCT_ID(PLATFORM_ID);
PRODUCT_VERSION(3);
SYSTEM_MODE(AUTOMATIC);

GoogleMapsDeviceLocator locator;

void setup() {
    Serial.begin(9600);
    // Issue a Wi-Fi scan request and send the results to the cloud every 30 secs
    locator.withLocatePeriodic(30);
}

void loop() {
    locator.loop();
}

Yes, that's right, Particle supports Arduino sketches. I just used the C++ version, which I guess is more or less identical. I made two calls to the library, one to scan for the available networks and send the results to the Particle cloud every 30 seconds and the other that runs forever to process all the library requests.

Last, we fork the 'firmware' repository (release/stable branch) from Particle, since it contains the core firmware and the makefiles to build our application firmware. Particle provides easier ways to build our application, but I prefer this method. It's what an embedded developer expects to see. What is missing is an ARM toolchain, which we'll deal with this later.

I don't want to go into too many details for the above setup, our main focus is to get ready to integrate DevOps in IoT device development. Nonetheless, if you have any questions, just leave a comment.

The CI/CD Gang

Jenkins, Travis CI, TeamCity, Bamboo, GitLab, CircleCI, Codefresh, Codeship, GoCD, VSTS, etc.

You've heard of them before. They are very popular and responsible for the creation of one of the biggest trends in software development, Continuous Integration and Continuous Delivery (CI/CD). Most of them use Docker containers and/or virtual machines to run multiple tasks like building and running test cases for an application. Nowadays, most of them run on the cloud, but there are still standalone solutions (did I say Jenkins?) that can be deployed on-premises.

So far so good, but how can we use them in an embedded product like ours? How can we integrate these services and tools with our application firmware? Do they still deliver what they promise? We will find it out very soon through a series of hands-on articles!

Any CI/CD Solutions for IoT?

It's very challenging to create a DevOps solution that provides continuous building, testing, and deployment of applications in embedded hardware instead of a hosted service for example. Some of the main challenges are listed below:

• The fact that we have to deal with hardware is challenging enough. If we add the number of IoT hardware platforms available today, things get worst.
• Application builds can get very complex, from generic cross toolchains to complex board support packages (BSPs) that need extra configuration.
• Continuous testing includes infrastructure testing, user testing, functional testing etc., and all these should be executed in a production-like environment. In other words, creating simulated environments in IoT is extremely difficult.
• The huge variety of IoT devices in terms of hardware and software creates a lot of challenges in creating generic software patterns in a DevOps solution. For example, find the similarities between a Cortex-M0 microcontroller running a simple scheduler with a few KBs of RAM against a 64bit ARMv8 monster with 1 GB RAM running Android. Both can be used to create an IoT product.
• There are so many IoT cloud platforms out there, which makes it really challenging to provide a universal CI/CD solution for IoT.

Looking for any good (or bad) CI/CD candidates for IoT is a tough task, and a Google search is not generous either. Still, I would like to mention two solutions that may fit well in our case.

The first is resin.io. It's not a CI/CD solution, but an IoT Platform with some DevOps touches like code deployment. Unfortunately, it supports only their own proprietary device OS based on Yocto Linux, which also limits the IoT hardware that can be used to only those that are powerful enough to run Linux. Regardless, it seems like a good solution for specific environments, and we should give it a try.

The other is Desploy, which is an attempt to provide full CI/CD in IoT product development. Desploy supports a number of IoT hardware platforms out of the box both for building, testing, and deploying phases. It provides test automation using Python scripts and API support from the most popular IoT cloud platforms as Python modules. We'll also give it a try!

Whats Next?

Let's call it a day. In the next article, we will start with Travis CI to see if it's good enough for IoT DevOps!