Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi

Working with IoT data is a layered process, not unlike a parfait. Scratch that, an Onion. In fact, an Onion Omega 2, which is a great device that I just got a few days ago, does IoT really easily. This is so much easier than setting up RPI or other platforms. It also has a ton of pluggable modules that stack on top of this small chip. It's pretty small-powered, but it's under $10.

The device is extremely well documented at their site.

Needed to run real tools, I added a USB stick and used that for storage and for extra SWAP space.

opkg update
opkg install kmod-usb-storage-extras e2fsprogs kmod-fs-ext4
umount /tmp/mounts/USB-A1
mkfs.ext4 /dev/sda1
mkdir /mnt/sda1
mount /dev/sda1 /mnt/sda1
mount /dev/sda1 /mnt/ ; tar -C /overlay -cvf - . | tar -C /mnt/ -xf - ; umount /mnt/
opkg update
opkg install block-mount
opkg update
opkg install swap-utils block-mount
dd if=/dev/zero of=/tmp/mounts/USB-A1/swap.page bs=1M count=256
mkswap /tmp/mounts/USB-A1/swap.page
swapon /tmp/mounts/USB-A1/swap.page
free

block detect > /etc/config/fstab
Adding GPS

ls /dev/ttyACM*
cat /dev/ttyACM0
opkg update
opkg install ogps
ubus list
/etc/init.d/rpcd restart
ubus call gps info
Using the GPS Expander, this is the JSON data returned from the utility:

{"age":0,"latitude":"40.2807","longitude":"-74.6418","elevation":"38.4","course":"","speed":"N"}
I then added Python and Paho MQTT Python Client for sending messages to my Cloud MQTT broker.

opkg install python
https://docs.onion.io/omega2-docs/installing-and-using-python.html#onion-python-modules
opkg install python-pip
pip install --upgrade setuptools
pip install paho-mqtt
crontab -e
/etc/init.d/cron restart
*/1 * * * * /opt/demo/run.sh

Once the data was sent an MQTT broker, it was easy to ingest with Apache NiFi. A few out-of-the-box processors do the work for me.

• ConsumeMQTT: Ingest MQTT messages from the broker (CloudMQTT, Mosquito, ...)

• EvaluateJSONPATH: Extra Fields from the JSON File

• UpdateAttribute: Format some parameters for SQL

• ReplaceText: Build my SQL String

• PutSQL: Upsert into Phoenix/HBase.

Setting the parameters is easy and avoids the dreaded Bobby Tables SQL attacks. So add your parameters and go. Note the type for speed, should read ${'speed'}.

This is the beautiful Web Console that comes prerunning on the tiny Onion Omega2 device.

And a report from the Table in Apache Zeppelin which comes with Hortonworks Data Platform 2.5 and 2.6.

I highly recommend the Onion Omega device with GPS dock, but make sure you add a USB 4 plug and a large, fast SSD card. You will need the extra virtual RAM for running and installing Python.