Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi

DZone 's Guide to

Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi

Of course, the big gain from IoT is the data. A Big Data infrastructure can ensure you're getting your money's worth, so let's bring in Hadoop and NiFi.

· IoT Zone ·
Free Resource

Working with IoT data is a layered process, not unlike a parfait. Scratch that, an Onion. In fact, an Onion Omega 2, which is a great device that I just got a few days ago, does IoT really easily. This is so much easier than setting up RPI or other platforms. It also has a ton of pluggable modules that stack on top of this small chip. It's pretty small-powered, but it's under $10.

The device is extremely well documented at their site.

Needed to run real tools, I added a USB stick and used that for storage and for extra SWAP space.

Image title

opkg update
opkg install kmod-usb-storage-extras e2fsprogs kmod-fs-ext4
umount /tmp/mounts/USB-A1
mkfs.ext4 /dev/sda1
mkdir /mnt/sda1
mount /dev/sda1 /mnt/sda1
mount /dev/sda1 /mnt/ ; tar -C /overlay -cvf - . | tar -C /mnt/ -xf - ; umount /mnt/
opkg update
opkg install block-mount
opkg update
opkg install swap-utils block-mount
dd if=/dev/zero of=/tmp/mounts/USB-A1/swap.page bs=1M count=256
mkswap /tmp/mounts/USB-A1/swap.page
swapon /tmp/mounts/USB-A1/swap.page

block detect > /etc/config/fstab
Adding GPS

ls /dev/ttyACM*
cat /dev/ttyACM0
opkg update
opkg install ogps
ubus list
/etc/init.d/rpcd restart
ubus call gps info
Using the GPS Expander, this is the JSON data returned from the utility:

I then added Python and Paho MQTT Python Client for sending messages to my Cloud MQTT broker.

opkg install python
opkg install python-pip
pip install --upgrade setuptools
pip install paho-mqtt
crontab -e
/etc/init.d/cron restart
*/1 * * * * /opt/demo/run.sh

Image title

Once the data was sent an MQTT broker, it was easy to ingest with Apache NiFi. A few out-of-the-box processors do the work for me.

  • ConsumeMQTT: Ingest MQTT messages from the broker (CloudMQTT, Mosquito, ...)

  • EvaluateJSONPATH: Extra Fields from the JSON File

  • UpdateAttribute: Format some parameters for SQL

  • ReplaceText: Build my SQL String

  • PutSQL: Upsert into Phoenix/HBase.

Image title
Image titleSetting the parameters is easy and avoids the dreaded Bobby Tables SQL attacks. So add your parameters and go. Note the type for speed, should read ${'speed'}.

This is the beautiful Web Console that comes prerunning on the tiny Onion Omega2 device.

Image title

And a report from the Table in Apache Zeppelin which comes with Hortonworks Data Platform 2.5 and 2.6.

Image title
I highly recommend the Onion Omega device with GPS dock, but make sure you add a USB 4 plug and a large, fast SSD card. You will need the extra virtual RAM for running and installing Python.

apache nifi, connected devices, data lake, iot, tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}