Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi
Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi
Of course, the big gain from IoT is the data. A Big Data infrastructure can ensure you're getting your money's worth, so let's bring in Hadoop and NiFi.
Join the DZone community and get the full member experience.
Join For FreeWorking with IoT data is a layered process, not unlike a parfait. Scratch that, an Onion. In fact, an Onion Omega 2, which is a great device that I just got a few days ago, does IoT really easily. This is so much easier than setting up RPI or other platforms. It also has a ton of pluggable modules that stack on top of this small chip. It's pretty small-powered, but it's under $10.
The device is extremely well documented at their site.
Needed to run real tools, I added a USB stick and used that for storage and for extra SWAP space.
opkg update
opkg install kmod-usb-storage-extras e2fsprogs kmod-fs-ext4
umount /tmp/mounts/USB-A1
mkfs.ext4 /dev/sda1
mkdir /mnt/sda1
mount /dev/sda1 /mnt/sda1
mount /dev/sda1 /mnt/ ; tar -C /overlay -cvf - . | tar -C /mnt/ -xf - ; umount /mnt/
opkg update
opkg install block-mount
opkg update
opkg install swap-utils block-mount
dd if=/dev/zero of=/tmp/mounts/USB-A1/swap.page bs=1M count=256
mkswap /tmp/mounts/USB-A1/swap.page
swapon /tmp/mounts/USB-A1/swap.page
free
block detect > /etc/config/fstab
Adding GPS
ls /dev/ttyACM*
cat /dev/ttyACM0
opkg update
opkg install ogps
ubus list
/etc/init.d/rpcd restart
ubus call gps info
Using the GPS Expander, this is the JSON data returned from the utility:
{"age":0,"latitude":"40.2807","longitude":"-74.6418","elevation":"38.4","course":"","speed":"N"}
I then added Python and Paho MQTT Python Client for sending messages to my Cloud MQTT broker.
opkg install python
https://docs.onion.io/omega2-docs/installing-and-using-python.html#onion-python-modules
opkg install python-pip
pip install --upgrade setuptools
pip install paho-mqtt
crontab -e
/etc/init.d/cron restart
*/1 * * * * /opt/demo/run.sh
Once the data was sent an MQTT broker, it was easy to ingest with Apache NiFi. A few out-of-the-box processors do the work for me.
ConsumeMQTT: Ingest MQTT messages from the broker (CloudMQTT, Mosquito, ...)
EvaluateJSONPATH: Extra Fields from the JSON File
UpdateAttribute: Format some parameters for SQL
ReplaceText: Build my SQL String
PutSQL: Upsert into Phoenix/HBase.
Setting the parameters is easy and avoids the dreaded Bobby Tables SQL attacks. So add your parameters and go. Note the type for speed, should read ${'speed'}.
This is the beautiful Web Console that comes prerunning on the tiny Onion Omega2 device.
And a report from the Table in Apache Zeppelin which comes with Hortonworks Data Platform 2.5 and 2.6.
I highly recommend the Onion Omega device with GPS dock, but make sure you add a USB 4 plug and a large, fast SSD card. You will need the extra virtual RAM for running and installing Python.
Like This Article? Read More From DZone
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}