DZone
IoT Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > IoT Zone > Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi

Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi

Of course, the big gain from IoT is the data. A Big Data infrastructure can ensure you're getting your money's worth, so let's bring in Hadoop and NiFi.

Tim Spann user avatar by
Tim Spann
CORE ·
Apr. 28, 17 · IoT Zone · Tutorial
Like (2)
Save
Tweet
7.15K Views

Join the DZone community and get the full member experience.

Join For Free

Working with IoT data is a layered process, not unlike a parfait. Scratch that, an Onion. In fact, an Onion Omega 2, which is a great device that I just got a few days ago, does IoT really easily. This is so much easier than setting up RPI or other platforms. It also has a ton of pluggable modules that stack on top of this small chip. It's pretty small-powered, but it's under $10.

The device is extremely well documented at their site.

Needed to run real tools, I added a USB stick and used that for storage and for extra SWAP space.

Image title


opkg update
opkg install kmod-usb-storage-extras e2fsprogs kmod-fs-ext4
umount /tmp/mounts/USB-A1
mkfs.ext4 /dev/sda1
mkdir /mnt/sda1
mount /dev/sda1 /mnt/sda1
mount /dev/sda1 /mnt/ ; tar -C /overlay -cvf - . | tar -C /mnt/ -xf - ; umount /mnt/
opkg update
opkg install block-mount
opkg update
opkg install swap-utils block-mount
dd if=/dev/zero of=/tmp/mounts/USB-A1/swap.page bs=1M count=256
mkswap /tmp/mounts/USB-A1/swap.page
swapon /tmp/mounts/USB-A1/swap.page
free

block detect > /etc/config/fstab
Adding GPS

ls /dev/ttyACM*
cat /dev/ttyACM0
opkg update
opkg install ogps
ubus list
/etc/init.d/rpcd restart
ubus call gps info
Using the GPS Expander, this is the JSON data returned from the utility:

{"age":0,"latitude":"40.2807","longitude":"-74.6418","elevation":"38.4","course":"","speed":"N"}
I then added Python and Paho MQTT Python Client for sending messages to my Cloud MQTT broker.

opkg install python
https://docs.onion.io/omega2-docs/installing-and-using-python.html#onion-python-modules
opkg install python-pip
pip install --upgrade setuptools
pip install paho-mqtt
crontab -e
/etc/init.d/cron restart
*/1 * * * * /opt/demo/run.sh

Image title

Once the data was sent an MQTT broker, it was easy to ingest with Apache NiFi. A few out-of-the-box processors do the work for me.

  • ConsumeMQTT: Ingest MQTT messages from the broker (CloudMQTT, Mosquito, ...)

  • EvaluateJSONPATH: Extra Fields from the JSON File

  • UpdateAttribute: Format some parameters for SQL

  • ReplaceText: Build my SQL String

  • PutSQL: Upsert into Phoenix/HBase.

Image title
Image titleSetting the parameters is easy and avoids the dreaded Bobby Tables SQL attacks. So add your parameters and go. Note the type for speed, should read ${'speed'}.

This is the beautiful Web Console that comes prerunning on the tiny Onion Omega2 device.

Image title


And a report from the Table in Apache Zeppelin which comes with Hortonworks Data Platform 2.5 and 2.6.

Image title
I highly recommend the Onion Omega device with GPS dock, but make sure you add a USB 4 plug and a large, fast SSD card. You will need the extra virtual RAM for running and installing Python.

Data (computing) Apache NiFi

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • API Security Tools: What To Look For
  • Chopping the Monolith: The Demo
  • Flask vs. Django: Which Python Framework to Choose?
  • How To Check for JSON Insecure Deserialization (JID) Attacks With Java

Comments

IoT Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo