DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Building a 24-Core Docker Swarm Cluster on Banana Pi Zero
  • Mastering Persistence: Why the Persistence Layer Is Crucial for Modern Java Applications
  • Beyond Code: The Crucial Role of Databases in Software
  • How VAST Data’s Platform Is Removing Barriers To AI Innovation

Trending

  • Microservices With Apache Camel and Quarkus (Part 5)
  • Memory Management in Java: An Introduction
  • Send Your Logs to Loki
  • Effective Tips for Debugging Complex Code in Java
  1. DZone
  2. Data Engineering
  3. IoT
  4. Ingesting GPS Data From Raspberry PI Zero Wireless With a USB GPS Device

Ingesting GPS Data From Raspberry PI Zero Wireless With a USB GPS Device

This guide delves into using Python and Apache NiFi to ingest GPS data from IoT devices. In this case, we'll use a Raspberry Pi Zero.

Tim Spann user avatar by
Tim Spann
CORE ·
May. 11, 17 · Tutorial
Like (3)
Save
Tweet
Share
7.57K Views

Join the DZone community and get the full member experience.

Join For Free

To quote my last article, "Working with IoT data is a layered process, not unlike a parfait." But whereas last time we worked with an Onion Omega 2 to ingest GPS data, we'll use a Raspberry Pi Zero. Let's run through the equipment you need and how to make it happen.

I recommend the BU-353-S4 USB GPS. It works well with Raspberry Pis and is very affordable. Connecting this to a RPIWZ, I can run this on a small battery and bring this everywhere for location tracking. Put it on your delivery truck, chasis, train, plane, drone, robot, and more. I'll track myself so I can own the data.

(Click to enlarge.)

What Do These GPS Fields Mean?

  • EPS = Error Estimate in Meter/Second

  • EPX = Estimated Longitude Error in Meters

  • EPV = Estimated Vertical Error in Meters

  • EPT = Estimated Timestamp Error

  • Speed = Speed!!!

  • Climb = Climb (Positive) or Sink (Negative) rate in meters per second of upwards or downwards movement.

  • Track = Course over ground in degrees from True North

  • Mode = NMEA mode; values are 0 - NA, 1 - No Fix, 2D and 3D.

My point is, "If you already have an estimate of the error, do something about it!"

Python Walkthrough

First, install the utilities you need for GPS and Python. We also install NTP to get as accurate time as possible.

sudo apt-get install gpsd gpsd-clients python-gps ntp


For testing to make sure that everything works, try two of these GPS utilities. Make sure you have the USB plugged in — you will need a RPIZero adapter to convert from little USB to normal size. I then connect a small USB hub to connect the GPS unit as well as sometimes mouse and keyboard. Get one of these, you will need it. You will also need an adapter from little to full size HDMI. You only really need the mouse, keyboard, and monitor while you are doing the first setup up Wi-Fi. Once that's set up, just SSH into your device and forget it. You'll want to keep an eye on these (we'll get to them shortly):

  •  cgps 

  • gpsmon 

  • gpxlogger dumps XML data in GPX format.

gpspipe -l -o test.json -p -w -n 10


Without -o, it goes to STDOUT/STIN.

These will work from the command line and give you a readout. It will take a few seconds or maybe a minute the first time to calibrate. If you don't get any numbers, stick your GPS on a window or put it outside.

If you have to manually run the GPS demon:

gpsd -n -D 2 /dev/ttyUSB0


I found some code to read the GPS sensor over USB with Python. From there, I modified the code for a slower refresh and no UI, as I want to just send this data to Apache NiFi over MQTT using Eclipse Paho MQTT client.

One enhancement I have considered is an offline mode to save all the data as a buffer and then on reconnect mass send the rest. Also you could search for other WiFi signals and try to use open and free ones. You probably want to then add SQL, encryption and some other controls. Or you could install and use Apache MiniFi Java or C++ agent on the Zero.

#! /usr/bin/python
# Based on
# Written by Dan Mandle http://dan.mandle.me September 2012
# License: GPL 2.0

import os
from gps import *
from time import *
import time
import threading
import json
import paho.mqtt.client as paho

gpsd = None 

class GpsPoller(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)
    global gpsd #bring it in scope
    gpsd = gps(mode=WATCH_ENABLE) #starting the stream of info
    self.current_value = None
    self.running = True #setting the thread running to true

  def run(self):
    global gpsd
    while gpsp.running:
      gpsd.next() #this will continue to loop and grab EACH set of gpsd info to clear the buffer

if __name__ == '__main__':
  gpsp = GpsPoller() # create the thread
  try:
    gpsp.start() # start it up
    while True:
      if gpsd.fix.latitude > 0:
        row = [ { 'latitude': str(gpsd.fix.latitude),
         'longitude': str(gpsd.fix.longitude),
         'utc': str(gpsd.utc),
         'time':   str(gpsd.fix.time),
         'altitude': str(gpsd.fix.altitude),
         'eps': str(gpsd.fix.eps),
         'epx': str(gpsd.fix.epx),
         'epv': str(gpsd.fix.epv),
         'ept': str(gpsd.fix.ept),
         'speed': str(gpsd.fix.speed),
         'climb': str(gpsd.fix.climb),
         'track': str(gpsd.fix.track),
         'mode': str(gpsd.fix.mode)} ]

        json_string = json.dumps(row)
        client = paho.Client()
        client.username_pw_set("jrfcwrim","UhBGemEoqf0D")
        client.connect("m13.cloudmqtt.com", 14162, 60)
        client.publish("rpiwzgps", payload=json_string, qos=0, retain=True)

        time.sleep(60)

  except (KeyboardInterrupt, SystemExit): #when you press ctrl+c
    gpsp.running = False
    gpsp.join() # wait for the thread to finish what it's doing


Example JSON Data

{
"track": "0.0",
"speed": "0.0",
"utc": "2017-05-01T23:49:46.000Z",
"epx": "8.938",
"epv": "29.794",
"altitude": "40.742",
"eps": "23.66",
"longitude": "-74.529216408",
"mode": "3",
"time": "2017-05-01T23:49:46.000Z",
"latitude": "40.268141521",
"climb": "0.0",
"ept": "0.005"
}


Ingest Any Data, Anywhere, Anytime, and from Any Dimension in Time and Space

(Click to enlarge)

  1. ConsumeMQTT

  2. InferAvroSchema

  3. ConvertJSONtoAvro

  4. MergeContent

  5. ConvertAvroToORC

  6. PutHDFS

Visualize Whirled Peas

We turn this raw data into Hive tables and then visualize into pretty tables and charts with Apache Zeppelin. You could also report with any ODBC and JDBC reporting tool like Tableau or PowerBI.

(Click to enlarge)

Store It Where?

su hdfs 
hdfs dfs -mkdir -p /rpwz/gps 
hdfs dfs -chmod -R 777 /rpwz/gps


If you store it, you can query it!

Why Do I Love Apache NiFi? Let Me Count the Ways....

Instead of hand-rolling some Hive DDL, NiFi will automagically generate all the DDL I need based on an inferred AVRO Schema (soon using Schema Registry lookup!). So you can easily drop in a file, convert to ORC, save to HDFS, generate an external Hive table and query it in seconds. All with no coding. Very easy to wire this to send a message to a front end via Web Sockets, JMS, AJAX, etc. So we can drop a file in S3 or HDFS, convert to ORC for mega fast LLAP queries and tell a front-end what the table is and it could query it.

Source code: https://community.hortonworks.com/repos/101680/rpi-zero-wireless-nifi-mqtt-gps.html?shortDescriptionMaxLength=140

Quick tip: Utilize Apache NiFi's scheduler to limit the number of calls you make to third-party services, a single NiFi instance can easily overwhelm most free tiers of services. I made 122 calls to Weather Underground in a few seconds. So set those times! For instance, for weather, once every 15-30 minutes or even 1 hour is good.

Note: GPS information can also be read from drones, cars, phones, and lots of custom sensors in IIoT devices.

USB Data (computing) raspberry pi Database

Published at DZone with permission of Tim Spann, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Building a 24-Core Docker Swarm Cluster on Banana Pi Zero
  • Mastering Persistence: Why the Persistence Layer Is Crucial for Modern Java Applications
  • Beyond Code: The Crucial Role of Databases in Software
  • How VAST Data’s Platform Is Removing Barriers To AI Innovation

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: