DZone
IoT Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > IoT Zone > IoT: Augmenting GPS Data with Weather (Part 2)

IoT: Augmenting GPS Data with Weather (Part 2)

You can add weather data to GPS streams by geocoding latitude and longitude values to City and State variables and retrieving reports from Weather Underground.

Tim Spann user avatar by
Tim Spann
CORE ·
May. 12, 17 · IoT Zone · Tutorial
Like (5)
Save
Tweet
5.16K Views

Join the DZone community and get the full member experience.

Join For Free

Before you continue, if you haven't already, take a look at Part 1, where we set up our system. So with Hortonworks Data Flow's Flow Processing, I can augment any stream of data in motion. My GPS data stream from my RPIZW is returning latitude and longitude. That is valuable data that can be input to a lot of different APIs. One that seemed relevant to me was the current weather conditions where the device was. You could also do some predictive analytics to figure where you might be and check the current weather conditions or a weather forecast depending on how long it will take you to get there.

A GPS is giving us latitude and longitude that can be used as input to many kinds of APIs. Once use case that works for a moving device is checking the current weather conditions where you are. Weather Underground has a REST API call that will convert Lat/Long to City/State (there are other APIs that will do this as well, feel free to post them). Once I have City and State, I use that to get current weather conditions where this device is.

Apache NiFi Flow

(Click to enlarge)

Augment Live Data

EvaluateJsonPath: Create two variables (latitude and longitude) by extracting JSON fields from the flow file sent via MQTT. $.latitude and $.longitude

EvaluateJsonPath: Create two variables (City and State) by extracting JSON fields from the flow file sent from REST call. $.location.city $.location.state.

InvokeHTTP: City has space and perhaps other HTTP GET unfriendly data.

EvaluateJsonPath: Extract all the weather fields I am interested in, such as $.current_observation.dewpoint_string.

UpdateAttribute: A few field names Avro Schema's do not like. ${'Last-Modified'} I convert that to lastModified.

AttributesToJSON: Convert just the fields I want to a new flow file. Set property to flowfile-content.

temp_c,temp_f,wind_mph,wind_degrees,temperature_string,weather,feelslike_string,state,longitude,windchill_string,wind_string,relative_humidity,pressure_mb,observation_time,city,windchill_f,windchill_c,latitude,precip_today_string,lastModified,dewpoint_string,heat_index_string,visibility_mi


InferAvroSchema: Let Nifi decide what this AVRO record should look like. I will add a part three using the schema registry and Apache NiFi 1.2, which will use a schema lookup.

ConvertJSONtoAvro: And make it SNAPPY!

MergeContent: As AVRO.

ConvertAvroToORC: ORC is the optimal choice for Hive LLAP and Spark SQL to query the data.

PutHDFS: Point to your configuration Nifi relative file system file /etc/hadoop/conf/core-site.xml. I like to set for replace conflict resolution and set your directory. Make sure you created your file directory and that Nifi has permissions to write there.

The quickest is to log into a machine with a HDFS client.

su hdfs
hdfs dfs -mkdir -p /rpwz/gpsweather   
hdfs dfs -chmod -R 777 /rpwz


Enhancement Ideas

Limit the number of calls to Weather Underground, try other weather services like NOAA and Weather Source. You may want to try other nation's weather APIs as well.

Create an External Hive Table and Display the Data (Zeppelin Can Do That)





Zeppelin lets me create the table with the DDL produced by the stream.

Then I can query it easily.

%jdbc(hive)
CREATE EXTERNAL TABLE IF NOT EXISTS gpsweather (observation_time STRING, dewpoint_string STRING, city STRING, windchill_f STRING, windchill_c STRING, latitude STRING, precip_today_string STRING, temp_c STRING, temp_f STRING, windchill_string STRING, wind_mph STRING, wind_degrees STRING, temperature_string STRING, weather STRING, feelslike_string STRING, wind_string STRING, heat_index_string STRING, state STRING, lastModified STRING, relative_humidity STRING, pressure_mb STRING, visibility_mi STRING, longitude STRING) 
STORED AS ORC LOCATION '/rpwz/gpsweather'
%jdbc(hive) 
select * from gpsweather 


Note: Weather Underground has a free API that can use up to a certain number of calls. You will easily blast past that if you are tracking live objects, even if you decide to only enrich when points change or at 15-minute intervals. For testing, you may want to store weather underground results in JSON files in a directory and use GetFile to stand in for those calls. The same with data from your devices. Download the files from the Data Provenance and you can use them as stand-ins for live data for integration testing, especially when you may be on a plane or somewhere where you cannot reach your device.

Data stream IoT

Published at DZone with permission of Tim Spann, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Streaming ETL with Apache Kafka in the Healthcare Industry
  • Revoking Access to JWTs With a Blacklist/Deny List
  • Artificial Intelligence (AI) And Its Assistance in Medical Diagnosis
  • Five Tips to Fasten Your Skewed Joins in Apache Spark

Comments

IoT Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo