Over a million developers have joined DZone.

Ingesting Flight Data ADS-B USB Receiver With Apache NiFi 1.5+

DZone's Guide to

Ingesting Flight Data ADS-B USB Receiver With Apache NiFi 1.5+

· Big Data Zone ·
Free Resource

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

I'm using the FlightAware Pro Stick plus an ADS-B USB receiver with a built-in filter on a Mac. I should hook this up to one of my Raspberry Pis and add a long antenna. For the most part, I am not getting a lot of data. You need a good antenna, a good location, and nothing blocking your signal. It also depends on what air traffic is nearby.

For a proof of concept, it's pretty cool to see air data going through a cheap USB stick into a computer stored in a file and loaded into Apache NiFi to send on for data processing.

There is a web server you can run to see the planes on a map, which is pretty cool, but I want to just ingest the data for processing.

You need three pieces of equipment: an antenna and a wire to connect it and the FlightAware USB stick. This is recommended for a Raspberry Pi, but I ran this on my Mac.

Image title

If you wish to watch the data flash by in a command-line interface, you can run with an interactive flag and watch all the updates.

We are dumping the data as it streams as raw text into a file. A snippet of it tailed in Apache NiFi is shown below:

We are also ingesting ADS-B data that is provided by an open data REST API from here.

Like everything else, we may want to add a schema to parse into records.

Our ingest flow:

I am getting the REST data from the ADSB Exchange REST API, tailing the raw text dump from dump1090 and reading the aircraft history JSON files produced by dump1090, as well.

For further processing, I send the Aircraft History JSON files to my server cluster to send to a cloud-hosted MongoDB database thanks to a free tier from mLab.

And our data quickly arrives as JSON in Mongo.

The main install is the dump1090 GitHub and is pretty straightforward.

Installation on OSX:

brew update brew install librtlsdr pkg-config


./run2.sh >> raw.txt 2>&1 run2.sh ./dump1090 --net --lat 40.265887 --lon -74.534610 --modeac --mlat --write-json-every 1 --json-location-accuracy 2 --write-json /volumes/seagate/projects/dump1090/data

I have entered my local latitude and longitude above. I also write to a local directory that we will read from in Apache NiFi.

Example data:

{ "now" : 1507344034.5, "messages" : 1448, "aircraft" : [ {"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1}, {"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9}, {"hex":"a8bcdd","flight":"NKS710 ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0}, {"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2}, {"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1}, {"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9}, {"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2} ] } cat /usr/local/var/dump1090-mut-data/history_75.json { "now" : 1507344034.5, "messages" : 1448, "aircraft" : [ {"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1}, {"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9}, {"hex":"a8bcdd","flight":"NKS710 ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0}, {"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2}, {"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1}, {"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9}, {"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2} ] }

There is also an open data API available here.

So I grabbed , again using my Latitude and Longitude.

Hortonworks Community Connection (HCC) is an online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub.  Join the discussion.

big data ,adsb ,sensors ,iot ,hortonworks ,apachenifi

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}