I'm getting ready to head to Sydney to speak on Tensorflow and Apache NiFi for the DataWorks Summit in a month. I've been experimenting with some new things, as TensorFlow and Apache NiFi have updated their versions. I have also been working with Streaming Analytics Manager, Druid, and Superset — which is a really cool combination. I'll be adding more articles soon as we head towards the event, so keep an eye on the and Big Data space here at DZone.
I could really use some comments here on what areas are of most importance or interest. All content from the event will be shared here at DZone including slides, videos, and source code.
Real-Time Ingesting and Transforming Sensor Data and Social Data With NiFi and TensorFlow
In this talk, I will show data engineers and architects how to run real-time TensorFlow Inception Image Recognition on images captured by remote sensors and images in tweets. In the same flow, I will also demonstrate how to apply real-time sentiment analysis and intelligent routing of data to Phoenix, email, and Slack. I will elaborate on a number of different sentiment analysis frameworks available for use within Apache NiFi including Python NLTK, Stanford CoreNLP, Python SpaCy, and Python TextBlob. This talk will be a deep dive into how to manage complex data flow pipelines ingesting from multiple streaming sources including social, public open data feeds, logs, drones, RDBMS and IoT with transformations, deep learning, machine learning, and business rules.
This talk will be based on several articles I have written:
- Ingest Remote Camera Images From Raspberry Pi
- Analyzing Images in HDF 2.0 Using TensorFlow
- HDF 2.0 Flow for Processing Real-Time Tweets
- Incrementally Streaming RDBMS Data to Your Hadoop DataLake
- Ingesting Remote Sensor Feeds Into Apache Phoenix via Apache NiFi
- Reading Sensor Data From Remote Sensors on Raspberry Pis
- Routing Logs through Apache NiFi to Phoenix, HDFS, and Offsite to Sumologic for Log Analytics
- Using Sentiment Analysis and NLP Tools With HDP 2.5 and HDF 2.1
- Processing Social Media Feeds in Stream With Apache NiFi 1.0.0 and NLTK
Data engineers will be shown the power of Apache NiFi for loading diverse sources of data, applying transformations in-stream, routing based on attributes, adding sentiment data to workflows, running deep learning algorithms in the stream, and storing data into Apache Phoenix on HBase. In this talk, I will walk through each step in the process like ingesting each source, applying filters, performing transformations, converting types, picking and converting fields, and finally storing data to Apache Phoenix on HBase. A quick data analysis to show streaming updates to data will be done in Apache Zeppelin running on HDP 2.x. This is based on a few talks I have given at the Future of Data Princeton meet-ups on various ingestion and processing patterns with Apache NiFi.