Over a million developers have joined DZone.

TensorFlow and NiFi: Big Data AI Sandwich

DZone 's Guide to

TensorFlow and NiFi: Big Data AI Sandwich

I'm getting ready to speak at a conference about real-time ingesting and transforming sensor data and social data with NiFi and TensorFlow.

· AI Zone ·
Free Resource

I'm getting ready to head to Sydney to speak on Tensorflow and Apache NiFi for the DataWorks Summit in a month. I've been experimenting with some new things, as TensorFlow and Apache NiFi have updated their versions. I have also been working with Streaming Analytics Manager, Druid, and Superset — which is a really cool combination. I'll be adding more articles soon as we head towards the event, so keep an eye on the and Big Data space here at DZone.

I could really use some comments here on what areas are of most importance or interest. All content from the event will be shared here at DZone including slides, videos, and source code.


  • Python NLTK
  • Stanford CoreNLP
  • Python SpaCy
  • Python TextBlob
  • TensorFlow
  • Apache NiFi
  • Java Custom NiFi Processors
  • Social Data
  • Sensor Data
  • RDBMS Data
  • Logs
  • Public REST Feeds
  • Drones
  • Devices like Raspberry Pi
  • Real-Time Ingesting and Transforming Sensor Data and Social Data With NiFi and TensorFlow

    In this talk, I will show data engineers and architects how to run real-time TensorFlow Inception Image Recognition on images captured by remote sensors and images in tweets. In the same flow, I will also demonstrate how to apply real-time sentiment analysis and intelligent routing of data to Phoenix, email, and Slack. I will elaborate on a number of different sentiment analysis frameworks available for use within Apache NiFi including Python NLTK, Stanford CoreNLP, Python SpaCy, and Python TextBlob. This talk will be a deep dive into how to manage complex data flow pipelines ingesting from multiple streaming sources including social, public open data feeds, logs, drones, RDBMS and IoT with transformations, deep learning, machine learning, and business rules.

    This talk will be based on several articles I have written:

    Data engineers will be shown the power of Apache NiFi for loading diverse sources of data, applying transformations in-stream, routing based on attributes, adding sentiment data to workflows, running deep learning algorithms in the stream, and storing data into Apache Phoenix on HBase. In this talk, I will walk through each step in the process like ingesting each source, applying filters, performing transformations, converting types, picking and converting fields, and finally storing data to Apache Phoenix on HBase. A quick data analysis to show streaming updates to data will be done in Apache Zeppelin running on HDP 2.x. This is based on a few talks I have given at the Future of Data Princeton meet-ups on various ingestion and processing patterns with Apache NiFi.

    big data ,nifi ,artificial intelligence ,tensorflow ,data analytics ,real-time data

    Opinions expressed by DZone contributors are their own.

    {{ parent.title || parent.header.title}}

    {{ parent.tldr }}

    {{ parent.urlSource.name }}