Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Integrating Keras (TensorFlow) YOLOv3 Into Apache NiFi Workflows

DZone's Guide to

Integrating Keras (TensorFlow) YOLOv3 Into Apache NiFi Workflows

Integrating live YOLO v3 feeds (TensorFlow) and ingesting their images and metadata.

· AI Zone ·
Free Resource

Insight for I&O leaders on deploying AIOps platforms to enhance performance monitoring today. Read the Guide.

For this article, I wanted to try the new YOLOv3 that's running in Keras.

Out of the box with video streaming, pretty cool:

git clone https://github.com/qqwweee/keras-yolo3 

wget https://pjreddie.com/media/files/yolov3.weights 

python convert.py yolov3.cfg yolov3.weights model_data/yolo.h5 

python yolo.py

See:

https://github.com/qqwweee/keras-yolo3

My article on Darknet original YOLO v3 is here:

https://community.hortonworks.com/articles/191259/integrating-darknet-yolov3-into-apache-nifi-workfl.html

Example JSON Output:

{
  "boxes" : "Found 8 boxes for img",
  "class7" : "diningtable",
  "score7" : "0.7484486",
  "left7" : "636",
  "right7" : "1096",
  "top7" : "210",
  "bottom7" : "693",
  "class6" : "sofa",
  "score6" : "0.31372178",
  "left6" : "1114",
  "right6" : "1276",
  "top6" : "172",
  "bottom6" : "381",
  "class5" : "chair",
  "score5" : "0.3455438",
  "left5" : "990",
  "right5" : "1048",
  "top5" : "183",
  "bottom5" : "246",
  "class4" : "chair",
  "score4" : "0.34554565",
  "left4" : "858",
  "right4" : "1000",
  "top4" : "186",
  "bottom4" : "244",
  "class3" : "chair",
  "score3" : "0.87056005",
  "left3" : "1114",
  "right3" : "1276",
  "top3" : "172",
  "bottom3" : "381",
  "class2" : "chair",
  "score2" : "0.9683409",
  "left2" : "958",
  "right2" : "1151",
  "top2" : "203",
  "bottom2" : "482",
  "class1" : "cup",
  "score1" : "0.49115792",
  "left1" : "691",
  "right1" : "770",
  "top1" : "84",
  "bottom1" : "229",
  "class0" : "person",
  "score0" : "0.9980049",
  "left0" : "187",
  "right0" : "709",
  "top0" : "2",
  "bottom0" : "720",
  "host" : "HW13125.local.fios-router.home",
  "end" : "1527443620.644303",
  "te" : "4.380625247955322",
  "battery" : 100,
  "systemtime" : "05/27/2018 13:53:40",
  "cpu" : 19.7,
  "diskusage" : "140607.7 MB",
  "memory" : 65.8,
  "yoloid" : "20180527175341_284c6051-a45d-4757-977a-fe1dd76f295b"
}

I created a YOLO ID so that the JSON, the ID, and the image would have that ID for keeping them in sync as we send them across various distributed networks into a cluster for storage. I also have some other metadata I thought would be helpful such as date time, run time, battery available, disk usage, cpu, and memory. Those are quick and easy to grab with Python and could be useful. This is running on a Mac laptop. This could be ported to the NVIDIA Jetson TX1. It may work on the RPI3 with Movidius, but I think it may be a touch slow. Even on a Mac with no GPU and some stuff running, I am getting an image every 2-3 seconds produced. It's very choppy, I would like to try this on an Ubuntu workstation with a few NVidia high-end GPUs and TensorFlow compiled for GPU.

NiFi Flow for Ingest of JSON and Images:

NiFi Server to process and store data in Hive and HBase:

Read JSON from the Logs Directory Written by Python 3:

Ingest Images from Images Directory Written by YOLO v3:

TensorFlow Analysis of Captured Image:

You can watch the data arrive:

To Write Data to HBase is Easy (Just create an HBase table with a column family):

Apache NiFi generates my Hive table:

Once we have a generate table it is populated with our data and we can query it in Apache Zeppelin (or any JDBC/ODBC Tool)

Using InferAvroSchema we had a schema created, we store it in Hortonworks Schema Registry for use.


Sample of Data Stored in HBase:

20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:boxes, timestamp=1527456044351, value=Found 9 boxes for img
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class0, timestamp=1527456044351, value=person
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class1, timestamp=1527456044351, value=chair
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class2, timestamp=1527456044351, value=chair
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class3, timestamp=1527456044351, value=chair
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class4, timestamp=1527456044351, value=chair
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class5, timestamp=1527456044351, value=chair
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class6, timestamp=1527456044351, value=chair
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class7, timestamp=1527456044351, value=sofa
 20180527175609_579164b3-031c-4993-bec1-50ff4cf58d1c       column=yolo:class8, timestamp=1527456044351, value=diningtable

Flow (Local, Server) and Zeppelin Notebook:

yolo-keras-json-server-save.xml

keras-tensorflow-yolo-v3-osx.xml

yolo-copy.json

Forked Python Code For Saving JSON and Images

https://github.com/tspannhw/yolo3-keras-tensorflow

Coming Soon:

Live Recording of YOLO v3 with Keras/TensorFlow recording of the capture stream.

TrueSight is an AIOps platform, powered by machine learning and analytics, that elevates IT operations to address multi-cloud complexity and the speed of digital transformation.

Topics:
keras ,tensorflow ,yolov3 ,apache nifi ,python ,deep learning ,hortonworks ,artificial intelligence ,ai

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}