A New Era of Open-Source Streaming

Check out some of the newest advances that are contributing to a new era of open-source streaming.

NiFi 1.2.0 RC-1 

The new Apache NiFi is available for download and building with Maven and JDK 8 on GitHub. It will take a bit of time depending on your hardware.

When done, you can go to: 
 ../nifi-nifi-1.2.0-RC1/nifi-assembly/target/nifi-1.2.0-bin/nifi-1.2.0/logs 

I was able to run it and see over 25 new processors, including many for record processing, MySQL CDC, enhanced Azure support, Google Cloud support, and a faster UI. The best feature is the really great new features for processing whole files or records. You now have a lot of options for working with all the cloud providers natively with fast processors for AWS, Azure, and Google Cloud. You now have one solution to run on-premise, in the cloud, on the edge, and on a ton of devices. Data ingestion, transmission, transformation, validation, and orchestration can now be done everywhere with full NSA-level security. This is the killer feature for getting your data to and from anywhere.  

There's a great presentation on the coming features that I highly recommend reading.

If that was all that was coming, that would be pretty awesome. But there's a lot more. The distributed HDF version of NiFi will be bundled with Ambari, Ranger, Zookeeper, Storm, and Kafka in various configurations and options. Big Data streaming is becoming an open-source enterprise toolkit.

Spoiler: Even more cool stuff is coming!

NiFi Registry

NiFi Registry — details. This will be a registry for flow files, variables, and extensions to the platform. This is really enabling Big Data microservices and service discovery and reuse.

Schema Registry

See this presentation at DataWorks Summit Munich. This amazing, great-versioned schema server will integrate with NiFi, SAM, and anyone who can use a REST API and JSON.

Streaming Analytics Manager

See this presentation from Data Works Summit Munich. This runs on Apache Storm and will eventually run on Apache Beam, Spark, Flink, and all the streaming engines. With its killer UI, this is the next generation tool.