A New Era of Open-Source Streaming
A New Era of Open-Source Streaming
There have been tons of new advances in open-source streaming technologies, including new processors, new paradigms and upscaled speed.
Join the DZone community and get the full member experience.
Join For FreeThe Architect’s Guide to Big Data Application Performance. Get the Guide.
Check out some of the newest advances that are contributing to a new era of open-source streaming.
NiFi 1.2.0 RC-1
The new Apache NiFi is available for download and building with Maven and JDK 8 on GitHub. It will take a bit of time depending on your hardware.
When done, you can go to:
../nifi-nifi-1.2.0-RC1/nifi-assembly/target/nifi-1.2.0-bin/nifi-1.2.0/logs
I was able to run it and see over 25 new processors, including many for record processing, MySQL CDC, enhanced Azure support, Google Cloud support, and a faster UI. The best feature is the really great new features for processing whole files or records. You now have a lot of options for working with all the cloud providers natively with fast processors for AWS, Azure, and Google Cloud. You now have one solution to run on-premise, in the cloud, on the edge, and on a ton of devices. Data ingestion, transmission, transformation, validation, and orchestration can now be done everywhere with full NSA-level security. This is the killer feature for getting your data to and from anywhere.
There's a great presentation on the coming features that I highly recommend reading.
If that was all that was coming, that would be pretty awesome. But there's a lot more. The distributed HDF version of NiFi will be bundled with Ambari, Ranger, Zookeeper, Storm, and Kafka in various configurations and options. Big Data streaming is becoming an open-source enterprise toolkit.
Spoiler: Even more cool stuff is coming!
NiFi Registry
NiFi Registry — details. This will be a registry for flow files, variables, and extensions to the platform. This is really enabling Big Data microservices and service discovery and reuse.
Schema Registry
See this presentation at DataWorks Summit Munich. This amazing, great-versioned schema server will integrate with NiFi, SAM, and anyone who can use a REST API and JSON.
Streaming Analytics Manager
See this presentation from Data Works Summit Munich. This runs on Apache Storm and will eventually run on Apache Beam, Spark, Flink, and all the streaming engines. With its killer UI, this is the next generation tool.
Learn how taking a DataOps approach will help you speed up processes and increase data quality by providing streamlined analytics pipelines via automation and testing. Learn More.
Like This Article? Read More From DZone
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}