DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > A New Era of Open-Source Streaming

A New Era of Open-Source Streaming

There have been tons of new advances in open-source streaming technologies, including new processors, new paradigms and upscaled speed.

Tim Spann user avatar by
Tim Spann
CORE ·
May. 10, 17 · Big Data Zone · Analysis
Like (4)
Save
Tweet
4.67K Views

Join the DZone community and get the full member experience.

Join For Free

Check out some of the newest advances that are contributing to a new era of open-source streaming.

Image title

Image title

NiFi 1.2.0 RC-1 

The new Apache NiFi is available for download and building with Maven and JDK 8 on GitHub. It will take a bit of time depending on your hardware.

When done, you can go to: 
 ../nifi-nifi-1.2.0-RC1/nifi-assembly/target/nifi-1.2.0-bin/nifi-1.2.0/logs 

I was able to run it and see over 25 new processors, including many for record processing, MySQL CDC, enhanced Azure support, Google Cloud support, and a faster UI. The best feature is the really great new features for processing whole files or records. You now have a lot of options for working with all the cloud providers natively with fast processors for AWS, Azure, and Google Cloud. You now have one solution to run on-premise, in the cloud, on the edge, and on a ton of devices. Data ingestion, transmission, transformation, validation, and orchestration can now be done everywhere with full NSA-level security. This is the killer feature for getting your data to and from anywhere.  

There's a great presentation on the coming features that I highly recommend reading.

If that was all that was coming, that would be pretty awesome. But there's a lot more. The distributed HDF version of NiFi will be bundled with Ambari, Ranger, Zookeeper, Storm, and Kafka in various configurations and options. Big Data streaming is becoming an open-source enterprise toolkit.

Spoiler: Even more cool stuff is coming!

NiFi Registry

NiFi Registry — details. This will be a registry for flow files, variables, and extensions to the platform. This is really enabling Big Data microservices and service discovery and reuse.

Schema Registry

See this presentation at DataWorks Summit Munich. This amazing, great-versioned schema server will integrate with NiFi, SAM, and anyone who can use a REST API and JSON.

Streaming Analytics Manager

See this presentation from Data Works Summit Munich. This runs on Apache Storm and will eventually run on Apache Beam, Spark, Flink, and all the streaming engines. With its killer UI, this is the next generation tool.

Open source Big data

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Artificial Intelligence (AI) And Its Assistance in Medical Diagnosis
  • How to Classify NSFW (Not Safe for Work) Imagery with AI Content Moderation using Java
  • The Importance of Semantics for Data Lakehouses
  • How to Solve Context Propagation Challenges in Distributed Tracing

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo