Over a million developers have joined DZone.

Akka, Spark, or Kafka? Selecting the Right Streaming Engine [Video]

DZone's Guide to

Akka, Spark, or Kafka? Selecting the Right Streaming Engine [Video]

A look at the criteria to consider when selecting, plus the context and background to make good decisions when it comes to adopting, streaming frameworks.

· Big Data Zone ·
Free Resource

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

Many engineers we talk to on a daily basis come to us with the same issue: that the batch-oriented architecture of Big Data–where data is captured in large, scalable stores, then processed later–is simply too slow. If you’re looking for a competitive advantage like our clients PayPal, HPE, Starbucks, and Capital One, then you’ve got to embrace streaming and “Fast Data” architectures, where data is processed as it arrives.

But for most people we’ve talked to, there is rarely a “one size fits all” technology that can handle all streaming use cases. With so many stream processing tools, which ones should you choose? There are several considerations when making the right selection for the specific needs of your application, such as:

  • Latency: How low is necessary? What types of tasks are you processing? 
  • Volume: How high is required? Is Complex Event Processing involved?
  • Integration with other tools: Which ones and how? We know that nothing lives in isolation in these systems.
  • Data processing: What kinds? In bulk? As individual events?

In this talk by Dean Wampler, PhD., VP of Fast Data Engineering at Lightbend, we’ll look at the criteria you need to consider when selecting technologies, plus the context and background to make good decisions when it comes to adopting streaming frameworks.

Using our Fast Data Platform as an example, which supports a host of Reactive and streaming technologies like Akka Streams, Kafka Streams, Apache Flink, Apache Spark, Mesosphere DC/OS, and our own Reactive Platform, we’ll look at how to serve particular needs and use cases in both Fast Data and microservices architectures.

Watch The Full Video (40 Min)


Hortonworks Community Connection (HCC) is an online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub.  Join the discussion.

big data ,akka ,spark ,kafka ,data streaming

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}