Over a million developers have joined DZone.

Akka, Spark, or Kafka? Selecting the Right Streaming Engine [Video]

DZone's Guide to

Akka, Spark, or Kafka? Selecting the Right Streaming Engine [Video]

A look at the criteria to consider when selecting, plus the context and background to make good decisions when it comes to adopting, streaming frameworks.

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

Many engineers we talk to on a daily basis come to us with the same issue: that the batch-oriented architecture of Big Data–where data is captured in large, scalable stores, then processed later–is simply too slow. If you’re looking for a competitive advantage like our clients PayPal, HPE, Starbucks, and Capital One, then you’ve got to embrace streaming and “Fast Data” architectures, where data is processed as it arrives.

But for most people we’ve talked to, there is rarely a “one size fits all” technology that can handle all streaming use cases. With so many stream processing tools, which ones should you choose? There are several considerations when making the right selection for the specific needs of your application, such as:

  • Latency: How low is necessary? What types of tasks are you processing? 
  • Volume: How high is required? Is Complex Event Processing involved?
  • Integration with other tools: Which ones and how? We know that nothing lives in isolation in these systems.
  • Data processing: What kinds? In bulk? As individual events?

In this talk by Dean Wampler, PhD., VP of Fast Data Engineering at Lightbend, we’ll look at the criteria you need to consider when selecting technologies, plus the context and background to make good decisions when it comes to adopting streaming frameworks.

Using our Fast Data Platform as an example, which supports a host of Reactive and streaming technologies like Akka Streams, Kafka Streams, Apache Flink, Apache Spark, Mesosphere DC/OS, and our own Reactive Platform, we’ll look at how to serve particular needs and use cases in both Fast Data and microservices architectures.

Watch The Full Video (40 Min)


Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

big data ,akka ,spark ,kafka ,data streaming

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}