Real-Time Stream Processing With Apache Kafka Part One

Today, with the rise of IoT and Smart Devices, we are generating data at an unprecedented speed. With distributed computing, data is generated somewhere and processed somewhere else. Sensors or UI on devices capture some data (manual or automated) as an event and send it to some other unit for processing. This happens continuously.

These events may be processed at a fixed rate or in bursts, resulting in a stream of events. This process is known as an Event Stream. In most scenarios, these events are generated at a very high speed (seconds or even milliseconds). So, we need to process these event streams at the same or higher processing rate.

Today, Apache Kafka provides a distributed stream processing engine, Kafka-Streams. In this series of articles, we will first try to cover:

1.  Kafka and its use cases.

2.  Kafka API and Kafka Streams API.

3.  Setting up a single node Kafka-Cluster.

4.  Processing GPS events for real-time analysis of online and offline vehicles.

The first three parts introduce you to concepts and terminologies related to Kafka and real-time stream processing. The code for part four is available at this Github repo.

Apache Kafka

Apache Kafka is an open-source distributed stream processing platform originally developed by LinkedIn and later donated to Apache in 2011.

We can describe Kafka as a collection of files, filled with messages that are distributed across multiple machines. Most of Kafka analogies revolve around tying these various individual logs together, routing messages from producers to consumers reliably, replicating for fault tolerance, and handling failure gracefully. Its architecture inherits more from storage systems like HDFS, HBase, or Cassandra than it does from traditional messaging systems that implement JMS or AMQP. The underlying abstraction is a partitioned log, essentially a set of append-only files spread over several machines. This encourages sequential access patterns. A Kafka cluster is a distributed system that spreads data over many machines both for fault tolerance and for linear scale-out.

Kafka has quickly evolved from a messaging system to a fully-fledged streaming platform with the following attributes:

• Scalable.

• Fault-tolerant.

• Publish-subscribe messaging system.

• Higher throughput compared with most messaging systems.

Kafka's Capabilities as a Streaming Platform

1. Publish and Subscribe to Streams of Records

We already have many messaging systems. Why do we need one more? At the heart of Kafka lies the humble, immutable commit log, and from there you can subscribe to it, and publish data to any number of systems or real-time applications. Unlike messaging queues, Kafka is a highly scalable, fault-tolerant distributed system. 

Kafka has stronger ordering guarantees than a traditional messaging system. A traditional queue retains records in order on the server, and if multiple consumers consume from the queue, the server hands out records in the order they are stored. However, although the server hands out records in order, the records are delivered asynchronously to consumers, so they may arrive out of order to different consumers.
Kafka does this more efficiently. Kafka can provide both ordering guarantees and load balancing over a pool of consumer processes

2. Store Streams of Records in a Fault-Tolerant Durable Way

In Kafka, data is written to disk in a fault-tolerant way using replication of data. Kafka allows producers to wait for acknowledgment for completion, and a write is not considered complete until it is fully replicated and guaranteed to persist even if the server written to fails. Kafka will perform the same whether you have 50 KB or 50 TB of persistent data on the server. As a result, we can think of Kafka as a kind of special purpose distributed file system dedicated to high-performance, low-latency commit log storage, replication, and propagation.

3. Process Streams of Records as They Occur

A streaming platform would not be complete without the ability to manipulate data as it arrives. The Streams API within Apache Kafka is a powerful, lightweight library that allows for on-the-fly processing. 

In Kafka, a stream processor is anything that takes continual streams of data from input topics, performs some processing on this input, and produces continual streams of data to output topics.
For example, a retail application might take input streams of sales and shipments and output a stream of reorders and price adjustments computed off this data.

Simple processing can be done directly using the producer and consumer APIs. For more complex transformations, Kafka provides a fully integrated Streams API.

Concepts in Apache Kafka

• Kafka is run as a cluster on one or more servers that can span multiple data centers.

• The Kafka cluster stores streams of records in categories called topics.

• Each record consists of a key, a value, and a timestamp.

Kafka Use Cases

Kafka is used in two broad classes of applications. It can build real-time streaming data pipelines that reliably move data between systems and applications. It can also be used to build real-time streaming applications that transform or react to streams of od data. 

Some use cases for these include: 

• Messaging.

• Real-time website activity tracking.

• Metrics.

• Log aggregation.

• Stream processing.

• Event sourcing.

• Commit Log.

In this part of the series, we have introduced Apache Kafka and its basic functionality. In part two, we will introduce Kafka's API. 

Please feel free to share any feedback or ask any questions you may have!