Kafka vs NATS: A Comparison for Message Processing

In a distributed architecture, communications between systems form the foundation of the entire infrastructure. The performance, scalability, and reliability of the infrastructure depend much on how events/messages/data are exchanged and persisted. 

Kafka and NATS are two popular tools for handling streaming and messaging. They have different architectures and different performance characteristics. They are suitable for specific use cases. In this article, we will compare the features of NATS with Kafka and explain the use cases I addressed at work.

1. Architecture and Complexity

NATS

NATS infrastructure has two main components:

Core NATS

Core NATS is the base messaging framework. This supports Publish-Subscribe (allows messages to be broadcasted to multiple subscribers), Request-Reply (enables synchronous communication), and Queue Groups (facilitates load balancing among multiple subscribers within a group). 

This is designed for simplicity, low latency, high performance, and scalability. It performs very well in scenarios that require low latency and high throughput. However, Core NATS alone provides only non-guaranteed delivery, meaning messages are delivered only to active subscribers. Data will be lost if the subscribers are offline. Core NATS is a good option when speed and scale take priority over durability.

JetStream

JetStream brings persistence capabilities to the top of Core NATS. This helped to provide message durability and reliability. It allows messages or events to be persisted (disk or memory) and replayed. Persisted messages can be replayed to new or recovering subscribers. With JetStream, users get additional features:

1. Stream retention: How long messages are retained. It can be based on size, time, or consumer limits.
2. Consumer durability: Enabling consumers to resume from where they left off.
3. Message acknowledgment: This ensures the reliability of the delivery.

JetStream adds a layer of complexity to Core NATS. However, this brings in the important feature of supporting the use cases of guaranteed delivery, persistence, and replayability.

Kafka

Kafka is a distributed messaging system built on a log-based broker architecture. Data in Kafka is arranged into Topics and can have multiple partitions. Consumers are connected to these partitions. This architecture allows Kafka to parallelize message consumption for a single topic. Data is appended to a topic/partitions sequentially. Kafka guarantees to order in a partition. In a Kafka cluster, there can be many brokers, each managing a list of topics and partitions. To achieve high availability and prevent data loss, Kafka relies on a replication factor, where partitions are replicated across multiple Kafka brokers. As you can see, there are multiple components that must be managed to achieve high throughput, fault tolerance, data retention, and horizontal scalability. This increases the architectural complexity of Kafka.

2. High Availability and Performance 

NATS 

All the nodes in a cluster are interconnected in a mesh, and the client can connect to any node. This configuration avoids a single point of failure. If one node fails, the client gets automatically connected to the other nodes without any manual intervention. This is called self-healing in NATS. A JetStream-enabled node distributes the streams among all the nodes. Streams are highly managed and load-balanced across the JetStream-enabled nodes within a mesh cluster. 

JetStream also supports data mirroring across multiple clusters or nodes. In JetStream, leaders are elected per stream. Replication of each stream can be configured. All these things ensure durability and availability in NATS.

Kafka

Kafka's high availability is based on the replication. Every topic can have one or more partitions. Each partition is replicated across Kafka Brokers. This ensures the data redundancy and availability. Kafka follows a Leader-Follower replication mechanism. A leader takes care of read and write. And the follower works on replicating the data. 

Kafka maintains something called ISR (In Sync Replicas) for each partition. If the leader fails, one of the ISRs becomes the leader. For cluster metadata management and leader election, Kafka relies on Zookeeper (KRaft in the newer versions).

3. Message Patterns

NATS

NATS uses subject-based messaging. This allows services and streams to use Pub-Sub, Request-Reply, and Queue Subscriber patterns. Subjects in NATS can be constructed with hierarchy and wild cards. A single NATS stream can store multiple subjects and Client applications can use server-side filtering to receive only the interested subjects. Connection in NATS is bi-directional and allows clients to publish and subscribe at the same time. NATS also supports Queueing very similar to RabbitMQ.

Kafka

Streams in Kafka support Pub-sub and topic-based messaging. Load balancing can be achieved through Consumer groups and partitioning the topics.

4. Delivery Guarantees

NATS

NATS supports various delivery Guarantees. NATS alone can support an at-most-once delivery guarantee. NATS servers with JetStream enabled can support an additional two types of guarantees. They are "at least once" and "exactly once" guarantees. NATS can send 'acks' to individual messages. Please refer to the NATS official documentation for the various 'acks' it supports. Based on the 'acks' type, NATS can re-deliver messages.

Kafka

Kafka supports at least once and exactly once guarantees. Message ordering is guaranteed at the Partition level. Global ordering is not possible in Kafka.

5. Message Retention and Persistence

NATS

NATS supports memory and file-based persistence. There are several options to replay the message. The replay of messages can be by time, count, or sequence number.

Kafka

KAFKA supports only file-based persistence. Messages can be replayed from the latest, earliest, or a specific offset. Log Compaction is supported in KAFKA.

6. Languages and Platform

NATS

Forty-eight known client types. Any architectures that support GOLANG can support NATS servers.

Kafka

Eighteen known client types. Kafka servers can run on platforms supporting JVM.

Use Cases

Use Case 1

Requirements

We have a data platform with a streaming pipeline. The platform uses Apache Flink engine for real time streaming and Apache Beam for writing the analytics pipeline. Below are the key requirements:

1. High throughput and low latency message processing
2. Support for checkpoint and back pressure handling
3. Handle messages in MBs
4. Message durability and persistence

Comparison

Kafka advantages:

• High throughput
• Data retention with configurable retention policies and replicate data for fault tolerance
• Support for at least one message delivery guarantee
• Reading messages from earliest/latest/specific offsets
• Server-side ‘acks’ for reliable delivery
• Handle massive data streams and large message size
• Support for Compaction Topic

Kafka drawbacks:

• High resource usage. Our cluster was on-premises and resource-constrained
• Kafka is only near real-time

NATS advantages:

• High performance with minimal resource usage. Ours is an on-premises cluster with resource constraints
• Support for at least once. We were looking for an at-least-once guarantee
• Low-latency message processing

NATS drawbacks:

• No connectors for Flink/Beam hence, integration was difficult
• Performance reduction with message size

Final Decision

After careful analysis, Kafka was chosen. We had to make a tradeoff between resource usage and the other benefits that Kafka was offering, especially the good integration available with Apache Beam and Flink. Another advantage of Kafka was its handling of large message sizes and high-throughput message processing.

Use Case 2

Requirements

Handle the events generated in an on-premises cluster, Ex: Audit Logs. Events should be processed with low latency. And support microservices communication. Durability and persistence were not a requirement. The message size was small. No need to do any analytics on the events. We were in a constrained environment. Resource usage and memory footprint should be minimal.

Decision

Why NATS was chosen:

• Efficient resource usage 
• Low latency event handling. 
• Since it is a Go application, the memory footprint is very low
• Ability to handle small message sizes 
• Request-Reply support that can help Microservices communication
• When JetStream is not configured, messages are not stored

Why Kafka was not chosen:

• By default, messages are stored on disk
• Resource usage is high compared to NATS
• Since it needs JVM, the memory footprint is very high

Summary

The choice between Kafka and NATS depends on your specific requirements across three key areas: Architecture and Complexity, Performance and Scalability, and Message Delivery Guarantees. Kafka is ideal for systems requiring robust event streaming, durable storage, and advanced processing capabilities, but it comes with higher complexity. NATS, on the other hand, is lightweight, easy to manage, and excels in low-latency, high-throughput scenarios with simpler messaging needs.

When designing a distributed messaging system, carefully evaluate these areas to align your choice with your application's goals and constraints. Both Kafka and NATS are powerful tools, and the right choice will depend on your use case.

Key areas to be considered before choosing between Kafka and NATS:

1. Architecture and complexity
2. High availability and performance
3. Message delivery guarantees

Kafka is ideal for distributed systems requiring event streaming, durable storage and advancing processing capabilities. However, Kafka comes with high resource usage and a memory footprint. And management complexity is very high compared to NATS.

On the other hand, NATS is lightweight and easy to manage. Low latency message processing is NATS signature capability. 

Ultimately, both Kafka and NATS are powerful event-handling tools. The choice depends on specific use cases.