Over a million developers have joined DZone.

Kafka's Cluster Replication Design

Visually compose APIs with easy-to-use tooling. Learn how IBM API Connect provides near-universal access to data and services both on-premises and in the cloud, brought to you in partnership with IBM.

Kafka, the distributed pub-sub messaging system designed by LinkedIn and used also at Twitter and Square, is some powerful tech.  This tutorial on intra-cluster replication, a relatively new feature, is a good tutorial on the subject written by Jun Rao, a staff engineer at LinkedIn.

With replication, Kafka clients will get the following benefits:
1. A producer can continue to publish messages during failure and it can choose between latency and durability, depending on the application.
2. A consumer continues to receive the correct messages in real time, even when there is failure.

--Jun Rao, Intra-cluster Replication in Apache Kafka

For more info on the design and the implementation, check out the Kafka Replication Wiki.

Visually compose APIs with easy-to-use tooling. Learn how IBM API Connect provides near-universal access to data and services both on-premises and in the cloud, brought to you in partnership with IBM.

Topics:

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}