DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • How To Install CMAK, Apache Kafka, Java 18, and Java 19 [Video Tutorials]
  • Event Mesh: Point-to-Point EDA
  • Kafka Fail-Over Using Quarkus Reactive Messaging
  • Next-Gen Data Pipes With Spark, Kafka and k8s

Trending

  • Going All In On the JVM 25 Virtual Threads
  • 7 Swift Features That Will Let You Delete Code Tomorrow
  • OwlVigil Open-Sources Its Go SDK for Enterprise AI
  • Building a Grounded Search Agent With Foundry, Azure AI Search, and Cosmos DB, Containerized With Docker and Observed With Grafana

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. Big Data
  4. Let's Think Kafka Cluster Without Zookeeper With KIP-500

Let's Think Kafka Cluster Without Zookeeper With KIP-500

Right now, Apache Kafka utilizes Apache ZooKeeper to store its metadata. Managing a ZooKeeper cluster creates an additional burden on the infrastructure and the admins.

By 
Siva Prasad Rao Janapati
·
Mar. 19, 21 · Tutorial
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
7.1K Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Right now, Apache Kafka utilizes Apache ZooKeeper to store its metadata. Information such as the partitions, configuration of topics, access control lists, etc. metadata stored in a ZooKeeper cluster. Managing a ZooKeeper cluster creates an additional burden on the infrastructure and the admins. With KIP-500, we are going to see a Kafka cluster without the ZooKeeper cluster where the metadata management will be done with Kafka itself.

Before KIP-500, our Kafka setup looks like depicted below. Here we have a 3 node ZooKeeper cluster and a 4 node Kafka cluster. This setup is a minimum for sustaining 1 Kafka broker failure. The orange Kafka node is a controller node.

3 Node Zookeeper Cluster and 4 Node Kafka Cluster

Let's see what issues we have with the above setup with the involvement of ZooKeeper:

  • Making the ZooKeeper cluster highly available is an issue as without the ZooKeeper cluster the Kafka cluster is DEAD.
  • Availability of the Kafka cluster if the controller dies. Electing another Kafka broker as a controller requires pulling the metadata from the ZooKeeper which leads to the Kafka cluster unavailability. If the number of topics and the partitions is more per topic, the failover Kafka controller time increases.
  • Kafka supports intra-cluster replication to support higher availability and durability. There should be multiple replicas of a partition, each stored in a different broker. One of the replicas is designated as a leader and the rest of the replicas are followers. If a broker fails, partitions on that broker with a leader temporarily become inaccessible. To continue serving the client requests, Kafka will automatically transfer the leader of those inaccessible partitions to some other replicas. This process is done by the Kafka broker who is acting as a controller. The controller broker should get metadata from the ZooKeeper for each of the affected partitions. The communication between the controller broker and the ZooKeeper happens in a serial manner which leads to unavailability of the partition if the leader broker dies.
  • When we delete or create a topic, the Kafka cluster needs to talk to ZooKeeper to get the updated list of topics. To see the impact of topic deletion or creation with the Kafka cluster will take time.
  • The major issue we see is the SCALABILITY issue.

Let's see how the Kafka cluster looks like post-KIP-500. Below is the Kafka cluster setup:

Post-KIP-500If you look at the post-KIP-500, the metadata is stored in the Kafka cluster itself. Consider that cluster as a controller cluster. The controller marked in orange color is an active controller and the other nodes are standby controllers. All the brokers in the cluster will be in sync. So, during the failure of the active controller node, electing the standby node as a controller is very quick as it doesn't require syncing the metadata. The brokers in the Kafka cluster will periodically pull the metadata from the controller. This design means that when a new controller is elected, we never need to go through a lengthy metadata loading process.

Post-KIP-500 will speed up the topic creation and deletion. Currently, the topic creation or deletion requires getting the full list of topics in the cluster from the ZooKeeper metadata. Post KIP-500, just the entry needs to add to the metadata partition. This speeds up the topic creation and deletion. Post-KIP-500, the metadata scalability increases which eventually improves the SCALABILITY of Kafka.

In the future, I want to see the elimination of the second Kafka cluster for controllers and eventually, we should be able to manage the metadata within the actual Kafka cluster. That reduces the burden on the infrastructure and the administrator's job to the next level. We will meet on another topic. Until then, Happy Messaging!!

kafka cluster

Published at DZone with permission of Siva Prasad Rao Janapati. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • How To Install CMAK, Apache Kafka, Java 18, and Java 19 [Video Tutorials]
  • Event Mesh: Point-to-Point EDA
  • Kafka Fail-Over Using Quarkus Reactive Messaging
  • Next-Gen Data Pipes With Spark, Kafka and k8s

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook