DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • How To Install CMAK, Apache Kafka, Java 18, and Java 19 [Video Tutorials]
  • Event Mesh: Point-to-Point EDA
  • Kafka Fail-Over Using Quarkus Reactive Messaging
  • Next-Gen Data Pipes With Spark, Kafka and k8s

Trending

  • Bad Software Examples: How Much Can Poor Code Hurt You?
  • TDD With FastAPI Is Easy
  • Choosing the Appropriate AWS Load Balancer: ALB vs. NLB
  • AWS Amplify: A Comprehensive Guide
  1. DZone
  2. Data Engineering
  3. Big Data
  4. How to Setup a Kafka Cluster

How to Setup a Kafka Cluster

In this article, we will discuss how to set up a Kafka cluster with 3 nodes on a single machine. After we're done, you'll be able to make as many nodes as you need!

Gaurav Garg user avatar by
Gaurav Garg
·
May. 28, 18 · Tutorial
Like (10)
Save
Tweet
Share
102.57K Views

Join the DZone community and get the full member experience.

Join For Free

In my previous article, I discussed how to setup Kafka with a single node. But, for better reliability and high availability of the Kafka service, we should set it up in cluster mode. In this article, we will discuss how to set up a Kafka cluster with 3 nodes on a single machine. At the end of this article, you will be able to set up a Kafka cluster with as many nodes as you want on different machines.

Download Kafka from Apache's site. Extract the zip file. Make two copies of the extracted folder. Add the suffix _1, _2, _3 to these folders name. So if your extracted folder name was kafka_2.11-1.1.0, now you will have the folders kafka_2.11-1.1.0_1, kafka_2.11-1.1.0_2, kafka_2.11-1.1.0_3. Go to the kafka_2.11-1.1.0_1 folder.

  1. Create a folder named logs. This is where Kafka logs will be stored.

  2. Go to the config directory and open the server.properties file. This file contains Kafka broker configurations.

  3. Set broker.id to 1. This is the id of the broker in a cluster. It must be unique for each broker.

  4. Uncomment the listener's configuration and set it to PLAINTEXT://localhost:9091. This configuration means the broker will be listening on port 9091 for connection requests.

  5. Set the log.dirs configuration with the logs folder path that we created in step 1.

  6. In the zookeeper.connect configuration, set the Zookeeper address. If Zookeeper is running in a cluster, then give the address as a comma-separated list, i.e.:localhost:2181,localhost:2182. 
  7. Above are the basic configurations that need to be set up for the development environment but if you want to change some advanced configurations, like retention policy, etc., then I have discussed that in another DZone article, here.

Our first broker configuration is ready. For the other two folders or brokers, follow the same steps with the following changes.

  • In step 3, change broker.id to 2 and 3, respectively.

  • In step 4, change the ports used to 9092 and 9093, respectively (you can provide any port number that is available).

Now our configuration is ready for all brokers. Go to the home directory of each Kafka folder and run the command ./bin/kafka-server-start.sh config/server.properties. 

Execute the command (all as one line):

 ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor

3 --partitions 50 --topic demo 

This command will create a topic demo with 50 partitions with a replication factor of three for each partition. A replication factor of three means that for a partition there will be one leader and two followers. When a message or record is sent in the leader it is copied in followers.

Execute this command:

./bin/kafka-topics.sh --describe --topic Hello-Kafka --zookeeper localhost:2181 This allows you to see which broker is the leader or follower for which partition. The output looks something like what I've given below.

Topic:demoPartitionCount:50ReplicationFactor:3Configs:
Topic: demoPartition: 0Leader: 2Replicas: 2,3,1Isr: 2,3,1
Topic: demoPartition: 1Leader: 3Replicas: 3,1,2Isr: 3,1,2
Topic: demoPartition: 2Leader: 1Replicas: 1,2,3Isr: 1,2,3
Topic: demoPartition: 3Leader: 2Replicas: 2,1,3Isr: 2,1,3
Topic: demoPartition: 4Leader: 3Replicas: 3,2,1Isr: 3,2,1
Topic: demoPartition: 5Leader: 1Replicas: 1,3,2Isr: 1,3,2
Topic: demoPartition: 6Leader: 2Replicas: 2,3,1Isr: 2,3,1
..................................................................
    ..................................................................
    ..................................................................

 For Partition 0, Broker 2 is the leader and for partition 1, Broker 3 is the leader. ISR means in sync replicas.

Now that we have set up Kafka cluster of three brokers you can setup a cluster with as many brokers as you want by making a few changes. 

kafka cluster

Opinions expressed by DZone contributors are their own.

Related

  • How To Install CMAK, Apache Kafka, Java 18, and Java 19 [Video Tutorials]
  • Event Mesh: Point-to-Point EDA
  • Kafka Fail-Over Using Quarkus Reactive Messaging
  • Next-Gen Data Pipes With Spark, Kafka and k8s

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: