Apache Kafka In Action
Produce and consumer message from a Topic in Kafka.
Join the DZone community and get the full member experience.Join For Free
Challenges and Limitations in Messaging
Messaging is a fairly simple paradigm for the transfer of data between applications and data stores. However, there are multiple challenges that are associated with it:
- Limited scalability due to a broker becoming a bottleneck.
- Strained message brokers due to bigger message size.
- Consumers being in a position to consume the messages at a reasonable rate.
- Consumers exhibiting non-fault tolerance by making sure that messages consumed are not gone forever.
Messaging Limitations Due to:
Messaging applications are hosted on a single host or a node. As a result, there is a possibility of the broker becoming the bottleneck due to a single host or local storage.
Also, if the Subscribers consumes data slowly or if there is no consumption of data, there is a possibility of the broker or publisher going down, which may result in complete denial of service.
There is a possibility of a bug in the Subscriber logic. Because of this, data might be processed incorrectly, poisoned, or adulterated. If the bug in the Subscriber is fixed, there must be a way to fetch the old data for processing. If the Subscriber stored the data, it would be helpful.
We also might have to reprocess all the messages once the bug is fixed.
Different apps that act as a publisher-subscriber have custom logic to write to a broker. Each of them has different error handling. Maintaining data consistency, in this case, could be difficult.
How Does Kafka Solve These Issues?
- Provides high-throughput for large volumes of data that are in Terabytes or beyond.
- Is horizontally scalable, so it is able to scale out by adding machines to seamlessly share the load.
- Provides reliability, so none of the data will be lost in case of failure.
- Has Publishers and Consumers loosely coupled, so they are only involved in data exchange.
- It makes use of Pub-Sub messaging semantic, so independent applications send data on the topic and interested Subscribers can consume data on the topic.
Let's follow this video tutorial to learn how to:
- Set up the Apache Kafka cluster.
- Set up the Zookeeper and Broker.
- Produce some messages on the Topic.
- Consume the messages from the same Topic.
Steps of the video tutorial above are listed below:
Step 1 - cd /../kafka_2.12-2.3.0 Step 2 - Start Zookepeer bin/zookeeper-server-start.sh config/zookeeper.properties Step 2 - Kafka Broker bin/kafka-server-start.sh config/server.properties Step 3 - Create Topic bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic Demo Step 4 - List the created Topics bin/kafka-topics.sh --list --bootstrap-server localhost:9092 Step 5 - Start the Producer bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Demo Step 6 - Start the Consumer bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Demo --from-beginning
Opinions expressed by DZone contributors are their own.