DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Apache Kafka In Action

Apache Kafka In Action

Produce and consumer message from a Topic in Kafka.

Shreyas Chaudhari user avatar by
Shreyas Chaudhari
·
Aug. 20, 19 · Tutorial
Like (9)
Save
Tweet
Share
16.17K Views

Join the DZone community and get the full member experience.

Join For Free

Challenges and Limitations in Messaging

Messaging is a fairly simple paradigm for the transfer of data between applications and data stores. However, there are multiple challenges that are associated with it:

  1. Limited scalability due to a broker becoming a bottleneck.
  2. Strained message brokers due to bigger message size.
  3. Consumers being in a position to consume the messages at a reasonable rate.
  4. Consumers exhibiting non-fault tolerance by making sure that messages consumed are not gone forever.

Messaging Limitations Due to:

High Volume

Messaging applications are hosted on a single host or a node. As a result, there is a possibility of the broker becoming the bottleneck due to a single host or local storage.

Also, if the Subscribers consumes data slowly or if there is no consumption of data, there is a possibility of the broker or publisher going down, which may result in complete denial of service.

Application Faults

There is a possibility of a bug in the Subscriber logic. Because of this, data might be processed incorrectly, poisoned, or adulterated. If the bug in the Subscriber is fixed, there must be a way to fetch the old data for processing. If the Subscriber stored the data, it would be helpful.

We also might have to reprocess all the messages once the bug is fixed.

Middleware logic

Different apps that act as a publisher-subscriber have custom logic to write to a broker. Each of them has different error handling. Maintaining data consistency, in this case, could be difficult.

How Does Kafka Solve These Issues?

  1. Provides high-throughput for large volumes of data that are in Terabytes or beyond.
  2. Is horizontally scalable, so it is able to scale out by adding machines to seamlessly share the load.
  3. Provides reliability, so none of the data will be lost in case of failure.
  4. Has Publishers and Consumers loosely coupled, so they are only involved in data exchange.
  5. It makes use of Pub-Sub messaging semantic, so independent applications send data on the topic and interested Subscribers can consume data on the topic.

Let's follow this video tutorial to learn how to: 

  1. Set up the Apache Kafka cluster.
  2. Set up the Zookeeper and Broker.
  3. Produce some messages on the Topic.
  4. Consume the messages from the same Topic.

Steps of the video tutorial above are listed below:

Step 1 - cd /../kafka_2.12-2.3.0

Step 2 - Start Zookepeer
bin/zookeeper-server-start.sh config/zookeeper.properties

Step 2 - Kafka Broker
bin/kafka-server-start.sh config/server.properties

Step 3 - Create Topic
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic Demo 

Step 4 - List the created Topics
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 5 - Start the Producer
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Demo

Step 6 - Start the Consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Demo --from-beginning


References

https://kafka.apache.org/

kafka Data (computing)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Project Hygiene
  • Fraud Detection With Apache Kafka, KSQL, and Apache Flink
  • A Beginner's Guide to Back-End Development
  • Exploring the Benefits of Cloud Computing: From IaaS, PaaS, SaaS to Google Cloud, AWS, and Microsoft

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: