DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Playwright JavaScript Tutorial: A Complete Guide
  • Managing Data Residency, the Demo
  • 13 Impressive Ways To Improve the Developer’s Experience by Using AI
  • What Is mTLS? How To Implement It With Istio

Trending

  • Playwright JavaScript Tutorial: A Complete Guide
  • Managing Data Residency, the Demo
  • 13 Impressive Ways To Improve the Developer’s Experience by Using AI
  • What Is mTLS? How To Implement It With Istio
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Kafka Queuing: Kafka as a Messaging System

Kafka Queuing: Kafka as a Messaging System

In this article, we will learn the concept of Apache Kafka Queuing and explore Kafka as a messaging system.

Rinu Gour user avatar by
Rinu Gour
·
Sep. 24, 18 · Opinion
Like (4)
Save
Tweet
Share
46.26K Views

Join the DZone community and get the full member experience.

Join For Free

Objective

In this article, we will learn the concept of Apache Kafka Queuing. Basically, Queuing in Kafka is one of the models for messaging traditionally. So, let’s begin with the brief introduction to Kafka as a Messaging System, which will help us to understand Kafka Queuing. Moreover, we will see some of the applications of Kafka Queue to clear the concept better.

Kafka as a Messaging System

There are two models for messaging traditionally, such as Kafka queuing and publish-subscribe in Kafka.

Let’s discuss both Kafka models in detail:

Kafka Queue

In this Kafka messaging system, a pool of Kafka consumers may read from a server. Also, each record goes to one of them here. It has some strengths as well as some weakness. Its strength is that it permits us to divide up the processing of data over multiple consumer instances, that help us scale our processing. But its weakness is, it is not multi-subscriber, as soon as one process reads the data it’s gone.

Kafka Publish-Subscribe

Whereas in this Kafka Publish-Subscribe system, the record is broadcast to all the Kafka consumers. It permits us to broadcast data to multiple processes. However, it also has some limitations, like there is no way of scaling processing because here every message goes to every subscriber.

In Kafka, these two concepts are generalized by the Kafka consumer group. However, the consumer group in Kafka permits us to divide up processing over a collection of processes, with a Kafka queue. Here the collection of processes refers to the members of the consumer group. Moreover, Kafka permits us to broadcast messages to multiple consumer groups, with Kafka publish-subscribe.

So, the main benefit of Kafka’s model is both these properties are available in every Kafka topic —it can scale processing as well as it is multi-subscriber. Hence, that implies we do not have to select one or the other.

Kafka has stronger ordering guarantees than a traditional messaging system, too.

Since there is no parallelism in the processing in the traditional system, Kafka performs it well with the notion of parallelism. Because Kafka can offer both ordering guarantees and load balancing over a pool of consumer processes.

Need for Kafka Queuing

One of Kafka Queuing application is Microservice architecture. It essentially demands some sort of message queuing system. Let’s understand microservice architecture first. It is a concept of decoupling an interconnected monolithic application into different independent modules as well as external data sources along with APIs. Thus, for the purpose of handling microservices-external-source and inter-microservice and communications, Message queuing comes into the picture.

In addition, while we divide a big monolithic application into smaller, microservices (loosely-coupled), at that time the REST API calls increases which are amongst those microservices, and the number of connections to external data sources also increases.

However, keeping this huge system synchronous is not desirable, because it can render the entire application unresponsive. Also, it can defeat the whole purpose of dividing into microservices in the first place.

Hence, having Kafka at that time makes the whole data flow easier. Because it is distributed, highly fault-tolerant and also it has constant monitoring of broker nodes through services like Zookeeper. So, it makes it efficient to work.

Message Queuing in ML Solutions Pipeline

Apart from that, we can also use Kafka queuing for the various ML solution pipelines. However, ML solutions are built as:

A user interface (on the client side, mobile/web) — →An API server and the database — →Machine learning (blackbox).

However, ML black box is very compute-heavy and also it’s not practically easy to have those requests on a blocking synchronous mode. Moreover, all the requests can be in a queue and configure consumer API to take those requests one by one and feed them into the ML black box, in this scenario. So, while it comes to compute-intensive tasks, this pipeline can easily handle such as recognizing objects from thousands of images, that might take considerable time even without missing any requests.

Basically, microservices deployed into containers, such as by fault-tolerant distributed clusters of Kafka broker nodes it is mediated and using a Zookeeper it monitors it, which seems like a new way of enterprising software development.

So, this was all about Kafka Queuing: Kafka as a messaging system. Hope you like our explanation.

Conclusion

Hence, we have seen the complete concept of Kafka Queuing. Moreover, we discussed why Kafka as a message queuing. Along with this, we learned the two models- Kafka queue and Kafka publish-subscribe. Also, we saw why we need Kafka Queuing. Lastly, we discussed message queuing in ML solution pipeline. However, if any query occurs regarding Queuing in Kafka, feel free to ask through the comment section.

kafka

Published at DZone with permission of Rinu Gour. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Playwright JavaScript Tutorial: A Complete Guide
  • Managing Data Residency, the Demo
  • 13 Impressive Ways To Improve the Developer’s Experience by Using AI
  • What Is mTLS? How To Implement It With Istio

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: