DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

SBOMs are essential to circumventing software supply chain attacks, and they provide visibility into various software components.

Related

  • Event-Driven Microservices: How Kafka and RabbitMQ Power Scalable Systems
  • System Coexistence: Bridging Legacy and Modern Architecture
  • Event-Driven Architectures: Designing Scalable and Resilient Cloud Solutions
  • How to Integrate Event-Driven Ansible With Kafka

Trending

  • Advancing DSLs in the Age of GenAI
  • Top 5 Trends in Big Data Quality and Governance in 2025
  • Lessons Learned in Test-Driven Development
  • Exploring Data Redaction Enhancements in Oracle Database 23ai
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Kafka Queuing: Kafka as a Messaging System

Kafka Queuing: Kafka as a Messaging System

In this article, we will learn the concept of Apache Kafka Queuing and explore Kafka as a messaging system.

By 
Rinu Gour user avatar
Rinu Gour
·
Sep. 24, 18 · Opinion
Likes (4)
Comment
Save
Tweet
Share
47.2K Views

Join the DZone community and get the full member experience.

Join For Free

Objective

In this article, we will learn the concept of Apache Kafka Queuing. Basically, Queuing in Kafka is one of the models for messaging traditionally. So, let’s begin with the brief introduction to Kafka as a Messaging System, which will help us to understand Kafka Queuing. Moreover, we will see some of the applications of Kafka Queue to clear the concept better.

Kafka as a Messaging System

There are two models for messaging traditionally, such as Kafka queuing and publish-subscribe in Kafka.

Let’s discuss both Kafka models in detail:

Kafka Queue

In this Kafka messaging system, a pool of Kafka consumers may read from a server. Also, each record goes to one of them here. It has some strengths as well as some weakness. Its strength is that it permits us to divide up the processing of data over multiple consumer instances, that help us scale our processing. But its weakness is, it is not multi-subscriber, as soon as one process reads the data it’s gone.

Kafka Publish-Subscribe

Whereas in this Kafka Publish-Subscribe system, the record is broadcast to all the Kafka consumers. It permits us to broadcast data to multiple processes. However, it also has some limitations, like there is no way of scaling processing because here every message goes to every subscriber.

In Kafka, these two concepts are generalized by the Kafka consumer group. However, the consumer group in Kafka permits us to divide up processing over a collection of processes, with a Kafka queue. Here the collection of processes refers to the members of the consumer group. Moreover, Kafka permits us to broadcast messages to multiple consumer groups, with Kafka publish-subscribe.

So, the main benefit of Kafka’s model is both these properties are available in every Kafka topic —it can scale processing as well as it is multi-subscriber. Hence, that implies we do not have to select one or the other.

Kafka has stronger ordering guarantees than a traditional messaging system, too.

Since there is no parallelism in the processing in the traditional system, Kafka performs it well with the notion of parallelism. Because Kafka can offer both ordering guarantees and load balancing over a pool of consumer processes.

Need for Kafka Queuing

One of Kafka Queuing application is Microservice architecture. It essentially demands some sort of message queuing system. Let’s understand microservice architecture first. It is a concept of decoupling an interconnected monolithic application into different independent modules as well as external data sources along with APIs. Thus, for the purpose of handling microservices-external-source and inter-microservice and communications, Message queuing comes into the picture.

In addition, while we divide a big monolithic application into smaller, microservices (loosely-coupled), at that time the REST API calls increases which are amongst those microservices, and the number of connections to external data sources also increases.

However, keeping this huge system synchronous is not desirable, because it can render the entire application unresponsive. Also, it can defeat the whole purpose of dividing into microservices in the first place.

Hence, having Kafka at that time makes the whole data flow easier. Because it is distributed, highly fault-tolerant and also it has constant monitoring of broker nodes through services like Zookeeper. So, it makes it efficient to work.

Message Queuing in ML Solutions Pipeline

Apart from that, we can also use Kafka queuing for the various ML solution pipelines. However, ML solutions are built as:

A user interface (on the client side, mobile/web) — →An API server and the database — →Machine learning (blackbox).

However, ML black box is very compute-heavy and also it’s not practically easy to have those requests on a blocking synchronous mode. Moreover, all the requests can be in a queue and configure consumer API to take those requests one by one and feed them into the ML black box, in this scenario. So, while it comes to compute-intensive tasks, this pipeline can easily handle such as recognizing objects from thousands of images, that might take considerable time even without missing any requests.

Basically, microservices deployed into containers, such as by fault-tolerant distributed clusters of Kafka broker nodes it is mediated and using a Zookeeper it monitors it, which seems like a new way of enterprising software development.

So, this was all about Kafka Queuing: Kafka as a messaging system. Hope you like our explanation.

Conclusion

Hence, we have seen the complete concept of Kafka Queuing. Moreover, we discussed why Kafka as a message queuing. Along with this, we learned the two models- Kafka queue and Kafka publish-subscribe. Also, we saw why we need Kafka Queuing. Lastly, we discussed message queuing in ML solution pipeline. However, if any query occurs regarding Queuing in Kafka, feel free to ask through the comment section.

kafka

Published at DZone with permission of Rinu Gour. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Event-Driven Microservices: How Kafka and RabbitMQ Power Scalable Systems
  • System Coexistence: Bridging Legacy and Modern Architecture
  • Event-Driven Architectures: Designing Scalable and Resilient Cloud Solutions
  • How to Integrate Event-Driven Ansible With Kafka

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: