Kafka Queuing: Kafka as a Messaging System

Objective

In this article, we will learn the concept of Apache Kafka Queuing. Basically, Queuing in Kafka is one of the models for messaging traditionally. So, let’s begin with the brief introduction to Kafka as a Messaging System, which will help us to understand Kafka Queuing. Moreover, we will see some of the applications of Kafka Queue to clear the concept better.

Kafka as a Messaging System

There are two models for messaging traditionally, such as Kafka queuing and publish-subscribe in Kafka.

Let’s discuss both Kafka models in detail:

Kafka Queue

In this Kafka messaging system, a pool of Kafka consumers may read from a server. Also, each record goes to one of them here. It has some strengths as well as some weakness. Its strength is that it permits us to divide up the processing of data over multiple consumer instances, that help us scale our processing. But its weakness is, it is not multi-subscriber, as soon as one process reads the data it’s gone.

Kafka Publish-Subscribe

Whereas in this Kafka Publish-Subscribe system, the record is broadcast to all the Kafka consumers. It permits us to broadcast data to multiple processes. However, it also has some limitations, like there is no way of scaling processing because here every message goes to every subscriber.

In Kafka, these two concepts are generalized by the Kafka consumer group. However, the consumer group in Kafka permits us to divide up processing over a collection of processes, with a Kafka queue. Here the collection of processes refers to the members of the consumer group. Moreover, Kafka permits us to broadcast messages to multiple consumer groups, with Kafka publish-subscribe.

So, the main benefit of Kafka’s model is both these properties are available in every Kafka topic —it can scale processing as well as it is multi-subscriber. Hence, that implies we do not have to select one or the other.

Kafka has stronger ordering guarantees than a traditional messaging system, too.

Since there is no parallelism in the processing in the traditional system, Kafka performs it well with the notion of parallelism. Because Kafka can offer both ordering guarantees and load balancing over a pool of consumer processes.

Need for Kafka Queuing

One of Kafka Queuing application is Microservice architecture. It essentially demands some sort of message queuing system. Let’s understand microservice architecture first. It is a concept of decoupling an interconnected monolithic application into different independent modules as well as external data sources along with APIs. Thus, for the purpose of handling microservices-external-source and inter-microservice and communications, Message queuing comes into the picture.

In addition, while we divide a big monolithic application into smaller, microservices (loosely-coupled), at that time the REST API calls increases which are amongst those microservices, and the number of connections to external data sources also increases.

However, keeping this huge system synchronous is not desirable, because it can render the entire application unresponsive. Also, it can defeat the whole purpose of dividing into microservices in the first place.

Hence, having Kafka at that time makes the whole data flow easier. Because it is distributed, highly fault-tolerant and also it has constant monitoring of broker nodes through services like Zookeeper. So, it makes it efficient to work.

Message Queuing in ML Solutions Pipeline

Apart from that, we can also use Kafka queuing for the various ML solution pipelines. However, ML solutions are built as:

A user interface (on the client side, mobile/web) — →An API server and the database — →Machine learning (blackbox).

However, ML black box is very compute-heavy and also it’s not practically easy to have those requests on a blocking synchronous mode. Moreover, all the requests can be in a queue and configure consumer API to take those requests one by one and feed them into the ML black box, in this scenario. So, while it comes to compute-intensive tasks, this pipeline can easily handle such as recognizing objects from thousands of images, that might take considerable time even without missing any requests.

Basically, microservices deployed into containers, such as by fault-tolerant distributed clusters of Kafka broker nodes it is mediated and using a Zookeeper it monitors it, which seems like a new way of enterprising software development.

So, this was all about Kafka Queuing: Kafka as a messaging system. Hope you like our explanation.

Conclusion

Hence, we have seen the complete concept of Kafka Queuing. Moreover, we discussed why Kafka as a message queuing. Along with this, we learned the two models- Kafka queue and Kafka publish-subscribe. Also, we saw why we need Kafka Queuing. Lastly, we discussed message queuing in ML solution pipeline. However, if any query occurs regarding Queuing in Kafka, feel free to ask through the comment section.