Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Building an Apache Kafka Messaging Consumer on Bluemix

DZone's Guide to

Building an Apache Kafka Messaging Consumer on Bluemix

Learn how to use Apache Kafka on IBM Bluemix to build a cloud-based Node.js chat application.

· Cloud Zone
Free Resource

What if you could learn how to use MongoDB directly from the experts, on your schedule, for free? We've put together the ultimate guide for learning MongoDB. Sign up and you'll receive instructions for how to get started!

The Message Hub service on Bluemix is based on Apache Kafka, which is a fast, scalable, and durable real-time messaging engine. For developers there are Java, Node and REST APIs to leverage Kafka. Yesterday I blogged about how to write a simple Kafka producer in Node.js. This article covers the counterpart – a Kafka consumer.

To understand how consumers work in Kafka I recommend reading the Kafka documentation. Essentially you define consumer groups with potentially multiple consumer instances. Only one instance per group receives one particular message. In order to scale you can simply increase the number of consumer instances. In Kafka each container instance has it’s own partition from which it receives messages for a certain topic (in the right order). However since the Message Hub only supports one partition in the current beta status, you cannot use this functionality yet.

The consumer code below is again a slightly modified version of the sample Node.js chat application from my colleague Niall Weedon. I separated the producer from the consumer code and removed the chat code.

The consumer instance is created in line 87 by passing in the consumer group name, the consumer instance id (maps to Bluemix Cloud Foundry application instance_id) and an offset to define which messages to read. The code in lines 66ff shows how the consumer actually polls for updates on a certain interval and reads the messages (as batch) in line 69. To make sure that other consumer instances receive messages when a particular consumer instance goes down, you need to tell this Kafka in line 125.

You can run the consumer either locally or on Bluemix. To run it on Bluemix, create a Bluemix Node.js application, add the Message Hub service and execute these commands from the project’s root directory.

cf login
cf push <mykafkaconsumer>

To run the consumer locally, execute these commands.

npm install
node app.js <message_hub_rest_endpoint> <message_hub_api_key>

In order to test the consumer you need to also run the producer as explained in the previous article. Both need to use the same Message Hub instance. Here is a screenshot of the producer and consumer running locally.

Image title


What if you could learn how to use MongoDB directly from the experts, on your schedule, for free? We've put together the ultimate guide for learning MongoDBSign up and you'll receive instructions for how to get started!

Topics:
kafka ,bluemix ,node js ,npm ,java ,rest

Published at DZone with permission of Niklas Heidloff, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}