Over a million developers have joined DZone.

Real-Time Data Processing With Anypoint Connector for Kafka

This article shows how to use Kafka and Anypoint Connector in a demo app that allows you to publish a message to a topic and to ingest a message from a topic.

· Integration Zone

Learn how API management supports better integration in Achieving Enterprise Agility with Microservices and API Management, brought to you in partnership with 3scale

Apache started at LinkedIn in 2010 as a simple messaging system to process massive data, and now it handles 1.4 trillion messages per day at LinkedIn. According to Kafka Summit 2016, it has gained lots of adoption (2.2 million downloads in the last two years) in thousands of companies including Airbnb, Cisco, Goldman Sachs, Microsoft, Netflix, Salesforce, Twitter, and Uber. MuleSoft has also been using Kafka to power its analytics engine.

Companies use Kafka in various use cases: application monitoring, data warehouse, asynchronous applications, system monitoring, recommendation/ decision engines, customer preferences/ personalizations, and security/ fraud detection. Moreover, MuleSoft customers also use Kafka in various ways as well. One of our customers uses Kafka as an event bus to log messages. Another customer processes real-time data from field equipment for faster decision making and automation with Kafka, and others aggregate data from different sources through Kafka. To help our customers quickly and easily ingest data from Kafka and/or publish data to Kafka, MuleSoft is thrilled to release the Anypoint Connector for Kafka today.

Here is a quick example of how to use the Kafka Connector based on Kafka 0.9. This demo app allows you to publish a message to a topic and to ingest a message from a topic. The app consists of three flows; the first flow shows you a web page where you can publish a message to Kafka, the second flow is for Kafka consumer, and the third flow is for Kafka producer.

Let’s configure this Kafka first. If you go to Global Elements, you will find “Apache Kafka.” After selecting “Apache Kafka,” please click on “Edit.”

Screen Shot 2016-06-01 at 4.23.25 PM

Configuration

In the “Apache Kafka: Configuration”, you can specify the configuration of your Kafka server. You could directly add your Bootstrap Server information in the configuration, but I recommend you use the properties file to add your configuration information.

properties

mule-app.properties include the following keys-value pairs:

  • config.bootstrapServers={your Kafka Server address}
  • config.consumerPropertiesFile=consumer.properties
  • config.producerPropertiesFile=producer.properties
  • # Consumer specific information
  • consumer.topic=one-replica
  • consumer.topic.partitions=1

Since Kafka provides various settings for producer and consumer, you can add your own settings in consumer.properties for consumer and producer.properties for producer under src/main/resource.

After you complete the configuration for your Kafka environment, run the app. When you open up a browser and hit localhost:8081, your browser will show the following page.

Producer

Since this demo app is listening to the “one-replica” topic, when you publish a message to the “one-replica” topic, you can see your message being logged in the Studio console by the consumer-flow.

logged

For new users, try the above example to get started, and for others, please share with us how you are planning to use the Kafka Connector! Also, feel free to check out our Anypoint Connector to see what other out-of-the-box connectors we have to offer.

Unleash the power of your APIs with future-proof API management - Create your account and start your free trial today, brought to you in partnership with 3scale.

Topics:
app ,connector ,server ,environment ,kafka ,apache kafka ,summit ,apache ,demo

Published at DZone with permission of Nathan Nam, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}