Over a million developers have joined DZone.

Keeping the Web API Layer in Kafka With a REST Proxy

DZone's Guide to

Keeping the Web API Layer in Kafka With a REST Proxy

Kafka is the quickest way I have seen to get started with real-time data streams. However, I've noticed many Apache products diverting from REST.

· Big Data Zone ·
Free Resource

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

I’m slowly learning more about Kafka and the other messaging and data streaming solutions gaining traction in the API space.

Image title

If you aren’t on the Kafka train yet, according to their website:

“Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.” I’m still learning about how Kafka works, and with no real production experience, it is something that is taking time.

As part of my conversations on the subject, I was introduced to Confluent, a platform version of Kafka, which is the quickest way I have seen to get started with real-time data streams. As part of the Confluent offering, I noticed they have a REST proxy for which you can find the API documentation here. You can find the code for the Kafka REST proxy on GitHub. According to the GitHub repo:

“The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.”

I’ve noticed that many of the other messaging and data streaming solutions out of Apache these days have diverted from using REST, which makes sense for speed and scale — but when it comes to reaching a wider audience, I can still see the need to have RESTful API, delivering a kind of multi-speed solution that allows developers to pick their speed based upon their skills, awareness, and need. I’m feeling like the platform approach of Confluent, combined with a RESTful layer, will give them an advantage over other Kafka service providers or just deploying the open-source solution out of the box.

REST isn’t always the most efficient or scalable solution, but when it comes to reaching a wide audience of developers and allowing consumers to get up and running quickly, REST is still a sensible approach. Honestly, I don’t think it is just REST; it is also about leveraging the web — not that everyone understands the web, but I think it is what a large number of developers have been exposed to, and have been building on in the last decade. I can see high-volume API solutions in the future often having a native protocol and client, but also supporting REST and gRPC to make their solutions more accessible, performant, scalable, and quickly adopted and integrated alongside existing infrastructure.

Hortonworks Community Connection (HCC) is an online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub.  Join the discussion.

big data ,rest api ,kafka ,web api

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}