Over a million developers have joined DZone.

Keeping the Web API Layer in Kafka With a REST Proxy

DZone's Guide to

Keeping the Web API Layer in Kafka With a REST Proxy

Kafka is the quickest way I have seen to get started with real-time data streams. However, I've noticed many Apache products diverting from REST.

· Big Data Zone
Free Resource

Learn best practices according to DataOps. Download the free O'Reilly eBook on building a modern Big Data platform.

I’m slowly learning more about Kafka and the other messaging and data streaming solutions gaining traction in the API space.

Image title

If you aren’t on the Kafka train yet, according to their website:

“Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.” I’m still learning about how Kafka works, and with no real production experience, it is something that is taking time.

As part of my conversations on the subject, I was introduced to Confluent, a platform version of Kafka, which is the quickest way I have seen to get started with real-time data streams. As part of the Confluent offering, I noticed they have a REST proxy for which you can find the API documentation here. You can find the code for the Kafka REST proxy on GitHub. According to the GitHub repo:

“The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.”

I’ve noticed that many of the other messaging and data streaming solutions out of Apache these days have diverted from using REST, which makes sense for speed and scale — but when it comes to reaching a wider audience, I can still see the need to have RESTful API, delivering a kind of multi-speed solution that allows developers to pick their speed based upon their skills, awareness, and need. I’m feeling like the platform approach of Confluent, combined with a RESTful layer, will give them an advantage over other Kafka service providers or just deploying the open-source solution out of the box.

REST isn’t always the most efficient or scalable solution, but when it comes to reaching a wide audience of developers and allowing consumers to get up and running quickly, REST is still a sensible approach. Honestly, I don’t think it is just REST; it is also about leveraging the web — not that everyone understands the web, but I think it is what a large number of developers have been exposed to, and have been building on in the last decade. I can see high-volume API solutions in the future often having a native protocol and client, but also supporting REST and gRPC to make their solutions more accessible, performant, scalable, and quickly adopted and integrated alongside existing infrastructure.

Find the perfect platform for a scalable self-service model to manage Big Data workloads in the Cloud. Download the free O'Reilly eBook to learn more.

big data ,rest api ,kafka ,web api

Published at DZone with permission of Kin Lane, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}