Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Don't Forget About the Kestrel Messaging Queue

DZone's Guide to

Don't Forget About the Kestrel Messaging Queue

Free Resource

The Integration Zone is brought to you in partnership with Cloud Elements. What's below the surface of an API integration? Download The Definitive Guide to API Integrations to start building an API strategy.

In the messaging queue lineup, the incumbents tend to be ActiveMQ or RabbitMQ and the upstarts who are making headway into the spotlight are usually ZeroMQ or Kafka.  I wanted to remind developers in need of a good messaging architecture to also keep an eye on a queue called " Kestrel" (formerly Scarling).

Eric Lindvall explained  some of he reasons why Kestrel was a perfect fit for Papertrail:

1. Robustness

Using Kestrel (or any system that doesn't require that all messages are held in RAM) protects you from a runaway producer sending too many messages and causing your system to run out of RAM.

2. Flexibility (both infrastructure and code)

The way Resque includes the class to execute along with the "work to be done" makes it very easy to get off the ground. For many cases where what you want done is some piece of code to just get run outside of the web request thread, it solves the problem perfectly.

What you run into when you get into the realm of processing thousands of these pieces of work per second, is that there are often optimizations that you can do to how things are processed.

The changes may include:
 * doing work in batches (it may allow you to cache certain lookups or do things that take round-trips all at once)
 * moving your code from being pushed work to pulling work

Having that separation between "the work" and "the code" can also make it simpler to do things like slowly migrate work from an older way of processing the work to a newer way (to allow you to test how well a newer solution works without needing to do a full deploy).


In that article they also compare Kestrel with Resque, which is based on Redis.  Kestrel would also be easy to modify for Java devs because it's built using Scala.

There are already two great plugins: Kestrel Overall and Kestrel Queue Monitoring.

Your API is not enough. Learn why (and how) leading SaaS providers are turning their products into platforms with API integration in the ebook, Build Platforms, Not Products from Cloud Elements.

Topics:

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}