Deploying a RabbitMQ Cluster at One Million Messages Per Second on Google Compute Engine

Message queuing systems are an integral part of many business models where scalable applications are at the heart of systems development. One of the most popular (and growing) message queuing systems is RabbitMQ. I know that a lot of DZone developers have backed it as a great choice for a high-volume low-latency MQ, and I've seen it grown in popularity even in the last year.

Last week, Google and Pivotal released a guide to getting started with RabbitMQ clusters on Google Compute Engine. Google has backed RabbitMQ in this regard due to what I mentioned previously--high through-put, low latency, and in-memory queuing--all in a way that is easy to set up and deploy. Now Google and Pivotal are backing up their RabbitMQ praise with a benchmark release that demonstrates a RabbitMQ cluster handling a million messages per second through Google Compute Engine--a fantastic feat that definitely seems to back up their claims about high-performance on their platform. They spoke a little bit about their motivations:

"We also wanted to demonstrate that a RabbitMQ cluster running on Google Compute Engine is a good fit for high-throughput and low-latency scenarios. To do so, we ran a series of benchmarks in various configurations. These test clusters were able to sustain a throughput of over 1 million messages published and consumed per second (a sustained combined ingress/egress of over two million messages per second)."

So the idea is that Google rounded up the engineers at Pivotal (the company behind RabbitMQ). For their Google Compute Engine platform setup, they provisioned a cluster of 32 virtual machines with 8 vCPUs and 30GB of RAM each, and deployed RabbitMQ on Debian. To generate the load they used the RabbitMQ PerfTest tool. Here's some evidence of their benchmarks:




These are definitely results that should interest the community at large. They reached a final conclusion:

Conclusion
We are excited to offer our customers an easy way to deploy RabbitMQ. We are proud that both Google and Pivotal benchmarks validate that Compute Engine is an excellent choice for running RabbitMQ in a public cloud

To deploy your own dedicated RabbitMQ cluster, go to Google Developer Console and navigate to an existing project, or create a new one, and you will have an option to deploy RabbitMQ from the project dashboard