Over a million developers have joined DZone.

MongoDB as a Message Queue

DZone 's Guide to

MongoDB as a Message Queue

· Java Zone ·
Free Resource

This is a live blog from MongoSV. Here’s a link to the entire series of posts.
About.me uses MongoDB for different pieces of infrastructure, but this talk is just about queuing.

Originally ran a 3-node RabbitMQ cluster, without disk persistence. Were having trouble diagnosing issues at scale. Looked at some other AMQP options, but decided on MongoDB.

Benefits: async ops, per-message (document) atomicity, batch processing, periodic processing, durability, sharding, operational familiarity (n.b. that would be the big one for me!). One drawback: AMQP push model needs to be emulated with MongoDB polling. To model topic matching, they’re using a regex. One thing they don’t (can’t) do with Mongo: fanout.

Use a capped collection? It has better performance but is limited to a single node and FIFO. They use an uncapped collection: can shard. Can get semi-FIFO but not strict.


Each message is a document. To create a message, just insert. The document has a queue field (string id) and a payload (serialized data).

To consume a message they use a findAndModify to grab and remove a document atomically. They index on (queue, _id).

That’s pretty much it! This would be pretty simple to implement in any language (he’s showing an example in the shell + in Python).

Benchmarks they ran showed MongoDB outperforming RabbitMQ for message creation by 19% (this is a single-node benchmark on a laptop, FYI). For consumption MongoDB again does very well (outperforming RabbitMQ for different levels of concurrency).

FindAndModify is blocking, so you will see high lock % w/ lots of concurrent consumers.

Pros and Cons

Pro: familiar, sharding, durability/persistence, low operational overhead, optional use of advanced queries.

Cons: Not AMQP, needs to poll, performance depends on polling frequency + concurrency, fewer libraries available (for Python there’s a library called Kombu), locking for findAndModify.

Source:  http://blog.fiesta.cc/post/13984644719/live-blogging-mongosv-mongodb-as-a-message-queue

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}