Distributed MapReduce with Sharded MongoDB and SpringData

in mongodb you have several options when it comes to aggregating data. there is the included javascript mapreduce framework, the new aggregation framework in v 2.2+ and also the hadoop connector for really heavy lifting.

when using the integrated mapreduce framework you have to be aware of the caveat that mongodb’s js interpreter is single-threaded . this means that regardless of how many cores your server has only one gets utilized. that’s a bit of a bummer because vertical scaling might not decrease execution times significantly. so to really bring down execution times and make use of mapreduce’s parallel computing abilities you have to scale out and shard. this brings mainly two advantages. for one you have now more “threads” and also the dataset that each of these has to deal with is getting smaller in relation of how many shards you have.

the latest of stable version of mongodb today is 2.0.6. which was initially used for our queries. using this version queries that we successfully issued against a single mongodb instance failed on the sharded setup.
it seems that we were hitting an issue similar or equal to https://jira.mongodb.org/browse/server-5536 .

as the issue states that it’s fixed in 2.1.2 we switched to the latest nightly build (2.1.2-pre) which worked fine.

unfortunately after switching to 2.1.2 we were confrontend with https://jira.springsource.org/browse/datamongo-378

the pragmatic albeit not beautiful was a hack that reimplements the following interfaces: mongooperations and applicationcontextaware. it basically works around the type cast to integer:

((number) counts.get(“input”)).intvalue(),
((number) counts.get(“emit”)).intvalue(),
((number) counts.get(“output”)).intvalue()

all of this resulted in the following performance improvements:

these numbers show how much parallelism can actually result in a massive performance increase. mongodb brings excellent out-of-the-box capabilites to simplify sharding and replication.