Reduce ^ 2 in RavenDB
Reduce ^ 2 in RavenDB
Join the DZone community and get the full member experience.Join For Free
Build vs Buy a Data Quality Solution: Which is Best for You? Gain insights on a hybrid approach. Download white paper now!
An interesting question that keeps popping up is how to re-reduce the results of a map/reduce. That is a really nice feature on the surface, but it has a lot of implications. For example, when and how you run the 2nd reduce, can you chain only 1 time, or multiple times, what happens when there are a lot of reduce results, etc.
But most of the time, what people want is to be able to do aggregation on the map/reduce results without too much hassle, and they don’t have a lot of aggregated results or they are fine with waiting for them if they are very large. And we have a really nice solution for that scenario.
You start by defining the base map/reduce operation, like so:
Note that we need to also output the fields that we care about reducing further. In this case, we start by reducing to postal code, but we keep the city, region and country options as well.
Then, we define a transformer. Note that this is a special transformer, in that it has a group by in it, and it takes some parameters from outside.
Using those two together, we can now get the following results…
Raw map/reduce output:
With loc = City, we get:
With loc = Country, we get:
Tada, we have reduced further the result of a map/reduce operation. Now, this is subject to the usual limitations of RavenDB paging, in that it will only go through the only 1024 results. That can be a problem, but that is why RavenDB has the Streaming API.
You can use streaming on a map/reduce index with a transformer (and even apply parameters on top of that). That end up giving you the ability to run a re-reduction on top of a map/reduce index regardless of size.
Of course, on very large result sets, that can take quite a while, but that is expected and usually fine. For that matter, if you need to, you can chain the stream into a bulk insert, and get the re-reduction in that manner.
Published at DZone with permission of Oren Eini, CEO RavenDB , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.