Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

RavenDB's Querying Streaming: Unbounded Results

DZone's Guide to

RavenDB's Querying Streaming: Unbounded Results

· Database Zone
Free Resource

Download the Guide to Open Source Database Selection: MySQL vs. MariaDB and see how the side-by-side comparison of must-have features will ease the journey. Brought to you in partnership with MariaDB.

 

By default, RavenDB make it pretty hard to shoot yourself in the foot with unbounded result sets. Pretty much every single feature has rate limits on it, and that is a good thing.

However, there are times where you actually do want to get all where all actually means everything damn you, really all of them. That has been somewhat tough to do, because it requires you to do paging, and if you are trying to do that on a running system, it is possible that incoming data will impact the way you are exporting, causing you to get duplicates or miss items.

We got several suggestions about how to handle that, but most of those were pretty complex. Instead, we decided to go with the following approach:

  • We will utilize our existing infrastructure to handle exports.
  • We don’t want to do that in multiple requests, because that means that state has to be kept on both client & server.
  • The model has to be a streaming based model, because otherwise we might get memory errors if you are trying to load millions of records out.
  • The stream you get out is frozen, that means that what you read (both indexes and data) is a snapshot of the data as it was when you started reading it.

And now, let me show you the API for that:

    using (var session = store.OpenSession())

    {

        var query = session.Query<User>("Users/ByActive")
                           .Where(x => x.Active);

        var enumerator = session.Advanced.Stream(query);

        int count = 0;

        while (enumerator.MoveNext())

        {

            Assert.IsType<User>(enumerator.Current.Document);

           count++;

       }

    

       Assert.Equal(1500, count);

   }

As you can see, we use standard Linq to limit our search, and the new method we have is Stream(), which allows us to get an IEnumerator, which will scan through the data.

You can see that we are able to get more than the default 1,024 limit of items from RavenDB. There are overloads there for getting additional information about the query as well (total results, timestamps, etags, etc).

Note that the values returning from the Stream() are not tracked by the session. And that if we had a users #1231 that was deleted 2 ms after the export began, you would still get it, since the data is frozen at the time of the export start.

If you want, mind, you can specify paging, by the way, and all the other indexing options are also available for you as well (transform results, for example).






Interested in reducing database costs by moving from Oracle Enterprise to open source subscription?  Read the total cost of ownership (TCO) analysis. Brought to you in partnership with MariaDB.

Topics:

Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}