Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Fast Paging With MongoDB

DZone's Guide to

Fast Paging With MongoDB

Paging through your data is one of the most common operations with MongoDB. Let's walk through an example to see the different ways of doing it.

· Database Zone
Free Resource

Learn how to create flexible schemas in a relational database using SQL for JSON.

Paging through your data is one of the most common operations with MongoDB. A typical scenario involves the need to display your results in chunks in your UI. If you are batch processing your data, it is also important to get your paging strategy correct so that your data processing can scale.

Let's walk through an example to see the different ways of paging through data in MongoDB. In this example, we have a CRM database of user data that we need to page through and display 10 users at a time. So in effect, our page size is 10. Here is the structure of our user document:

{
    _id,
    name,
    company,
    state
}

Approach 1: Using skip() and limit()

MongoDB natively supports the paging operation using the skip() and limit() commands. The skip(n) directive tells MongoDB that it should skip ‘n’ results and the limit(n) directive instructs MongoDB that it should limit the result length to ‘n’ results. Typically, you will be using the skip() and limit() directives with your cursor, but to illustrate the scenario, we provide console commands that would achieve the same results. Also for the sake of brevity in the code, the limits checking code is also excluded.

//Page 1
db.users.find().limit (10)
//Page 2
db.users.find().skip(10).limit(10)
//Page 3
db.users.find().skip(20).limit(10)
........

You get the idea. In general, to retrieve page n, the code looks like this:

db.users.find().skip(pagesize*(n-1)).limit(pagesize)

However, as the size of your data increases, this approach has serious performance problems. The reason is that every time the query is executed, the full result set is built up, then the server has to walk from the beginning of the collection to the specified offset. As your offset increases, this process gets slower and slower.  Also, this process does not make efficient use of the indexes. So, typically the ‘skip()’ and ‘limit()’ approach is useful when you have small data sets. If you are working with large data sets you need to consider other approaches.

Approach 2: Using find() and limit()

The reason the previous approach does not scale very well is the skip() command. So, the goal in this section is to implement paging without using the ‘skip()’ command. For this, we are going to leverage the natural order in the stored data like a time stamp or an id stored in the document. In this example, we are going to use the ‘_id’ stored in each document. ‘_id’ is a MongoDB ObjectID structure which is a 12-byte structure containing timestamp, machined, processid, counter, etc. The overall idea is as follows:

1. Retrieve the _id of the last document in the current page
2. Retrieve documents greater than this “_id” in the next page

//Page 1
db.users.find().limit(pageSize);
//Find the id of the last document in this page
last_id = ...

//Page 2
users = db.users.find({'_id'> last_id}). limit(10);
//Update the last id with the id of the last document in this page
last_id = ...

This approach leverages the inherent order that exists in the “_id” field. Also since the “_id” field is indexed by default the performance of the find operation is very good. If the field you are using is not indexed your performance will suffer – so it is important to make sure that field is indexed.

Also, if you would like your data sorted in a particular order for your paging then you can also use the sort() clause with the above technique.  It is important to ensure that the sort process is leveraging an index for best performance. You can use the .explain() suffix to your query to determine this.

users = db.users.find({'_id'> last_id}). sort(..).limit(10);
//Update the last id with the id of the last document in this page
last_id = ...

Create flexible schemas using dynamic columns for semi-structured data. Learn how.

Topics:
mongo db ,mongo hosting ,mongodb database service provider ,mongodb enterprise server ,mongodb management

Published at DZone with permission of Dharshan R.. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}