While experimenting with some time series collections I needed a large data set to check that our aggregation queries don’t become a bottleneck in case of increasing data loads. We settled for 50 million documents, since beyond this number we would consider sharding anyway.

Each time event looks like this:

{
        "_id" : ObjectId("5298a5a03b3f4220588fe57c"),
        "created_on" : ISODate("2012-04-22T01:09:53Z"),
        "value" : 0.1647851116706831
}

As we wanted to get random values, we thought of generating them using JavaScript or Python (we could have tried it in Java, but we wanted to write it as fast as possible). We didn’t know which one will be faster so we decided to test them.

Our first try was with a JavaScript file run through the MongoDB shell.

Here is what it looks like:

var minDate = new Date(2012, 0, 1, 0, 0, 0, 0);
var maxDate = new Date(2013, 0, 1, 0, 0, 0, 0);
var delta = maxDate.getTime() - minDate.getTime();

var job_id = arg2;

var documentNumber = arg1;
var batchNumber = 5 * 1000;

var job_name = 'Job#' + job_id
var start = new Date();

var batchDocuments = new Array();
var index = 0;

while(index < documentNumber) {
    var date = new Date(minDate.getTime() + Math.random() * delta);
    var value = Math.random();
    var document = {       
        created_on : date,
        value : value
    };
    batchDocuments[index % batchNumber] = document;
    if((index + 1) % batchNumber == 0) {
        db.randomData.insert(batchDocuments);
    }
    index++;
    if(index % 100000 == 0) {  
        print(job_name + ' inserted ' + index + ' documents.');
    }
}
print(job_name + ' inserted ' + documentNumber + ' in ' + (new Date() - start)/1000.0 + 's');

This is how we run it and what we got:

mongo random --eval "var arg1=50000000;arg2=1" create_random.js
Job#1 inserted 100000 documents.
Job#1 inserted 200000 documents.
Job#1 inserted 300000 documents.
...
Job#1 inserted 49900000 documents.
Job#1 inserted 50000000 in 566.294s

Well, this is beyond my wild expectations already (88293 inserts/second).

Now it’s Python’s turn. You will need to install pymongo to properly run it.

import sys
import os
import pymongo
import time
import random

from datetime import datetime

min_date = datetime(2012, 1, 1)
max_date = datetime(2013, 1, 1)
delta = (max_date - min_date).total_seconds()

job_id = '1'

if len(sys.argv) < 2:
    sys.exit("You must supply the item_number argument")
elif len(sys.argv) > 2:
    job_id = sys.argv[2]   

documents_number = int(sys.argv[1])
batch_number = 5 * 1000;

job_name = 'Job#' + job_id
start = datetime.now();

# obtain a mongo connection
connection = pymongo.Connection("mongodb://localhost", safe=True)

# obtain a handle to the random database
db = connection.random
collection = db.randomData

batch_documents = [i for i in range(batch_number)];

for index in range(documents_number):
    try:           
        date = datetime.fromtimestamp(time.mktime(min_date.timetuple()) + int(round(random.random() * delta)))
        value = random.random()
        document = {
            'created_on' : date,   
            'value' : value,   
        }
        batch_documents[index % batch_number] = document
        if (index + 1) % batch_number == 0:
            collection.insert(batch_documents)     
        index += 1;
        if index % 100000 == 0:
            print job_name, ' inserted ', index, ' documents.'     
    except:
        print 'Unexpected error:', sys.exc_info()[0], ', for index ', index
        raise
print job_name, ' inserted ', documents_number, ' in ', (datetime.now() - start).total_seconds(), 's'

We run it and this is what we got this time:

python create_random.py 50000000
Job#1  inserted  100000  documents.
Job#1  inserted  200000  documents.
Job#1  inserted  300000  documents.
...
Job#1  inserted  49900000  documents.
Job#1  inserted  50000000  in  1713.501 s

This is slower compared to the JavaScript version (29180 inserts/second), but lets not get discouraged. Python is a full-featured programming language, so how about taking advantage of all our CPU cores (e.g. 4 cores) and start one script per core, each one inserting a fraction of the total documents number (e.g. 12500000).

import sys
import pymongo
import time
import subprocess
import multiprocessing

from datetime import datetime

cpu_count = multiprocessing.cpu_count()

# obtain a mongo connection
connection = pymongo.Connection('mongodb://localhost', safe=True)

# obtain a handle to the random database
db = connection.random
collection = db.randomData

total_documents_count = 50 * 1000 * 1000;
inserted_documents_count = 0
sleep_seconds = 1
sleep_count = 0

for i in range(cpu_count):
    documents_number = str(total_documents_count/cpu_count)
    print documents_number
    subprocess.Popen(['python', '../create_random.py', documents_number, str(i)])

start = datetime.now();

while (inserted_documents_count < total_documents_count) is True:
    inserted_documents_count = collection.count()
    if (sleep_count > 0 and sleep_count % 60 == 0): 
        print 'Inserted ', inserted_documents_count, ' documents.'     
    if (inserted_documents_count < total_documents_count):
        sleep_count += 1
        time.sleep(sleep_seconds)  

print 'Inserting ', total_documents_count, ' took ', (datetime.now() - start).total_seconds(), 's'     

Running the parallel execution Python script goes like this:

python create_random_parallel.py
Job#3  inserted  100000  documents.
Job#2  inserted  100000  documents.
Job#0  inserted  100000  documents.
Job#1  inserted  100000  documents.
Job#3  inserted  200000  documents.
...
Job#2  inserted  12500000  in  571.819 s
Job#0  inserted  12400000  documents.
Job#3  inserted  10800000  documents.
Job#1  inserted  12400000  documents.
Job#0  inserted  12500000  documents.
Job#0  inserted  12500000  in  577.061 s
Job#3  inserted  10900000  documents.
Job#1  inserted  12500000  documents.
Job#1  inserted  12500000  in  578.427 s
Job#3  inserted  11000000  documents.
...
Job#3  inserted  12500000  in  623.999 s
Inserting  50000000  took  624.655 s

This is very good indeed (80044 inserts/seconds) even if still slower than the first JavaScript import. So lets adapt this last Python script to run the JavaScript through multiple MongoDB shells.

Since I couldn’t supply the required arguments to the mongo command, to the sub-process started by the main python script, I came up with the following alternative:

for i in range(cpu_count):
    documents_number = str(total_documents_count/cpu_count)
    script_name = 'create_random_' + str(i + 1) + '.bat'
    script_file = open(script_name, 'w')
    script_file.write('mongo random --eval "var arg1=' + documents_number +';arg2=' + str(i + 1) +'" ../create_random.js');
    script_file.close()
    subprocess.Popen(script_name) 

If you enjoy reading this article, you might want to subscribe to my newsletter and get a discount for my book as well.

We generate shell scripts dynamically and let python run them for us.

Job#1 inserted 100000 documents.
Job#4 inserted 100000 documents.
Job#3 inserted 100000 documents.
Job#2 inserted 100000 documents.
Job#1 inserted 200000 documents.
...
Job#4 inserted 12500000 in 566.438s
Job#3 inserted 12300000 documents.
Job#2 inserted 10800000 documents.
Job#1 inserted 11600000 documents.
Job#3 inserted 12400000 documents.
Job#1 inserted 11700000 documents.
Job#2 inserted 10900000 documents.
Job#1 inserted 11800000 documents.
Job#3 inserted 12500000 documents.
Job#3 inserted 12500000 in 574.782s
Job#2 inserted 11000000 documents.
Job#1 inserted 11900000 documents.
Job#2 inserted 11100000 documents.
Job#1 inserted 12000000 documents.
Job#2 inserted 11200000 documents.
Job#1 inserted 12100000 documents.
Job#2 inserted 11300000 documents.
Job#1 inserted 12200000 documents.
Job#2 inserted 11400000 documents.
Job#1 inserted 12300000 documents.
Job#2 inserted 11500000 documents.
Job#1 inserted 12400000 documents.
Job#2 inserted 11600000 documents.
Job#1 inserted 12500000 documents.
Job#1 inserted 12500000 in 591.073s
Job#2 inserted 11700000 documents.
...
Job#2 inserted 12500000 in 599.005s
Inserting  50000000  took  599.253 s

This is fast too (83437 inserts/second), but still can’t beat our first attempt.

If you enjoyed this article, I bet you are going to love my book as well.

Conclusion

My PC configuration is nothing out of the ordinary, and the only optimization is that I have a SSD drive on which MongoDB runs.

The first attempt yielded the best results, and monitoring CPU resources I realized MongoDB leverages all of them even for a single shell console. The Python script running on all cores was also fast enough and it has the advantage of allowing us to turn this script into a fully operational application, if we want to.

Code available on GitHub.