Benchmarking RavenDB 4.0 vs. CouchDB

I ran into this post, which was pretty interesting to read. It compares a bunch of ways to index inside CouchDB, so I decided to see how RavenDB 4.0 compares.

I wrote the following code to generate the data inside RavenDB:

public class User
{
    public int Score;
    public string Name;
    public DateTime CreatedAt;
}

private static char[] _buffer = new char[6];
private static string RandomName(Random rand)
{
    _buffer[0] = (char)rand.Next(65,91);
    for (int i = 1; i < 6; i++)
    {
        _buffer[i] = (char) rand.Next(97, 123);
    }
    return new string(_buffer);
}
static void Main(string[] args)
{
    using (var store = new DocumentStore
    {
        Url = "http://localhost:8080",
        DefaultDatabase = "bench"
    }.Initialize())
    {
        var sp = Stopwatch.StartNew();
        using (var bulk = store.BulkInsert())
        {
            var rand = new Random();
            for (int i = 0; i < 100*1000; i++)
            {
                bulk.Store(new User
                {
                    CreatedAt = DateTime.Today.AddDays(rand.Next(356)),
                    Score = rand.Next(0, 5000),
                    Name = RandomName(rand)
                });
            }
        }
        Console.WriteLine(sp.Elapsed);
    }
}

In the CouchDB post, this took… a while. With RavenDB, this took 7.7 seconds and the database size at the end was 48.06 MB. This is my laptop, a 6th gen i7 with 16 GB RAM and SSD drive. The CouchDB tests were run over a similar machine, but with 8 GB RAM, but RavenDB didn’t get to use much memory at all throughout the benchmark.

It is currently sitting on a roughly 205MB working set and allocated a total of 70 MB managed memory and 19 MB of native memory.

I then created the following index:

This does pretty much the same as the indexes created in CouchDB. Note that in this case, this is running in process, so the nearest equivalent would be Erlang native views.

This took 1.281 seconds to index 100,000 documents, giving us a total of 78,064 indexed documents per second.

The working set grew to 312MB during the indexing process.

But those were small documents. What about larger ones? In the linked post, there was another test done, this time with each index also having a 300KB text property. Note that this is a single property.

The math goes like this: 300KB * 100,000 documents = 30,000,000 KB = 29,296 MB = 28.6 GB.

Inserting those took 1 minute and 40 seconds. However, the working set for RavenDB peaked at around 500 MB, and the total size of the database at the end was  512.06 MB. It took me a while to figure out what was going on.

One of the features that we have with the blittable format is the fact that we can compress large string values. The large string property was basically lorem ipsum repeated 700 times, so it compressed really well. And the beauty of this is that unless we actually need to touch that property and do something to it, it can remain compressed.

Indexing that amount of data took 1.45 seconds, for 68,965 indexes documents per second.

My next test was to avoid creating a single large property and go with a lot of smaller properties.

var entity = new User
{
    CreatedAt = DateTime.Today.AddDays(rand.Next(356)),
    Score = rand.Next(0, 5000),
    Name = RandomName(rand),
};
for (int j = 0; j < rand.Next(150,1500); j++)
{
    entity.CustomProperties[RandomName(rand)] = Random600CharString(rand);
}

This generates 100,000 documents in the 90–900KB range. Inserting this amount of data took a while longer. This is mostly because of the amount of data that we write in this scenario, which is in the many GB range. In fact, this is what my machine looked like while the insert was going on:

Overall, the insert process took 8:01 minutes to complete. And the final DB size was around 12 GB.

Indexing that amount of information took a lot longer and consumed a few resources:

Time to index? Well, it was about twice as much as before, with RavenDB taking a total of 2.58 seconds to index 100,000 documents totaling 12 GB in size.

That comes to 38,759 indexes documents per second.

A large part of that is related to the way we are storing information in blittable format. We don’t need to do any processing to it, so we are drastically faster.

Comparing to the CouchDB results in the linked post, we have:

In other words, we are talking about being four times faster than the faster CouchDB option.