wekeroad - Hello Biggy

A File-based Document Store for .NET

I've been using NeDB (a file-based document store for Node) for a few projects and I utterly love it. Such a simple idea, so fast, so elegant and many times just what I need! I had assumed that such a thing must be around for .NET because there are about 100 different kinds of lists in C#... someone must have made one with a persistent backing store!

But I looked around and couldn't find it, so I made it (as I'll need this in the coming months).

The idea is basically this: I want to use LINQ, I like Dynamics, and I like speed. So that's it, and here's Biggy:

...

Reading and Writing

So, by now you should be wondering why this is useful. The simple answer is that if you have a high-read application (like a blog, CMS, etc) then something like Biggy could speed things up.

Whenever you instantiate a new BiggyList it tries to read it's data from disk - this is good, and it's bad. It's good because from that point on whenever you try to query your data (using LINQ) it's an in-memory operation and you can't get much faster than that.

It's bad because this means you probably want to have a single DB instance around for the life of your app. This might be easy for some, might be repulsive to others. I'm used to doing this kind of thing with Node (all modules in Node are cached which means you always hit the same module instance).

For a blog engine, this might be a very fun thing to have - no database installs, superfast, easy to use. For a Twitter clone... not so much.

...

robconery / biggy

Biggy: SQLite for Documents and .NET

This is just a goofy idea at this point, inspired by NeDB which is basically the same thing, but with Node.

I like the idea of SQLite (a file-based relational data-store), but wouldn't it be fun to have this kind of thing for a Document database too? One nice thing about C# (among many) is the built-in LINQ stuff, another nice thing is that C# has Dynamics now too. Biggy is simply an implementation of ICollection with a JSON backing store. I added a few helpy things in there (like events and a few other things) and this might be completely dumb but I like the idea.

...

What It's Good For

The only disk activity occurs when you call "Save()" and when you instantiate the List itself - everything else happens in memory. This makes Biggy incredibly fast but it also means we're doing file management - which can be tricky.

This is one place that I hope I can get a PR for - I'm dropping the entire contents to disk on every save and YES if you try this will millions of records it will probably cause you some problems. But with 100 or so, it shouldn't be that big of a problem.

That makes Biggy compelling for high-read situations, such as a blog, product catalog, etc. At least that's what I've used NeDB for and it works great.

Performance

In the Tasks project (a Console app) there are simple loops that write 1000 records to disk at once (in a batch) as well as a simple read. You can see the results for yourself... they are OK.

Writing 1000 records in a batch takes about 30ms (give or take), writing in a loop takes about 4 seconds (!), but reading records out is too small to record :):):).

There's a lot to do to make this a bit more functional, but for now it does what I envisioned.

Wanna Help?

..."

Now that's an interesting project to play with... hum... and it looks like of fun too. [i.e. queued for a futureCoding4Fun blog post ;]