"Schemas" in CouchDB

schema noun ( pl. schemata or schemas )

1 technical a representation of a plan or theory in the form of an outline or model: a schema of scientific reasoning.

2 Logic a syllogistic figure.

3 (in Kantian philosophy) a conception of what is common to all members of a class; a general or essential type or form.

CouchDB is a schema-less document store, but there are times when a schema is a good thing to have around, one way or another. So can you have your cake and eat it too?

Below I'll take a high level look at adding a kind of schema to an application and the benefits and draw backs associated with this way of working. What I describe below isn't for everyone. It goes against some of the core principles of CouchDB and makes your data much less human readable, but there are cases where that trade off is worth making.

Schemas: WTF?!

It might seem a bit weird to add a schema to a schema-less database but sometimes it is a very useful thing indeed. When you're dealing with large datasets verbose object key names can be a problem (e.g. cost you money) so you end up stuck between a rock and a hard place; either make your data terse and hard to use or be explicit and spend more on storage and network.

{
    "shape": "triangle",
    "colour_label": "red",
    "opposite_length_in_mm": 767.12254256805875,
    "angle_in_radians": 1.5514293603308698,
    "adjacent_length_in_mm": 73.59881843627835
}

What usually happens is some middle ground where a nice descriptive name like "angle_in_radians" gets reduced to "angle" or "rads". That's fine in that it reduces the storage and network required to deal with all that data.

{
    "adj": 73.59881843627835,
    "shape": "triangle",
    "angle": 1.5514293603308698,
    "opp": 767.12254256805875,
    "colour": "red"
}

However, by making this small change you move the description of the data out of your database and into some undefined place; higher level code, documentation, shared knowledge, a whiteboard, a notebook, someones head.

As your data becomes more terse you might rely on duck typing (deriving from the data itself what the data describes) to get data that quacks right in your application. That's fine so long as you have data that is sufficiently distinguishable from the other ducks on the pond; if I rely on pulling a triangle object from the database because it has an angle member I might accidentally pull out a rhombus or an icosahedron.

To make sure you get the data you expect you might add an explicit type field to each data (e.g. "type=goose" or "shape=triangle") something which I've always felt was rather odd. This starts to add up on storage (remember you have a large dataset/flock of ducks) and, more importantly, it doesn't help with where the description of the data is held - you know that you have a goose but don't know what a goose is.

This last point is important, especially if you're working in a team of developers. Knowing what describing a shape as a triangle means is vital in producing consistent code that many people can work on. The straight jacket of a SQL schema looks pretty comfy sometimes.

Okay, I'll buy that a schema might be useful...

So how do you add a schema into a CouchDB database, something that is inherently schema-less? Can I get the best of both worlds? Here's a little trick that might help.

First you define a document that is the schema for a particular type of data:

{
    "_id": "datatype/triangle/v1",
    "fields": [
        "opposite_length_in_mm",
        "adjacent_length_in_mm",
        "angle_in_radians",
        "colour_label"
    ]
}

Then you change your document structure to reference that "schema":

{
    "datatype": "triangle/v1",
    "data": [
        879.07395066446952,
        84.607510245708468,
        1.4444230241122715,
        "red"
    ]
}

Note that the schema is versioned and that ordering in the data list is important here!

I now know precisely what the data represents without having to store that description in the data itself. This way of working has benefits beyond disk storage; you reduce wire traffic, and there is less for a client to parse before rendering it. This is especially useful if you're rendering into a browser based visualisation - you don't need a complex set of objects to make a bar chart, just a list of x and y values.

I can also share the data structure with colleagues and be reasonably confident that when I'm talking about a "v1 triangle" they'll know that lengths are in millimeters, are the opposite and adjacent sides and that the angle is in radians, hopefully reducing the chance of costly mistakes.

Isn't that error prone?

Yes and no. If you make a mistake in the ordering of your fields then, yes you are going to have issues. This is reasonably easy to manage with some form of client verification (e.g. validation on a web form) and generating the interface from the data (e.g. use the schema definition to build the GUI).

If you're adding these data into the database by hand (e.g. via a curl or futon) then you aren't going to be in the regime where this trick is useful; your dataset needs to be large for this to make sense.

Things still quack

What's particularly nice about this way of working is that I can still duck type the data, add additional fields to annotate it etc. since the schema isn't strictly enforced. Nothing stops me from having a triangle document like:

{
    "datatype": "triangle/v1",
    "data": [
        879.07395066446952,
        84.607510245708468,
        1.4444230241122715,
        "red"
    ],
    "owner": "Simon",
    "location" "space"
}

My views that deal with the data with a schema will still work (by ignoring these additional fields), my MVC framework will still render my pages, and I'll still have all the data I want in my database.

Nesting

You could have a nested object structure like:

{
  "datatype": "pattern/v1",
    "data": [
    {
      "datatype": "triangle/v1",
      "data": [
        879.07395066446952,
        84.607510245708468,
        1.4444230241122715,
        "red"
      ],
      "owner": "Simon",
      "location" "space"
    },
    {
      "datatype": "triangle/v1",
      "data": [
        879.07395066446952,
        84.607510245708468,
        1.4444230241122715,
        "blue"
      ],
      "owner": "Fred",
      "location" "space"
    },
    {
      "datatype": "square/v1",
      data: [
        10,
        "green"
      ]
    }
  ]
}

But if you're going to have a schema you may as well reflect the nesting inside it, e.g say that you have a list of triangles and a list of squares:

{
  "_id": "datatype/pattern/v1",
    "fields": [
      ["triangle/v1"],
      ["square/v1"]
    ]
}


{
  "datatype": "pattern/v1",
    "data": [
      [
        {
          "data": [
            879.07395066446952,
            84.607510245708468,
            1.4444230241122715,
            "red"
          ],
          "owner": "Simon",
          "location" "space"
        },
        {
          "data": [
            879.07395066446952,
            84.607510245708468,
            1.4444230241122715,
            "blue"
          ],
          "owner": "Fred",
          "location" "space"
        }
      ],
    [
      {
        data: [
          10,
          "green"
        ]
      }
    ]
}

Schema evolution

A nice feature of this way of working is that you can deal with schema evolutions; changing the format of your data.

{
  "_id": "datatype/triangle/v2",
  "fields": [
    "opposite_length_in_cm",
    "hypotenuse_length_in_cm",
    "angle_in_degrees",
    "colour_label"
  ]
}

There are only so many ways you can represent the data. While sometimes you may have a major schema evolution, one where old data is completely unusable, often changes are just tweaks for consistency (say changing the units of a quantity) or extending the schema by adding in optional data. In either case you should be able to use data from multiple schema versions together by using appropriate manipulations on the data. For example you could instantiate shape objects via a factory which knows how to create the right object for different schema versions.

Validation

The above does no validation of the data; the color field in the input data could be set to a number instead of a string, the angle to something non- physical etc. If you really needed validation you could do it with CouchDB's validation functions.

If you go the fully validated route you'd want to define the schema in the design document (instead of as a normal doc) and use a CommonJS include to make sure that the validator in the app was doing the same thing as the schema. This ties you to a version of the design document (which is where the validators live), which may or may not be an issue. It will also considerably slow down insertion rate as CouchDB has to do more work to add your data.

Personally I prefer to put validation logic in the client making writes.

Views

If I were using this way of working I would want to have a view which returned all the schema's defined on the database. This then allows me to build objects appropriately. A view to return schema's documents would look like:

function(doc) {
  if (doc._id.slice(0, 'datatype'.length) == 'datatype') {
    emit (doc._id.slice('datatype/'.length, doc._id.length), doc.fields)
  }
}

You can pull out documents that have a schema with a simple view like:

function(doc) {
  if (doc.datatype){
    emit(doc.datatype, doc.data);
  }
}

This can be queried to find objects of a given shape using CouchDB's view slicing (e.g. ?startkey="square/v1"&endkey="square/v2") which returns data like:

{"id":"datatype/square/v1","key":["square/v1",0],"value":["side_length_in_mm","colour_label"]},
{"id":"f98ffe7e4cd91cbb0d904f9098499ca8","key":["square/v1",1],"value":[872.4342711412228,"green"]},
{"id":"f98ffe7e4cd91cbb0d904f909849a218","key":["square/v1",1],"value":[370.29971491443905,"yellow"]},
{"id":"f98ffe7e4cd91cbb0d904f909849acd0","key":["square/v1",1],"value":[8.799279300193753,"yellow"]}

You'll notice the name of the "schema" is the key and the values are held in value. This means I can parse the data into a set of appropriate objects with something like:

var objects = [];

function build(schema, data){
  // Build the appropriate object for the schema...
}

for (row in data){
  // build up the objects in a factory
  var obj = build(row.key, row.value);
  objects.push(obj);
}

If I wanted all versions of a shape the query would be, and used a vNUMERIC_COUNTER notation for versioning, ?startkey="square/v1"&endkey="square/vXXX" as numbers sort lower than strings.

Taking it to the extreme

If you are really worried about data size you can take this technique to the extreme by encoding the data arrays as a byte string and using the schema documents to describe that byte array. This effectively turns your JSON structure into something not dissimilar to a protocol buffer, at the expense of human readability and view complexity. If you are particularly concerned with data size over the wire (for example are writing an MMORPG) then this may be an acceptable trade off.

Reminder

This trick isn't suitable for every dataset. If you modify the data by hand it is prone to error. If you have a small dataset, or only ever send a small subset of the data to the client it's massive overkill. But if you have a large dataset of machine generated data, that needs to be frequently accessed over the WAN (think a monitoring app or game) then this is a nice way to reduce storage, network IO and browser render time.

It's also worth reiterating that the schema is not enforced, you could have a square with 3 sides, and that adding strict schema enforcement with a validation function will considerably slow down insert rate.