NoSQL for Storage, Relational for Reporting? No Friggin' Way.

given i think there is a shift afoot towards client-side mvc dealing in whole json documents up and down the wire, i’m wondering if there is also need for a technology to transparently create/maintain relational record sets for the pertinent document writes. relational records are desirable as they would support easy sql -using reporting tools. tools like crystal or jasper reports.

i’m not so much interested in the enhancements to sql that allow for json fragments in queries . nor am i interested the custom search engine solutions like elasticsearch that allow for sophisticated querying of an index. i’m looking for something that does secondary writes to a normal relational schema (whether defined or automatically generated), in such a way that crystal or jasper reports (or equivalent) could easily consume records for reporting. the resulting schema (aside from the cost of the secondary writes), would be optimized for queries.

martin fowler, as ever, sets the foundation for what i’m wanting with a bliki entry reportingdatabase

a worked example

here’s json fragment similar to the first one on: http://en.wikipedia.org/wiki/ json

{
	 "id"       : 101,
     "firstname": "john",
     "lastname" : "smith",
     "age"      : 25,
     "address"  :
     {
         "streetaddress": "21 2nd street",
         "city"         : "new york",
         "state"        : "ny",
         "postalcode"   : "10021"
     },
     "phonenumber":
     [
         {
           "type"  : "home",
           "number": "212 555-1234"
         },
         {
           "type"  : "fax",
           "number": "646 555-4567"
         }
     ]
}

if that resource were overwritten in a nosql database, then it would be one i/o. a trigger could pick that write up and, in postgres at least, could handle the economic updates as well as the deletes/writes as applicable into a relational schema. in this case that would be four writes (hopefully to a separate database/server to the document store):

person

personaddress

personphonenumber

extracting relational fields from json declaratively.

ideally, there would be a declarative xpath -like language that would allow us to tersely declare the sections of a json document we would want to explode into rows in a relational schema. two alternates exist already – jaql or jsonpath , but we’d want to use them in a way that would batch process documents into cohesive sql statements:

{
     "tables":
     [
         {
           "name"  : "person",
           "key": "id"
		   "fields": "firstname, lastname, age"
         },
         {
           "name"  : "personaddress",
           "key": "id"
           "fields": "address.streetaddress, address.city, address.state, address.postalcode"
         },
         {
           "name"  : "personphonenumber",
           "key": "id, phonenumber.@index as ix"
		   "fields": "phonenumber[ix].type, phonenumber[ix].number"
         }
     ]
}

whether ddl could be automatically extracted to issue the correct create table statements is debatable. it’s also not clear whether json will be a successful notation for extracting relational data from json documents.

put versus post

web apps are classically constructed from get and post applications. here is a timeline of web technologies in two blog entries – 1993, 2000, 2006 and its follow-up 2012 , lastly one extolling the virtues of document is the single source of truth . in the second and third i talk about the potential for put as a mechanism for writing back documents that were ‘worked on’ in a page. in relation to today’s blog entry, the put of the whole document all the way back to the database is powerful because it facilitates a trigger on the old and the new document in one operation. you may be driven to post because of a larger document size, and have to do additional processing in your persistence tier to make sense of that in a document store, but the consequential conversion to a relational form would still be possible.

of course, whether put or post , json docs or fragments would still need re-validating on the server-side as well as auth/audit/access control.

it’s important to note that many esteemed thoughtworks colleagues counsel against put and for post to do discrete updates to a backend document.

blast from the past: oracle’s intermedia.

eleven years ago oracle had a technology intermedia that would (amongst other things) take a write of xml to a clob and write out additional records to related tables to allow direct indexing of xml attributes/elements. they don’t so much talk of it any more. i can’t remember whether that was done in real-time or not, but it feels like it was in the same space as what i’m wanting.

microsoft’s sql server analysis services

like oracle’s intermedia, microsoft have a technology sql server analysis services . unlike intermedia, it is most definitely current.

amongst many other olap things, it explodes xml clobs into eminently searchable cubes. using this mas (?) piece, it is easy to make cubes for various sets of data, and from that perform easy reporting very much offline. in the end though, it is not quite what i’m looking for.

follow ups.

nic ferrier for one is itching to write something more technical for the nosql to relational idea. i’ll link to it when he finishes it.