Reactive OrientDB — Live Queries

The NoSQL movement has significantly matured in the past few years. It's clearly much more than just a new set of data models. NoSQL has questioned data structures (no more tables, just key/value pairs, documents, graphs), data schema, validation (schemaless vs. fixed schema) and data consistency (transaction support, eventual consistency). Now the time has come to question the very concept of a “query.”

What is a Query?

We all know what a (database) query is: It's just a request to the database, with some constraints, that will return a consistent result set that reflects the status of the database (typically a portion of it) at the moment of execution. It's important to note that this "consistent snapshot" behavior is what we typically expect (see transaction isolation levels).

This is definitely good for some use cases, but for many others it's not. Just consider applications that manage changes, like news feeds, train/flight timetables or system monitoring. In all these cases, we are more interested in capturing the change rather than the status. When we're at the airport, we look at arrival timetables to know if our flight is delayed, and we want to know it as soon as it happens. When our favourite football team scores a goal, we'd like to receive the notification immediately.

Sticking to the old concept of a query, we have the following two alternatives:

• Polling: Execute the same query every minute (every second?) and check to see if changes happened.

• Relying on other external components, like event queues, which means admitting that the database will not help with this.

LiveQuery

OrientDB, since v. 2.1, supports a new type of query called LiveQuery. It is specifically focused on changes, rather than the current status.

The syntax of a LiveQuery statement is very similar to a normal query. However, the difference is in the behavior. A LiveQuery will not return a full result set when it is executed (in some cases, it will return nothing at all, even if the database is full). It will start returning results to the client as soon as the query result set changes, sending to the client only the differences compared to the initial situation (or to the last received update).

Let's clarify it with an example.

Example

Suppose you have a flight timetable that has to be updated every time a flight is rescheduled at your local airport. All of your flight data exists in your database, and you have multiple data sources populating it (including your DBA that just found an inconsistency in your database and is fixing it with a manual UPDATE).

Using a traditional approach, you have to query the database with a statement like following:

SELECT * FROM Flight WHERE AIRPORT = ? AND updateTimestamp > ?

And you will execute this query perhaps every second, even if nothing changed in the database (you never know).

With OrientDB LiveQuery, you can execute the following statement:

LIVE SELECT FROM Flight WHERE AIRPORT = ?

(Please note that I removed the condition on the timestamp)

Here's what to keep in mind:

• Provide a callback function.

• The database will not return anything at the beginning

• As soon as a record is inserted or updated, and it matches your WHERE condition, your callback function will be invoked

• The query will remain "live" until you explicitly "unsubscribe."

Avoid Wasting Calculation Resources

This is what happens with a traditional polling approach:

You are forced to execute the query multiple times in order to determine if something changed in the database (and maybe it will never happen). This results in a significant waste of calculation resources on the server, especially if your query involves heavy calculation or large result sets.

With LiveQuery, you will avoid this waste of resources because polling is not required.

Immediate feedback

When you define a polling strategy, you have to make a choice:

• You can execute the query at a high rate (< 1 query per second), having prompt updates, but at the risk of overloading the server.

• You can query at a lower rate, reducing the server load, but at the risk of having late updates:

With LiveQuery, you always have the lowest delay possible because changes are notified to the client as soon as they happen in the database.

No Information Loss

In some scenarios, a polling strategy can lose some changes. This can occur if two updates happen on the same record in the interval between two queries. If the second update overwrites the first one, your polling query will only notify the second change.

With LiveQuery, you will receive both updates, in the correct sequence.

Code Examples: Java

Writing a LiveQuery using the native OrientDB Java client is as easy as writing a normal asynchronous query. Here is a code sample:

Step 1: Define your query listener. It will contain application logic to manage data changes (live query results):

class MyLiveQueryListener implements OLiveResultListener {
   public void onLiveResult(int iLiveToken, ORecordOperation iOp) throws OException {
      ODocument doc = iOp.record.getRecord();
      if(iOp.type == ORecordOperation.CREATED){
        // your application logic here
      } else if(iOp.type == ORecordOperation.UPDATED){
        // your application logic here
      } else if(iOp.type == ORecordOperation.DELETED){
        // your application logic here
      } 
   }
}

Step 2: Instantiate the listener and execute the LiveQuery:

ODatabaseDocumentTx db = new ODatabaseDocumentTx("remote:localhost/testdb");
MyLiveQueryListener listener = new MyLiveQueryListener();
db.query(new OLiveQuery("live select from test", listener));
// this is non blocking, if you add other code here it will be executed immediately after the query dispatching 

Code Examples: Node.js

var OrientDB = require('orientjs')

var server = OrientDB({host: 'localhost', port: 2424});
var db = server.use({name: 'testdb', username: 'admin', password: 'admin'});

db.liveQuery("live select from test")
   .on('live-insert', function(data){
     //your app logic here
   })
   .on('live-update', function(data){
     //your app logic here
   })
   .on('live-delete', function(data){
     //your app logic here
   })

References

LiveQuery is currently supported in the following OrientDB drivers:

• Java 

• OrientJS (Node.js) 

• Elixir