Real-Time Dashboard with Node.js & NuoDB

This post comes from Luke Bayas at the NuoDB Blog.

In a previous article we discussed Moonshot and our approach to designing a simple real time statistics dashboard powered by NuoDB. So how does it work? In this article we are going to talk about implementing a simple node.js web app that captures state changes in NuoDB and publishes these changes to listening clients. I am assuming a general familiarity and comfort with web related technologies. Here’s the agenda:

1. Front End Development with Backbone.js
2. HTML5 Websockets
3. Back End Development with Node.js + NuoDB
4. Putting It All Together into a sample web application

Introduction 

Dashboards are great ways of representing complex data sets in the simplest terms so one can gain value immediately from the information presented. Real time dashboards are particularly cool because it gives instant feedback to enable people to make informed decisions as state changes on a system.

Moreover, so you want to build a statistics dashboard, fast, that (1) takes a metric and stores it in NuoDB and (2) enables multiple users (possibly tens or hundreds, maybe even thousands) to view your real time dashboard.

Pretty intense. What is a viable, scalable solution?

Enter node.js and NuoDB to the rescue. As I mentioned in my previous article, Getting Started: Node.js & NuoDB, depending on your specific project requirements, NuoDB and node.js are a great pairing for building highly performant, scalable web applications or SOA based web products.

We are going to discuss at a high level a few technologies to create a simple dashboard interface to monitor statistics. Let’s get started.

Enter backbone.js

Part 1 — Front End Development w/ Backbone.js

There has been a lot of advancement in front end web development tools over the years directly related to increased client side development . Backbone.js is a minimalist tool kit that provides structure to a web application adhering to an MVC type paradigm. The reason I chose backbone.js versus angular.js, knockout.js, ember.js or others was more a matter of preference & familiarity than anything else, the same can easily be done in any front end framework. Check out the provided links to see which framework suits your taste, but for the purpose of this exercise, we will stick to backbone.js

The key responsibility of backbone.js in this exercise will be to (1) render the initial dashboard view, (2) establish a web socket connection with node.js and (3) subscribe to the “metrics:update” event from the server to update our view.

After the backbone.js app renders and the socket is connected, we will bind to the “metrics:update” event issued from node.js if and only if a state change happens. Until this happens, the front end and back end don’t need to talk to each other. The only time they talk is when state changes happen OR information is requested from either side. That is basically it for on the client side, the only thing we need now is data so we can dynamically change the user interface.

So how are we going to get this data in a bi-directional fashion?

Enter web sockets

Part 2 — HTML5 Web Sockets

The emergence of HTML5 into mainstream browsers has introduced a variety of really cool features, one of my favorites being web sockets. Why are web sockets really cool? Well, web sockets allow bi-directional communication between client to server. It was fairly common practice, prior to HTML5, to perform some type of polling mechanism from client to server to fetch new data from the server to emulate a real time experience. The issue with AJAX polling, meaning executing an HTTP GET against a server side resource every few seconds in many cases, is that it is extremely expensive and doesn’t scale well. There was also AJAX long polling (Comet) which would keep the connection alive before closing and re-opening again indefinitely which is better than the previously described solution, and arguably the best solution at the time, but it still was a hack in many cases.

Web sockets establish a connection between server and client so either can send or get data whenever they want, which is extremely powerful. In addition, the overhead of fetching data via web sockets is substantially lower in comparison to AJAX because it doesn’t have to re-send HTTP header info and, in some cases, terminate SSL connections for multiple requests of data. This makes web sockets particularly useful for web applications that have what I call a real time component inherent in the core of the application and I honestly feel most dynamic web apps are moving in this direction.

Moreover, there are a number of web socket libraries available that enable web socket support, but one of the most popular in the node.js community is socket.io. This library is really simple to install via npm in node, and once installed you can reference it from your client. So why the focus on web sockets? Well the primary means by which the backbone.js app and our NuoDB + node.js backend communicate is via JSON and we want to ensure this channel is highly performant for a real time dashboard.

So we talked about backbone.js as a front end framework for structuring our front end code, websockets as a means of transporting JSON from the back end to the front end, but who is responsible for publishing these updates to listening clients?

Enter node.js

Part 3 — Back End Development w/ NuoDB & Node.js

Now the backend comprises of a node.js service that is responsible for (1) servicing the initial HTML request that will load the backbone.js app and (2) publishing any NuoDB state changes on the web socket so the client can re-render changes. For our sample real time dashboard, metrics are (1) stored in NuoDB and (2) published on the “metrics:update” web socket event. This event may look familiar because in Part 1 we discussed subscribing to a “metrics:update” event, this is the other half, where we publish the update. Note that these have to match in order for the user interface to receive the correct event. The idea here is as new metrics come in from HTTP POST, we will store this information and publish, simple as that, and the views should update instantly.

For all intensive purposes the node.js server is extremely simple, less than 50 lines of code to implement what we just described.

Now that we discussed all the major components, let’s put it together with some boilerplate code

Part 4 – Putting It All Together

So we talked about the front end, the backend and how cool HTML5 web sockets are. The following is some boilerplate code to help you get started. For help setting up node.js & NuoDB, read the Getting Started: Node.js & NuoDB blog.

To get started, in your HTML page create a new div element with an id attribute of “metric”. This will be where we load new metric values from the node.js server.

Here is a simple Backbone.js view that listens on the web socket and executes an update method to change the state of a metric.

Here is our Backbone view that binds to the socket “metrics:update” event and calls the update() method:

view = Backbone.View.extend({
     initialize: function(){
          _.bindAll(this, "render", "update");
          Backbone.socket = io.connect(window.location.host);
          Backbone.socket.on("metrics:update", this.update);
     },
     update: function(data){          
          $("#metric").html(data.foo);
     },
     render: function(){
          return this;
     }
});

Awesome, next we need to setup node.js so that it can listen for socket connections and push updates out one it. In your web application, install socket.io via npm:

npm install socket.io

And our simple node.js server code looks like this:

http     = require("http");
express  = require("express");
io       = require("socket.io");
nuodb    = require("db-nuodb");

app      = express();
sio      = io.listen(app);

app.get("/", function(req, res){
     res.render("index", { title: "demo" });
});

app.post("/metrics", function(req, res){
     sio.sockets.emit("metrics:update", req.body);
     res.end();
});

Let’s enhance this to update a stat table in NuoDB that collects stat related information before publishing to all the listening clients

http     = require("http");
express  = require("express");
io       = require("socket.io");
nuodb    = require("db-nuodb");

app      = express();
sio      = io.listen(app);

new nuodb.Database({
     hostname: 'localhost',
     user: 'root',
     password: 'password',
     database: 'node'
}).connect(function(error) {
     if (error) {
          return console.log("ERROR: " + error);
     }

     app.get("/", function(req, res){
          res.render("index", { title: "demo" });
     });

     app.post("/metrics", function(req, res){
          cmd = "INSERT INTO METRICS(FOO) VALUES (" + req.body.foo + ")";
          this.query().execute(cmd, function(error) {
               if (error) {
                    return console.log('ERROR: ' + error);
               }
               metrics = { foo: req.body.foo };
               sio.sockets.emit("metrics:update", metrics);
               res.end();
          });
     });     
});

Now we need to drive some type of load onto the system. We can use a simple REST client to issue an HTTP POST to our node.js resource or we could implement a simple loader that executes an HTTP POST few seconds with a randomly generated stat. Regardless how one executes the HTTP POST, you will notice that as soon as the node.js post handler gets executed, the data will be persisted in NuoDB and all listening clients will be updated with the new metric.

These are the basic tools, so now go crazy! Checkout my other article, An Enterprise Management UI for Project Moonshot, for a look at our dashboard we implemented in a couple days using the above paradigm. For those interested in developing web apps with node, I highly recommend checking out Express, a web application framework for node.js.