How to Keep Node.js Processes Running

Node.js/Express.js is great for Web API’s and applications. In contrast to known enterprise technologies, Node.js is very special. It’s a single process/threaded environment. If an unhanded exception occurs, the Node.js virtual machine simply stops, leaving the application in an unresponsive state.

Due to the async nature of Node.js, try/catch doesn't always work, even with domains and stuff you have a chance that the application crashes on production while you sleep.

To mitigate the issue, there are few known solutions. The common idea is that there is a watchdog keeping an eye on node processes and if crashed, it restarts the application again.

Recently I’ve used a great library by @mafintosh called respawn. I liked its minimalistic style and decided to try it out.

The bare-bones code is very simple. Without modification of your application, just create file monitor.js with the following code:

var respawn = require('respawn');

var monitor = respawn(['node', 'server.js'], {
    env: {ENV_VAR:'test'}, // set env vars
    cwd: '.',              // set cwd
    maxRestarts:10,        // how many restarts are allowed within 60s
    sleep:1000,            // time to sleep between restarts
});

monitor.start(); // spawn and watch

monitor will spawn new node process and in case of crash it will be restarted. You can also specify maxRestars (I recommend to do that, if something is really bad it won’t be restarted infinitely) and sleep time.

I’ve tried that, by implementing /fail end-point in my app, to see that respawn really works.

app.get('/fail', function (req, res, next) {
  setTimeout(function () {
      var nu = null;
      nu.access();

      res.send('Hello World');
  }, 1000);
});

if I try to hit /fail I’ll see no results in browser, but if I go back to / the application is running in normal state.

But simple respawning of the application is not a complete solution. You need to know what exactly happened to be able to fix issue. Proper logging of your application is essential. I’ll show my small setup around respawn that sends critical message to Logentries, so all crashes are logged.

var respawn = require('respawn');
var util = require('util');
var logger = require('./source/utils/logger');

var proc = respawn(['node', 'app.js'], {
  cwd: '.',
  maxRestarts: 10,
  sleep: 1000,
});

proc.on('spawn', function () {
  util.print('application monitor started...');
});

proc.on('exit', function (code, signal) {
  logger.fatal({msg: 'process exited, code: ' + code + ' signal: ' + signal});
});

proc.on('stdout', function (data) {
  util.print(data.toString());
});

proc.on('stderr', function (data) {
  logger.error({msg: 'process error', data: data.toString()});
});

proc.start();

(Details of logger you can find in this post).

All process output goes to stdout, which is convinient for development, but in case of stderr or exit everything is logged to cloud and notification to dev-team sent.

It worked really nice, now I’m not worried even if something bad happens on production. respawn will make sure that the rest of users are not affected. As a developer, you can much quicker find bugs and push hotfixes.