How to Interact With a Database Using Various async Patterns in Node.js

It seems simple enough: Get a connection to the database, use it to do some work, then close it when you're done. But due to the asynchronous nature of Node.js, coding this sequence isn't as straightforward as it seems. There are lots of options for writing asynchronous code with Node.js, and each one requires the sequence to be coded differently. In this series, I'll provide some examples that demonstrate how to get, use, and close a connection using various async patterns.

In this parent post, I'll provide a little context on how async programming varies from traditional programming. The details of how a particular async pattern is used will be covered in its own post (see the links at the bottom).

try...catch...finally

JavaScript has a try…catch…finally statement. Thetry…catchpart is easy: run some code and catch exceptions so they can be handled gracefully. But why do we need finally? finallyprovides a way of running cleanup code, regardless of whether an exception is raised in the try or catch blocks. Cleanup code typically includes closing file handles and database connections.

Here’s an example:

try {
  console.log('try 1');
  throw new Error('foo');
  console.log('try 2');
} catch (err) {
  console.log('catch 1', err);
  throw new Error('bar');
  console.log('catch 2');
} finally {
  console.log('finally!');
}

If you run that in your browser console (or in Node.js), you should see something like this in the console:

try 1
catch 1 Error: foo
    at <anonymous>:3:9
finally!
Uncaught Error: bar
    at <anonymous>:7:9

Notice that the bar exception raised in the catch block does eventually bubble up as an unhandled exception, but not before thefinally block is executed.

If all of the code we ran in Node.js were synchronous, we’d be able to do something like this:

try {
  // get a connection to the DB
  // use the connection to do work
} catch (error) {
  // handle the error
} finally {
  // close the connection
}

Unfortunately, try…catch…finally isn’t very useful with asynchronous code — at least not until async functions are available to you. The reason has to do with the way async work is done in Node.js.

Asynchronous/Evented Processing

Here’s an overview of how asynchronous operations, such as making HTTP requests and executing database queries, are performed.

All of your JavaScript code runs on the main thread. Asynchronous APIs are invoked from the main thread and passed callback functions. Depending on the type, the async work may be completely evented, or it might leverage a thread pool. When the async work is complete, the callback function is added to the callback queue to be invoked on the main thread as soon as possible.

With this architecture, errors that occurred during the async processing are brought back to the main thread in an entirely different call stack. You can’t catch errors raised in a different call stack — it’s too late!

Here’s a very simple demo of invoking an async API:

setTimeout(function() {
  console.log('hello');
}, 2000);

console.log('world');

If you run that script in a browser console or Node.js, the output will be:

world
hello

Seeing “world” two seconds before “hello” can surprise folks new to async programming. The secret sauce that makes it work is the callback function passed to setTimeout. It will “call back,” or be executed on the main thread when the specified time has passed.

Callback functions can be traced back to the beginnings of JavaScript, which was designed as a language to bring life to the web. JavaScript developers needed a means of associating code with events such as the loading of a page, click of a mouse, or press of a key on a keyboard. Over time, DOM APIs such as the following were introduced.

window.addEventListener('load', function() {
  // do some work
});

The addEventListener method above takes the anonymous function passed in as the second parameter and adds it to a list of listeners related to the load  event on the window element. When the event occurs, all listeners are added to the callback queue and will be invoked as soon as possible.

For callbacks to work properly, functions in JavaScript needed some important features: they needed to be first-class objects and they needed closure.

Functions Are First-Class

The concept of functions as first-class objects sounds more complex than it is. Most programming languages provide a means of declaring variables and creating named units of code, often called functions or procedures. There’s usually a clear distinction between these two constructs. For example, you can declare variables and pass them into functions, but you can’t pass a function into a function.

JavaScript, on the other hand, allows functions to be used more like standard data types such as Number, String, and Boolean. Functions can be declared, passed around from function to function, and invoked at some point in the future. Treating functions as first-class objects is a prerequisite for passing a callback function into another function, but the feature can aid with code organization too.

Imagine you wanted to execute the same code when a page is loaded and when a user clicks the window. You could do something like this:

window.addEventListener('load', function() {
  // do some work
});

window.addEventListener('click', function() {
  // do the same work here
});

The problem with the code above is that we have to maintain two functions that do the same work. However, knowing that functions are first-class objects, we could declare a single, named function and pass a reference to it as needed:

function doWork() {
  // do some work
}

window.addEventListener('load', doWork);
window.addEventListener('click', doWork);

As you can see, functions as first-class objects is a simple but powerful feature of JavaScript.

Functions Provide Closure

The concept of a closure is probably a bit harder to wrap one’s head around at first — but it’s crucial for async/evented programming. Put simply, a closure is a function that refers to variables defined in its enclosing scope.

Many languages allow developers to nest functions within functions and child functions can refer to variables declared in the parent function’s scope. Developers using other languages may never wonder, “What would happen if the run-time invoked a child function after the parent function finished executing?” It’s simply not possible. But that’s not the case with JavaScript!

Remember that functions in JavaScript are first-class objects, so like any other value assigned to a variable, they can escape the confines of their parent function by being passed around. When this happens, the references to variables in the original enclosing scope (lexical scope) will still exist. So what should happen when the child function is invoked in the future?

Closure ensures that the child function will be able to access those variables as long as the run-time may need to invoke the child function. Such variables will not be garbage collected as they normally would be.

Here’s an example of a closure:

<html>
  <body>
    <button id="my-button">Click me!</button>

    <script>
    function onLoad() { // "onLoad" is the parent function.
      var button = document.getElementById('my-button');

      function onClick() { // "onClick" is the child function (closure).
        button.parentNode.removeChild(button); // "button" referes to the enslosing scope.
      }

      // "onClick" can be invoked after "onLoad" finishes but the reference to "button"
      // will still be valid thanks to closure.
      button.addEventListener('click', onClick);
    }

    window.addEventListener('load', onLoad);
    </script>
  </body>
</html>

You can copy and paste this code into a file with a .html extension and open it in a browser. You should see a button that says “Click me!”. When the window loads, the onLoad function registers the onClick function with the  click event on the button.

Note that onClick was not invoked within onLoad. Instead, a reference was passed to an API that can invoke the function in the future. Because onClick refers to the button variable declared in the onLoad (parent) function, closure ensures that onClick will have access to button when it’s invoked in the future.

Now that we’ve addressed how some of the core concepts related to async programming in JavaScript, let’s turn our attention to some of the async patterns that have evolved in Node.js.

Common Asynchronous Patterns

Currently, the most common (and generic) patterns used to write asynchronous code with Node.js are callbacks, the async module, and promises. Node.js v7.6 got an update to the V8 JavaScript engine which introduced a new means of doing async processing called async functions.

Building on top of the work done for generators and promises, async functions allow JavaScript code to be written synchronously while it executes asynchronously. Best of all, synchronous constructs such as loops and try…catch…finally work as you’d expect them to! Async functions are a serious game changer for JavaScript, but knowledge of promises, and thus async processing in general, will still be important.

Each pattern I'll be covering will have a dedicated post. Each post will explain the basics of the pattern and provide a demo app that uses the pattern. The demo apps will perform the same three async operations: get a connection to the database, use it to execute a simple query, then close the connection.

To run a demo app locally, assuming Node.js and the OCI client libraries are already installed, just clone the related Gist or create the various files in a directory. Next, open the directory in a terminal and run npm install followed by node index.js.

Tip: See this blog post if you’d like to set up a local sandbox for working through the examples.

Here are the links for each pattern (links will become active as new posts in the series are published):

• Callbacks
• Async module
• Promises
• Async functions