DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Databases
  4. MongoDB Connection Pooling for Express Applications

MongoDB Connection Pooling for Express Applications

Working directly with databases is great, but the real world means using connection pooling. Let's set it up between your Express application and your Mongo database.

Chris Chang user avatar by
Chris Chang
·
May. 24, 17 · Tutorial
Like (0)
Save
Tweet
Share
15.17K Views

Join the DZone community and get the full member experience.

Join For Free

Express is the most popular Node.js web framework and the fourth most depended-upon package on the NPM registry. As a result of its popularity, there is an abundance of tutorials and examples for getting started with new Express apps — we too have created a "getting started" MEAN stack tutorial for the Heroku DevCenter. However, "getting started" apps generally don't show you how to handle the more serious parts of real-world systems.

In this post, we are going to demonstrate how a production Node.js application might connect to multiple MongoDB databases. Our example will demonstrate how to create a single connection pool to a MongoDB deployment and how to structure an Express app to reuse that pool across multiple modules.

You can find the code for this example here on GitHub.

What Is a Connection Pool and Why Is It Important?

A connection pool is a cache of authenticated database connections maintained by your driver, from which your application can borrow connections when it needs to run database operations. After each operation is complete, the connection is kept alive and returned to the pool. When properly used, connection pools allow you to minimize the frequency of new connections and the number of open connections to your database.

It's important to minimize the frequency and number of new connections to the database because creating and authenticating connections to the database is expensive — these processes require both CPU time and memory. When we reuse connections by using a connection pool we avoid this resource cost to the server and also benefit from lower latency since the app doesn't need to wait for an authentication to finish before a query can be sent.

The default pool size in the Node driver is 5. If your app typically needs more than 5 concurrent connections to the database, you can increase the value of the poolSize setting.

Common Mistake in Connection Management for Express Apps

A common mistake developers make when connecting to the database is to call MongoClient.connect() in every route handler to get a database connection.

For example, code like this...

app.route('/foo', function(req, res) {
    MongoClient.connect('url', function(err, db) {
        ...
    })
})

app.route('/bar', function(req, res) {
    MongoClient.connect('url', function(err, db) {
        ...
    })
})


...will create 5 connections to MongoDB for every request, including concurrent requests to the same route. Connection counts can quickly skyrocket with this approach.

Example for Connection Pooling in an Express App

For our example we'll implement database connection pooling for the recommended Express project structure (as outlined by the Express app generator).

Our example has the following structure:

|-- README.md
|-- app.js
|-- dbs
    |-- index.js
|-- routes
    |-- index.js


The app.js file is the main file which starts the app. Inside dbs/index.js is our database connection logic, and inside routes/index.js is our request routing logic.

We use the index.js file naming convention because it allows parent modules the ability to import the module by its directory name, the index.js file is implicitly found. For example, later you'll see we require the "dbs" module with require('./dbs'), rather than require('./dbs/index.js').

Set Up Database Connections

We'll first start with creating and managing database connections in the dbs/index.js file.

var async = require('async');
var MongoClient = require('mongodb').MongoClient;

// Note: A production application should not expose database credentials in plain text.
// For strategies on handling credentials, visit 12factor: https://12factor.net/config.
var PROD_URI = "mongodb://<dbuser>:<dbpassword>@<host1>:<port1>,<host2>:<port2>/<dbname>?replicaSet=<replicaSetName>";
var MKTG_URI = "mongodb://<dbuser>:<dbpassword>@<host1>:<port1>,<host2>:<port2>/<dbname>?replicaSet=<replicaSetName>";

var databases = {
    production: async.apply(MongoClient.connect, PROD_URI),
    marketing: async.apply(MongoClient.connect, MKTG_URI)
};

module.exports = function(cb) {
    async.parallel(databases, cb);
};


To use the "dbs" module, you might write some code like this:

var initDatabases = require('./dbs);
        initDatabases(function(err, dbs) {....
        });


The "initDatabases" function call initializes database connections and makes the connections accessible via the dbs variable. You could also create a module for other application startup tasks such as connecting to third party APIs or loading data or configuration.

Note: Following 12-factor methodology, rather than storing connection strings in code you should use config variables.

Reuse Database Connections in Routes (or Other App) Files

With the logic written for establishing connections at app startup, we now need to structure the routing logic so their handlers can use those existing connections.

We need a strategy in code to reuse database connections in the "routes" module (routes/index.js file). We can't just call require('./dbs') again in this file, as that would create a second connection pool. We also need a reference to the Express app created in app.js, but we can't use require('../app.js') because app.js requires the "routes" module and circular dependencies should be avoided wherever possible.

Instead, our module will expose a function which takes the Express app and databases as parameters. The parent module that imports this file can then be responsible for finding those values and making them accessible to the module. The technical name for this pattern is "Dependency Injection", as a parent module injects the values that another module depends on. Let's take a look at how it works in practice:

module.exports = function(app, dbs) {

    app.get('/production', function(req, res) {
        dbs.production.collection('test').find({}).toArray(function(err, docs) {
            if (err) {
                console.log(err);
                res.error(err);
            } else {
                res.json(docs);
            }
        });
    });

    app.get('/marketing', function(req, res) {
        dbs.marketing.collection('test').find({}).toArray(function(err, docs) {
            if (err) {
                console.log(err);
                res.error(err);
            } else {
                res.json(docs);
            }
        });
    });

    return app;
};


The route definitions are established in exactly the same way as other Express apps, but the app and the database references are injected by the parent module when the module is loaded.

Tie the "routes" and "dbs" Modules Together in the Main App Module

We can understand how the "routes" and "dbs" modules are tied together by looking at our app.js code. Our app code:

  1. Initializes an Express app
  2. Uses the "dbs" module to create connection pools to our databases
  3. Passes both "dbs" and "app" modules into the "routes" module
  4. Starts the app on port 3000 only after the "routes" module configures the app's route handlers
var express = require('express');
var app = express();

var initializeDatabases = require('./dbs');
var routes = require('./routes');

initializeDatabases(function(err, dbs) {
    if (err) {
        console.error('Failed to make all database connections!');
        console.error(err);
        process.exit(1);
    }

    // Initialize the application once database connections are ready.
    routes(app, dbs);

    app.listen(3000, function() {
        console.log('Listening on port 3000');
    });
});


Using this structure, our app will maintain and reuse a single connection pool to each database. This reduces pressure on the database caused by creating and authenticating new connections or by maintaining multiple unused connection pools. As an added performance and latency benefit, reusing existing connection pools allows the application to quickly retrieve database results without having to go through the connection creation and authentication process each time.

If you have questions or thoughts please email our team at for help!

Database connection Express app application MongoDB

Published at DZone with permission of Chris Chang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Building a Scalable Search Architecture
  • How to Secure Your CI/CD Pipeline
  • Secrets Management
  • Apache Kafka vs. Memphis.dev

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: