DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Java EE 6 Pet Catalog with GlassFish and MySQL
  • Create a Multi-Tenancy Application in Nest.js, Part 4: Authentication and Authorization Setup
  • Getting Started With JPA/Hibernate
  • Application Self-Healing: Common Failures and How to Avoid Them

Trending

  • How to Merge HTML Documents in Java
  • Detection and Mitigation of Lateral Movement in Cloud Networks
  • Creating a Web Project: Caching for Performance Optimization
  • Secrets Sprawl and AI: Why Your Non-Human Identities Need Attention Before You Deploy That LLM
  1. DZone
  2. Data Engineering
  3. Data
  4. Deep Dive into Connection Pooling

Deep Dive into Connection Pooling

By 
Chris Chang user avatar
Chris Chang
·
Nov. 07, 13 · Interview
Likes (2)
Comment
Save
Tweet
Share
24.1K Views

Join the DZone community and get the full member experience.

Join For Free

As your application grows in functionality and/or usage, managing resources becomes increasingly important. Failure to properly utilize connection pooling is one major “gotcha” that we’ve seen greatly impact MongoDB performance and trip up developers of all levels.

Connection Pools

Creating new authenticated connections to the database is expensive. So, instead of creating and destroying connections for each request to the database, you want to re-use existing connections as much as possible. This is where connection pooling comes in.

A Connection Pool is a cache of database connections maintained by your driver so that connections can be re-used when new connections to the database are required. When properly used, connection pools allow you to minimize the frequency and number of new connections to your database.

Connection Churn

Used improperly however, or not at all, your application will likely open and close new database connections too often, resulting in what we call “connection churn”. In a high-throughput application this can result in a constant flood of new connection requests to your database which will adversely affect the performance of your database and your application.

Opening Too Many Connections

Alternately, although less common, is the problem of creating too many MongoClient objects that are never closed.  In this case, instead of churn, you get a steady increase in the number of connections to your database such that you have tens of thousands of connections open when you application could almost certainly due with far fewer. Since each connection takes RAM, you may find yourself wasting a good portion of your memory on connections which will also adversely affect your application’s performance.

Although every application is different and the total number of connections to your database will greatly depend on how many client processes or application servers are connected, in our experience, any connection count great than 1000 – 1500 connections should raise an eyebrow, and most of the time your application will require far fewer than that.

MongoClient and Connection Pooling

Most MongoDB language drivers implement the MongoClient class which, if used properly, will handle connection pooling for you automatically.

The syntax differs per language, but often you do something like this to create a new connection-pool-enabled client to your database:

mongoClient = new MongoClient(URI, connectionOptions);

Here the mongoClient object holds your connection pool, and will give your app connections as needed. You should strive to create this object once as your application initializes and re-use this object throughout your application to talk to your database. The most common connection pooling problem we see results from applications that create a MongoClient object way too often, sometimes on each database request. If you do this you will not be using your connection pool as each MongoClient object maintains a separate pool that is not being reused by your application.

Example with Node.js

Let’s look at a concrete example using the Node.js driver.

Creating new connections to the database using the Node.js driver is done like this:

mongodb.MongoClient.connect(URI, function(err, db) {
  // database operations
});

The syntax for using MongoClient is slightly different here than with other drivers given Node’s single-threaded nature, but the concept is the same. You only want to call ‘connect’ once during your apps initialization phase vs. on each database request.

Let’s take a closer look at the difference between doing the right thing vs. doing the wrong thing.

Note: If you clone the repo from here, the logger will output your logs in your console so you can follow along.

Consider the following examples:

var express = require('express');
var mongodb = require('mongodb');
var app = express();

var MONGODB_URI = 'mongo-uri';

app.get('/', function(req, res) { 
  
  // BAD! Creates a new connection pool for every request
  
  mongodb.MongoClient.connect(MONGODB_URI, function(err, db) {
  if(err) throw err;
  
  var coll = db.collection('test');

    coll.find({}, function(err, docs) {
      docs.each(function(err, doc) {
        if(doc) {
          res.write(JSON.stringify(doc) + "\n");
        }
        else {
          res.end();
        }
      });
    });
  });  
});

// App may initialize before DB connection is ready

app.listen(3000);
console.log('Listening on port 3000');

The first (no pooling):

  • calls connect() in every request handler
  • establishes new connections for every request (connection churn)
  • initializes the app (app.listen()) before database connections are made
  • var express = require('express');
    var mongodb = require('mongodb');
    var app = express();
    
    var MONGODB_URI = 'mongodb-uri';
    var db;
    var coll;
    
    // Initialize connection once
    
    mongodb.MongoClient.connect(MONGODB_URI, function(err, database) {
      if(err) throw err;
     
      db = database;
      coll = db.collection('test');
    
      app.listen(3000);
      console.log('Listening on port 3000');
    });
    
    // Reuse database/collection object 
    
    app.get('/', function(req, res) { 
      coll.find({}, function(err, docs) {
        docs.each(function(err, doc) {
          if(doc) {
            res.write(JSON.stringify(doc) + "\n");
          }
          else {
            res.end();
          }
        });
      });
    });
      

The second (with pooling):

  • calls connect() once
  • reuses the database/collection variable (reuses existing connections) waits to initialize the app until after the database connection is established

If you run the first example and refresh your browser enough times, you’ll quickly see that your MongoDB has a hard time handling the flood of connections and will terminate.

Further Consideration – Connection Pool Size

Most MongoDB drivers support a parameter that sets the max number of connections (pool size) available to your application. The connection pool size can be thought of as the max number of concurrent requests that your driver can service. The default pool size varies from driver to driver, e.g. for Node it is 5, whereas for Python it is 100. If you anticipate your application receiving many concurrent or long-running requests, we recommend increasing your pool size- adjust accordingly!



Database connection application

Published at DZone with permission of Chris Chang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Java EE 6 Pet Catalog with GlassFish and MySQL
  • Create a Multi-Tenancy Application in Nest.js, Part 4: Authentication and Authorization Setup
  • Getting Started With JPA/Hibernate
  • Application Self-Healing: Common Failures and How to Avoid Them

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!