Finding and Terminating Long-Running Operations in MongoDB

When your MongoDB becomes unresponsive, it’s imperative that you can quickly identify the cause.

Although there can be many reasons for unresponsiveness, we sometimes find that particularly long-running and/or blocking operations (either initiated by a human or an application) are the culprit. Some examples of common operations that can bog down the database are:

• operations on unindexed fields

• index builds

• expensive map-reduce jobs

One way to quickly see if one or more operations are particularly long-running is to use db.currentOp().

Introducing db.currentOp()

The db.currentOp() method reports in-progress operations on your mongod process. In other words, it will return information on all active operations running on your instance. This allows you to quickly identify long-running and/or blocking operations and focus your attention on problematic areas.

Note: On deployments with authentication enabled, db.currentOp() can only be used by an admin database user. If you have a Dedicated plan with us at MongoLab, you can create an admin database user by navigating to your deployment and clicking on the “admin” database under “System Databases”.

Calling db.currentOp()

The raw output that results from calling db.currentOp() from the mongo shell is somewhat hard to read and filter, so we usually wrap it in some code to make the output easier to use.

Our general “go-to” function call for db.currentOp() returns every operation that has been running for longer than 5 seconds:

db.currentOp().inprog.forEach(
  function(op) {
    if(op.secs_running > 5) printjson(op);
  }
)

We recommend using this function because it provides a comprehensive view of all the slow operations on the MongoDB instance and is easily customizable for your use case. You can change the threshold on a case-by-case basis – 5 seconds is a good starting point. You can also filter by other attributes.

If you pass “true” as a parameter to this method – db.currentOp(true) – it will return a more verbose output, including idle connections and system operations.  However, this typically returns operations you are not interested in.

Example db.currentOp() output

Let’s examine the output from a test deployment.

Query:
  {
    "locks": {"^myDB": "R"},
    "ns": "myDB.bar",
    "op": "query",
    "opid": 1349152,
    "query": {"test": 1},
    "secs_running": 15,
    "waitingForLock": true
  }

Update:
  {
    "locks": {
      "^": "w",
      "^local": "W",
      "^myDB": "W"
    },
    "ns": "myDB.bar",
    "op": "update",
    "opid": 1344808,
    "query": {},
    "secs_running": 53,
    "waitingForLock": false
  }

Note: We stripped out some of the output in this example. Visit MongoDB’s documentation on db.currentOp() to see the full output.

Analysis

The example on the left shows an unindexed query that has been running for 15 seconds. While the time required for the query to complete is troublesome, potentially more problematic is its impact the system as a whole, and how it might indirectly slow other operations.

The example on the right shows an update that has been running for 53 seconds, which holds the database write lock. During this time, other read and write operations queue behind the lock.

db.killOp()

Once you’ve examined the output, you may decide to terminate certain operations to help stabilize your instance. To terminate operations, grab the “opid” value and use db.killOp(). Using the update query from above, you would run the following:

> db.killOp(1344808)

You’re all set!

The next time your MongoDB is feeling sluggish remember that db.currentOp() can help you save time finding troublesome database operations. We hope you get back up and running quickly!