DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modern Digital Website Security: Prepare to face any form of malicious web activity and enable your sites to optimally serve your customers.

Containers Trend Report: Explore the current state of containers, containerization strategies, and modernizing architecture.

Low-Code Development: Learn the concepts of low code, features + use cases for professional devs, and the low-code implementation process.

E-Commerce Development Essentials: Considering starting or working on an e-commerce business? Learn how to create a backend that scales.

Related

  • Want To Build Successful Data Products? Start With Ingestion and Integration
  • Powering Enterprise Applications: Unleashing the Java-MongoDB Synergy
  • Java and MongoDB Integration: A CRUD Tutorial [Video Tutorial]
  • Stateless vs. Stateful Widgets: Make the Right Choice for Your Flutter App

Trending

  • Decompose Legacy System Into Microservices: Part 2
  • How To Use KubeDB and Postgres Sidecar for Database Integrations in Kubernetes
  • CodeCraft: Agile Strategies for Crafting Exemplary Software
  • Evolution of Software Architecture: From Monoliths to Microservices and Beyond
  1. DZone
  2. Data Engineering
  3. Databases
  4. How to Stop a Runaway Index Build in MongoDB

How to Stop a Runaway Index Build in MongoDB

Oh no! You've triggered your index build on a large collection and now your cluster is unresponsive. No matter what you do, you can't stop it. What to do?!

Dharshan Rangegowda user avatar by
Dharshan Rangegowda
·
Sep. 22, 17 · Tutorial
Like (4)
Save
Tweet
Share
10.6K Views

Join the DZone community and get the full member experience.

Join For Free

Index builds in MongoDB can have an adverse impact on the availability of your MongoDB cluster. If you trigger a foreground index build on a large collection on your production server, you may find that your cluster is unresponsive until the index build is complete. On a large collection, this could take several hours or days, as described in the perils of index building in MongoDB.

The recommended best practice is to trigger index builds in the background. However, on large collection indexes, we've seen multiple problems with this approach. In the case of a three-node cluster, both secondaries start building the index and stop responding to any requests. Consequently, the primary does not have quorum and moves to the secondary state, taking your cluster down. Also, the default index builds triggered from the command line are foreground index builds — making this a now widespread problem. In future releases, we're hopeful that this becomes background by default.

Once you've triggered an index, simply restarting the server does not solve our problem. MongoDB will pick up the index build from where it left off. If you were running a background index build previously after the restart, it now becomes a foreground index build, so in this case, the restart could make the problem worse.

If you've already triggered an index build, how do you stop it? Luckily, it's relatively easy to stop an index build.

Option 1: Kill the Index Build Process

Locate the index build process using db.currentOp() and then kill the operation using db.killOp(<opid>). The index operation will look something like this:

{
"opid" : 820659355,
"active" : true,
"lockType" : "write",
....
"op" : "insert",
"ns" : "xxxx",
"query" : {
},
"client" : "xxxx",
"desc" : "conn",
"msg" : "index: (2/3) btree bottom up 292168587/398486401 64%"
}

If the node where the index is building does not respond to new connections or the killOp does not work, use Option 2 below.

Option 2: Configuring noIndexBuildRetry and Restart

MongoDB provides a -noIndexBuildRetry option, which instructs MongoDB to stop building incomplete indexes on restart.

This parameter doesn't appear to be supported by the config file, only as a parameter for the mongod process. We don't prefer to run mongod manually with this option because if you accidentally run the mongod process as an elevated user (i.e. root), it ends up changing the permissions of all the files. Also, once run as root, we've had intermittent problems running the process as mongod again.

A simpler option is to edit the /etc/init.d/mongod file. Looks for this line:

OPTIONS=" -f $CONFIGFILE"

Replace with this line:

OPTIONS=" -f $CONFIGFILE --noIndexBuildRetry"

Detailed Steps

For the purposes of this discussion, we're providing instructions for CentOS/RedHat/Amazon Linux.

  1. Configure -noIndexBuildRetry. Add the -noIndexBuildRetry  option to all your data nodes, as explained above.
  2. Restart all the nodes building the index. Look at the mongod log file for each data server and determine if it's building the index. If it is, then restart the server using service mongod restart.
  3. Drop the incomplete index. Once all of the relevant nodes are restarted, look at the list of indexes and drop the incomplete index if you see it on the list.
  4. Remove -noIndexBuildRetry. Edit the /etc/init.d/mongod file to remove the -noIndexBuildRetry option that you added in Step 1 so we can revert back to the default behavior of resuming the index build.
Build (game engine) MongoDB

Published at DZone with permission of Dharshan Rangegowda, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Want To Build Successful Data Products? Start With Ingestion and Integration
  • Powering Enterprise Applications: Unleashing the Java-MongoDB Synergy
  • Java and MongoDB Integration: A CRUD Tutorial [Video Tutorial]
  • Stateless vs. Stateful Widgets: Make the Right Choice for Your Flutter App

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: