DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Loading XML into MongoDB
  • MLOps: How to Build a Toolkit to Boost AI Project Performance
  • Mastering Full-Stack Development: A Comprehensive Beginner’s Guide to the MERN Stack
  • Snowflake Empowers Developers to Easily Build Data-Driven Apps and Chatbots

Trending

  • A Guide to Developing Large Language Models Part 1: Pretraining
  • Beyond Linguistics: Real-Time Domain Event Mapping with WebSocket and Spring Boot
  • Kubeflow: Driving Scalable and Intelligent Machine Learning Systems
  • Mastering Fluent Bit: Installing and Configuring Fluent Bit on Kubernetes (Part 3)
  1. DZone
  2. Data Engineering
  3. Databases
  4. How to Stop a Runaway Index Build in MongoDB

How to Stop a Runaway Index Build in MongoDB

Oh no! You've triggered your index build on a large collection and now your cluster is unresponsive. No matter what you do, you can't stop it. What to do?!

By 
Dharshan Rangegowda user avatar
Dharshan Rangegowda
·
Sep. 22, 17 · Tutorial
Likes (4)
Comment
Save
Tweet
Share
11.3K Views

Join the DZone community and get the full member experience.

Join For Free

Index builds in MongoDB can have an adverse impact on the availability of your MongoDB cluster. If you trigger a foreground index build on a large collection on your production server, you may find that your cluster is unresponsive until the index build is complete. On a large collection, this could take several hours or days, as described in the perils of index building in MongoDB.

The recommended best practice is to trigger index builds in the background. However, on large collection indexes, we've seen multiple problems with this approach. In the case of a three-node cluster, both secondaries start building the index and stop responding to any requests. Consequently, the primary does not have quorum and moves to the secondary state, taking your cluster down. Also, the default index builds triggered from the command line are foreground index builds — making this a now widespread problem. In future releases, we're hopeful that this becomes background by default.

Once you've triggered an index, simply restarting the server does not solve our problem. MongoDB will pick up the index build from where it left off. If you were running a background index build previously after the restart, it now becomes a foreground index build, so in this case, the restart could make the problem worse.

If you've already triggered an index build, how do you stop it? Luckily, it's relatively easy to stop an index build.

Option 1: Kill the Index Build Process

Locate the index build process using db.currentOp() and then kill the operation using db.killOp(<opid>). The index operation will look something like this:

{
"opid" : 820659355,
"active" : true,
"lockType" : "write",
....
"op" : "insert",
"ns" : "xxxx",
"query" : {
},
"client" : "xxxx",
"desc" : "conn",
"msg" : "index: (2/3) btree bottom up 292168587/398486401 64%"
}

If the node where the index is building does not respond to new connections or the killOp does not work, use Option 2 below.

Option 2: Configuring noIndexBuildRetry and Restart

MongoDB provides a -noIndexBuildRetry option, which instructs MongoDB to stop building incomplete indexes on restart.

This parameter doesn't appear to be supported by the config file, only as a parameter for the mongod process. We don't prefer to run mongod manually with this option because if you accidentally run the mongod process as an elevated user (i.e. root), it ends up changing the permissions of all the files. Also, once run as root, we've had intermittent problems running the process as mongod again.

A simpler option is to edit the /etc/init.d/mongod file. Looks for this line:

OPTIONS=" -f $CONFIGFILE"

Replace with this line:

OPTIONS=" -f $CONFIGFILE --noIndexBuildRetry"

Detailed Steps

For the purposes of this discussion, we're providing instructions for CentOS/RedHat/Amazon Linux.

  1. Configure -noIndexBuildRetry. Add the -noIndexBuildRetry  option to all your data nodes, as explained above.
  2. Restart all the nodes building the index. Look at the mongod log file for each data server and determine if it's building the index. If it is, then restart the server using service mongod restart.
  3. Drop the incomplete index. Once all of the relevant nodes are restarted, look at the list of indexes and drop the incomplete index if you see it on the list.
  4. Remove -noIndexBuildRetry. Edit the /etc/init.d/mongod file to remove the -noIndexBuildRetry option that you added in Step 1 so we can revert back to the default behavior of resuming the index build.
Build (game engine) MongoDB

Published at DZone with permission of Dharshan Rangegowda, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Loading XML into MongoDB
  • MLOps: How to Build a Toolkit to Boost AI Project Performance
  • Mastering Full-Stack Development: A Comprehensive Beginner’s Guide to the MERN Stack
  • Snowflake Empowers Developers to Easily Build Data-Driven Apps and Chatbots

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!