In this post, we'll examine a couple of ways for upgrading MongoDB replica set.
With the release of MongoDB 3.2 comes a rash of new features and improvements. One of these enhancements is improved replica sets. From MongoDB: "A replica set in MongoDB is a group of Mongod processes that maintain the same data set. Replica sets provide redundancy and high availability and are the basis for all production deployments."
Config servers are replica sets!
This is HUGE. It signals a significant advancement in backups, metadata stability, and overall maturity. It is a very long-awaited feature that shows MongoDB is maturing. It means:
- Mongos can retry connection vs. error
- Unified and consistent backups!
- Up to 50 secondaries
- Remove bugs with Mongos not near config servers!
How do we activate all these new awesome features? Let's do it!
Upgrading to 3.2
- Replace binaries and restart one secondary at a time.
- Then primaries as well.
- Restart configs in reverse order.
- If configdb=con1, con2, con3
- Restart con3, con2, and then finally con1 with 3.2.
- Do con1 as FAST as possible, while the balancer is also disabled.
- You no longer need to restart a Mongos –upgrade (as of 3.2)
- Restart all Mongos, this will reset ALL connections at some point (whether you do at once or space it out).
- If configdb=con1, con2, con3
Upgrading the replset to the new protocol
This is by far the easiest upgrade bit but DON'T do it until you know your stable on 3.2. Log into each primary and run:
x=newMongo(shard.host);/* Assumes no auth needed */
The quick upgrade scripts
Does what it says: kills every process and launches them on 3.2 binaries with no other changes.
Simply runs the quick rs.reconfig() on each primary, adds the new settings to enable to new replication features.
Let's upgrade the configs the right way!
This is not included as part of a normal upgrade so only do this AFTER you're stable and don't do it before upgrading the protocolVersion we just talked about. (I mean it! Disregard this advice and your life will seriously not be awesome!)
Upgrading to a Config ReplicaSet ( the official way)
- Run rs.initiate on the first config in the list (must be 3.2.4+).
- Must be a fully configured document with configsrv:true defined.
- Restart same config server adding:
- configsvrMode = sccc
- replSet = <name used in rs.initiate()>
- storageEngine= WiredTiger
- Start the new config servers for the other two nodes (should be a new dbpath and port).
- Add those nodes to the replSet and check their status.
- Remove the second original config server from the running.
- Restart the 1st node you set "sccc" on to not have that setting.
- At this point, the first node will transition to removed if using MMAP.
- Restart a Mongos with a new configdb line:
- –configdb <replSetName>/node1:port,node2:port,…
- Only replset members should be listed
- Verify you can work and query through Mongos.
- Repeat on all Mongos.
- Remove the 1st node with rs.remove.
- Shutdown final original config and enable balancer.
There is also an easy way.
The easy way, with a small maintenance window, which lets you just restore a good backup and have a nice and simple rollback plan:
- Stop all Mongos after backing up the config directory.
- Run rs.initiate on first config server.
- Stop the second, then the third, restarting them with an empty dbpath directory.
- Check the rs.status now.
- Stop the first config server and restart with an empty dbpath directory.
- Check status.
- Restart all Mongos, adding <replSetName>/ to the front of the configdb line.
Oh look, here's a quick script we have made for you:
- Kill all config and Mongos processes.
- Restart the first config server on non-standard port.
- Mongodump config database.
- Restart c1 as WiredTiger, clearing that data path.
- Import dump back into first config server.
- Restart on normal port.
- Initialize Replica Set.
- Restart second and third config server after clearing dbpath folder.
- After the initial sync, start all theMongos.
- Done and script exits!