Deploying MongoDB for High Availability

This was a live blog for the recent MongoSV conference.

Going to go over HA best practices: keeping data online and safe.

What about a single node? This will have downtime. If the node crashes, intervention might be necessary. If it disappears will need a backup.

Replica set v1: Single datacenter, single switch, single power source. But automatically recovers from a single node crash. A good start but not great.

The next step up is still single datacenter, but w/ multiple power/network zones. Like EC2 in a single region but multiple AZs. Still some points of failure (datacenter / two node failure). With an arbiter, we can’t do w=2 writes and remain up. With 3 non-arbiters we can use w=2 but are still vulnerable to datacenter failure.

The next step up is multi datacenter w/ a single DR (disaster recovery) node in a different DC. We can’t always stay up, but we at least have a DR option now.

Now let’s look at the ideal: three datacenters, five nodes. One has a single delayed slave (that helps recover from fat-finger incidents like accidental db.drop). The other two DCs each have 2 active nodes. We can lose an entire DC and still have a majority w/ the other two DCs. Can do `w={dc: 2}` to guarantee write in 2 DCs.

Moving on to HA sharding

Each shard needs to be a replica set - same rules apply as above. Balancing can be run in a window (this is cool!) can set an activeWindow to only run the balancer at night. Sweet!

Config servers need to be on at least 2 different power/network zones. Ideally just put in three separate DCs. Use host names rather than IP addresses: much easier to move a config server. Take backups of config servers. Important note (saw this earlier too in Richard’s talk): not a replica set. To bring new nodes online you need to manually move the data. Not a problem if a config server is down for a day or something: just won’t do splits/migrates.

Run one mongos per app-server. Don’t need to worry about scaling mongos. Saves a network hop for many ops. If you really don’t like this, run a pool per power region with a load-balancer in front.

Application-level tips:

Handle spikes: queue non-synchronous writes, isolate components and features. Can your site handle going into a read-only mode? That helps a lot when dealing w/ issues.

Monitor! Load, disk, CPU, but most importantly I/O (iostat). Alerts go hand in hand w/ monitoring.Have good procedures for backups, adding replica set members, adding shards, etc. Practice (in staging) (which ought to be the same as prod). Randomly shut down boxes & load test as much as possible.

Recap

That’s all for today. Another great event by 10gen; I think my favorite talks of the day were Kyle’s and Richard’s, but all of them were great. Hope you enjoyed the blog posts!

Source: http://blog.fiesta.cc/post/13989346153/deploying-mongodb-for-high-availability-mongosv