Deploying MongoDB for High Availability
Join the DZone community and get the full member experience.
Join For FreeThis was a live blog for the recent MongoSV conference.
Going to go over HA best practices: keeping data online and safe.
What about a single node? This will have downtime. If the node
crashes, intervention might be necessary. If it disappears will need a
backup.
Replica set v1: Single datacenter, single switch, single power
source. But automatically recovers from a single node crash. A good
start but not great.
The next step up is still single datacenter, but w/ multiple
power/network zones. Like EC2 in a single region but multiple AZs. Still
some points of failure (datacenter / two node failure). With an
arbiter, we can’t do w=2 writes and remain up. With 3 non-arbiters we
can use w=2 but are still vulnerable to datacenter failure.
The next step up is multi datacenter w/ a single DR (disaster
recovery) node in a different DC. We can’t always stay up, but we at
least have a DR option now.
Now let’s look at the ideal: three datacenters, five nodes. One has a
single delayed slave (that helps recover from fat-finger incidents like
accidental db.drop). The other two DCs each have 2 active nodes. We can
lose an entire DC and still have a majority w/ the other two DCs. Can
do `w={dc: 2}` to guarantee write in 2 DCs.
Moving on to HA sharding
Each shard needs to be a replica set - same rules apply as above.
Balancing can be run in a window (this is cool!) can set an activeWindow
to only run the balancer at night. Sweet!
Config servers need to be on at least 2 different power/network
zones. Ideally just put in three separate DCs. Use host names rather
than IP addresses: much easier to move a config server. Take backups of
config servers. Important note (saw this earlier too in Richard’s talk):
not a replica set. To bring new nodes online you need to manually move
the data. Not a problem if a config server is down for a day or
something: just won’t do splits/migrates.
Run one mongos per app-server. Don’t need to worry about scaling
mongos. Saves a network hop for many ops. If you really don’t like this,
run a pool per power region with a load-balancer in front.
Application-level tips:
Handle spikes: queue non-synchronous writes, isolate components and
features. Can your site handle going into a read-only mode? That helps a
lot when dealing w/ issues.
Monitor! Load, disk, CPU, but most importantly I/O (iostat). Alerts go hand in hand w/ monitoring.Have good procedures for backups, adding replica set members, adding
shards, etc. Practice (in staging) (which ought to be the same as prod).
Randomly shut down boxes & load test as much as possible.
Recap
That’s all for today. Another great event by 10gen; I think my favorite talks of the day were Kyle’s and Richard’s, but all of them were great. Hope you enjoyed the blog posts!
Source: http://blog.fiesta.cc/post/13989346153/deploying-mongodb-for-high-availability-mongosv
Opinions expressed by DZone contributors are their own.
Comments