DZone
Web Dev Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Web Dev Zone > Backing up MongoDB Instances on EBS

Backing up MongoDB Instances on EBS

Daniel Gottlieb user avatar by
Daniel Gottlieb
·
Nov. 28, 11 · Web Dev Zone · Interview
Like (0)
Save
Tweet
4.07K Views

Join the DZone community and get the full member experience.

Join For Free

Even with replica sets and journaling, it’s still a good idea to regularly back up your data. I’m actually frequently surprised to hear from relatively sophisticated services that don’t have regularly scheduled backups (both MongoDB users & others). I thought it’d be helpful to give a quick overview of our (simple) backup infrastructure at Fiesta; if anybody else wants to chime in about how they’re handling this stuff we’d love to hear from you!

Architecture

We run a single replica set on Amazon EC2. We have a passive node that we use for all of our backups. Since we’re on EBS, we can use Amazon’s snapshotting to take the actual backup, which will then be hosted on S3. All of our data is split between MongoDB and S3, so the focus of our backups is just MongoDB data.

Journaling

In 1.8, MongoDB introduced journaling. If journaling is enabled, it’s possible to take hot snapshots of a MongoDB data directory. We don’t currently run with journaling, so we need to be a bit more careful. The approach we take is to fsync and lock the passive node and then take the snapshot from there. That’s all handled by the code below.

The code

The following code is what actually handles taking a backup. The lock_and_backup() function gets run from what’s essentially a nightly cron job.

import subprocess

from pymongo import Connection
import settings


def do_ebs_backup():
    env = {"EC2_HOME": settings.ec2_home,
           "JAVA_HOME": settings.java_home}
    keyfile = EC2_KEY
    certfile = EC2_CERT
    args = ["-K", keyfile, "-C", certfile]

    # We use the 'backup' tag to find the volumes to back up
    out = subprocess.Popen([settings.ec2_tools + "bin/ec2-describe-volumes",
                            "-F", "tag:env=backup"] + args,
                           stdout=subprocess.PIPE,
                           env=env).communicate()[0]
    volumes = set([v for v in out.split() if v.startswith("vol-")])
    if not volumes:
        raise Exception("No volumes to backup?")

    # Create a snapshot for each volume we found above
    snaps = []
    for volume in volumes:
        snap = subprocess.Popen([settings.ec2_tools + "bin/ec2-create-snapshot"] + \
                                     args + [volume],
                                stdout=subprocess.PIPE,
                                env=env).communicate()[0]
        snaps.append(snap)
    return snaps


def lock_and_backup():
    conn = Connection(slave_okay=True)
    try:
        conn.admin.command("fsync", lock=True)
        do_ebs_backup()
    finally:
        conn.admin["$cmd.sys.unlock"].find_one()

We use the EC2 command-line tools to do the snapshot, and use Python’s subprocess module to interact with those tools. The important bit really happens in lock_and_backup(), where we run the “fsync” command and write-lock the server before actually taking the snapshot.


source: http://blog.fiesta.cc/post/13127613694/backing-up-mongodb-instances-on-ebs

MongoDB

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Open Source Monitoring and Metrics Landscape
  • Spring, IoC Containers, and Static Code: Design Principles
  • Progressive Web Apps vs Native Apps: Differences and Similarities
  • Migrating Legacy Applications and Services to Low Code

Comments

Web Dev Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo