Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

MongoDB Point-in-Time Backups Made Easy

DZone's Guide to

MongoDB Point-in-Time Backups Made Easy

See how to protect your data on MongoDB with point-in-time backups. This article walks you through the steps to configure and use your backups to ensure your data's safety.

· Database Zone ·
Free Resource

Running out of memory? Learn how Redis Enterprise enables large dataset analysis with the highest throughput and lowest latency while reducing costs over 75%! 

MongoDB point-in-time backupsIn this blog post, we’ll look at MongoDB point-in-time backups — and work with them.

Mongodump is the base logical backup tool included with MongoDB. It takes a full BSON copy of database/collections, and optionally includes a log of changes during the backup used to make it consistent to a point in time. Mongorestore is the tool used to restore logical backups created by Mongodump. I’ll use these tools in the steps in this article to restore backed-up data. This article assumes a mongodump-based backup that was taken consistently with oplog changes (by using the command flag “–oplog”), and the backup is being restored to a MongoDB instance.

In this example, a mongodump backup is gathered and restored for the base collection data, and separately the oplogs/changes necessary to restore the data to a particular point-in-time are collected and applied to this data.

Note: Percona developed a backup tool named mongodb_consistent_backup, which is a wrapper for ‘mongodump’ with added cluster-wide backup consistency. The backups created by mongodb_consistent_backup (in Dump/Mongodump mode) can be restored using the same steps as a regular “mongodump” backup.

Stage 1: Get a Mongodump Backup

Mongodump Command Flags

–host/–port (and –user/–password)

This is required, even if you’re using the default host/port (localhost:27017).  If authorization is enabled, add –user/–password flags also.

–oplog 

Required for any replset member! Causes “mongodump” to capture the oplog change log during the backup for consistent to one point in time.

–gzip

Optional. For mongodump >= 3.2, this enables inline compression on the backup files.

Steps

  1. Get a mongodump backup via (pick one):
    • s/options to take a backup (w/oplog) of the data:
$ mongodump --host localhost --port 27017 --oplog --gzip

2016-08-15T12:32:28.930+0200    writing wikipedia.pages to
2016-08-15T12:32:31.932+0200    [#########...............]  wikipedia.pages  674/1700   (39.6%)
2016-08-15T12:32:34.931+0200    [####################....]  wikipedia.pages  1436/1700  (84.5%)
2016-08-15T12:32:37.509+0200    [########################]  wikipedia.pages  2119/1700  (124.6%)
2016-08-15T12:32:37.510+0200    done dumping wikipedia.pages (2119 documents)
2016-08-15T12:32:37.521+0200    writing captured oplog to
2016-08-15T12:32:37.931+0200    [##......................]  .oplog  44/492   (8.9%)
2016-08-15T12:32:39.648+0200    [########################]  .oplog  504/492  (102.4%)
2016-08-15T12:32:39.648+0200    dumped 504 oplog entries


    • Use the latest daily automatic backup, if it exists.

Stage 2: Restore the Backup Data

Steps

  1. Locate the shard PRIMARY member.
  2. Triple check that you’re restoring the right backup to the right shard/host!
  3. Restore a mongodump-based backup to the PRIMARY node using the steps in this article: Restore a Mongodump Backup.
  4. Check for errors.
  5. Check that all SECONDARY members are in sync with the PRIMARY.

Stage 3: Get Oplogs for Point-in-Time-Recovery

In this stage, we will gather the changes needed to roll the data forward from the time of backup to the time/oplog-position to which we would like to restore.

In this example below, let’s pretend someone accidentally deleted an entire collection at oplog timestamp: “Timestamp(1470923942, 3)” and we want to fix it. If we decrement the Timestamp increment (2nd number) of “Timestamp(1470923942, 3)” we will have the last change before the accidental command, which in this case is: “Timestamp(1470923942, 2)“. Using the timestamp, we can capture and replay the oplogs from when the backup occurred to just before the issue/error.

A start and end timestamp are required to get the oplog data. In all cases, this will need to be gathered manually, case-by-case.

Helper Script

Script Usage

$ ./dump_oplog_range.sh
Usage ./dump_oplog_range.sh: [Start-BSON-Timestamp] [End-BSON-Timestamp] [Extra-Mongodump-Flags (in quotes for multiple)]


Steps

  1. Find the PRIMARY member that contains the oplogs needed for the PITR restore.
  2. Determine the “end” Timestamp() needed to restore to. This oplog time should be before the problem occurred.
  3. Determine the “start” Timestamp() from right before the backup was taken.
    1. This timestamp doesn’t need to be exact, so something like a Timestamp() object equal-to “a few min before the backup started” is fine, but the more accurate you are, the fewer changes you’ll need to re-apply (which saves on restore time).
  4. Use the MongoToolsAndSnippets script: “get_oplog_range.sh” (above in “Helper Script”) to dump the oplog time-ranges you need to restore to your chosen point-in-time. In this example I am gathering the oplog between two point-in-times (also passing in –username/–password flags in quotes the 3rd parameter):
    1. The starting timestamp: the BSON timestamp from before the mongodump backup in “Stage 2: Restore Collection Data” was taken, in this example. “Timestamp(1470923918, 0)” is a time a few seconds before my mongodump was taken (does not need to be exact).
    2. The end timestamp: the end BSON Timestamp to restore to, in this example. “Timestamp(1470923942, 2)” is the last oplog-change BEFORE the problem occurred.
    3. Example:
      $ wget -q https://raw.githubusercontent.com/percona/MongoToolsAndSnippets/master/rdba/dump_oplog_range.sh
      $ bash ./dump_oplog_range.sh 'Timestamp(1470923918, 0)' 'Timestamp(1470923942, 2)' '--username=secret --password=secret --host=mongo01.example.com --port=27024'
      # Dumping oplogs from 'Timestamp(1470923918, 0)' to 'Timestamp(1470923942, 2)'...
      2016-08-12T13:11:17.676+0200    writing local.oplog.rs to stdout
      2016-08-12T13:11:18.120+0200    dumped 22 documents
      # Done!



  5. Double check that it worked by looking for the ‘oplog.bson‘ file and checking that the file has some data in it (168mb in the below example):
$ ls -alh dump/oplog.bson
-rw-rw-r--. 1 tim tim 168M Aug 12 13:11 dump/oplog.bson


Stage 4: Apply Oplogs for Point-in-Time Recovery (PITR)

In this stage, we apply the time-range-based oplogs gathered in Stage 3 to the restored data set to bring it from the time of the backup to a particular point in time before a problem occurred.

Mongorestore Command Flags

–host/–port (and –user/–password)

Required, even if you’re using the default host/port (localhost:27017).  If authorization is enabled, add –user/–password flags also.

–oplogReplay

Required. This is needed to replay the oplogs in this step.

–dir

Required. The path to the mongodump data.

Steps

  1. Copy the “dump” directory containing only the “oplog.bson”. file (captured in Stage 3) to the host that needs the oplog changes applied (the restore host).
  2. Run “mongorestore” on the “dump” directory to replay the oplogs into the instance. Make sure the “dump” dir contains only “oplog.bson”!
  3. $ mongorestore --host localhost --port 27017 --oplogReplay --dir ./dump
    2016-08-12T13:12:28.105+0200    building a list of dbs and collections to restore from dump dir
    2016-08-12T13:12:28.106+0200    replaying oplog
    2016-08-12T13:12:31.109+0200    oplog   80.0 MB
    2016-08-12T13:12:34.109+0200    oplog   143.8 MB
    2016-08-12T13:12:35.501+0200    oplog   167.8 MB
    2016-08-12T13:12:35.501+0200    done
  4. Validate the data was restored with the customer or using any means possible (examples: .count() queries, some random .find() queries, etc.).

Running out of memory? Never run out of memory with Redis Enterprise databaseStart your free trial today.

Topics:
backup ,mongodb ,consistency ,timestamp

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}