Backing Up Couchbase Clusters to Minio
Minio is an open source object storage solution based on the same APIs as Amazon S3. It's a simple solution for backing up and storing data.
Join the DZone community and get the full member experience.Join For Free
When operating a Couchbase Server cluster, or any other kind of service, it is critical to keep backups, whether it be daily, weekly, monthly or something else. However, after you've taken that backup, where does it end up? Saving your cluster data to the server hosting Couchbase probably isn't a good idea because if it fails you'll lose your backup.
There are plenty of remote storage solutions available. For example, you could
rsync the backups to a dedicated backup server or you could upload them to Amazon S3 or Amazon Glacier. The possibilities don't end with those three solutions.
Minio, like other object storage solutions, protects against hardware failure and silent data corruption using erasure code making it a solid solution for backup storage.
Let's take a look at how you might transfer your backups to Minio if you're operating your own distributed object storage. Starting by taking a backup might be a good place to start.
If you're using Couchbase Server Enterprise Edition, you'll want to execute the following:
cbbackupmgr config --archive </path/to/archive/directory> --repo <repo_name> cbbackupmgr backup --cluster http://<node-ip>:8091 --username <username> --password <password> --archive </path/to/archive/directory> --repo <repo_name>
The above commands will configure an archive and backup any particular cluster to that archive. Remember, backups using cbbackupmgr will be incremental, but it shouldn't stop us from backing them up.
If you're not already running Minio server, download the appropriate binary for your platform. You shouldn't be running Minio server on one of your Couchbase Server nodes. From the Terminal, execute the following:
./minio server /path/to/storage/location
Make note of the access key and secret key values because they will be necessary when configuring the Minio client application.
Now you can download the Minio client application to the Couchbase Server node that you wish to run the backups on.
Before you can push your backups, you'll want to configure the client to connect to your Minio server. Execute the following from the Terminal to make this possible:
./mc config host add backups http://<minio-server-ip>:9000 <access_key> <secret_key>
A fresh server install of Minio will have no object buckets. Before the data can be pushed, execute the following to create a new bucket:
./mc mb backups/couchbase
Now there are several ways to push data to this couchbase storage bucket. We can mirror changes to the bucket by doing the following:
./mc mirror /path/to/couchbase/archive backups/couchbase
Some sources online suggest that object storage should be a write once, read many solution. If following this strategy, you probably don't want to mirror your backups since they are incremental and files will change rather than being created. Instead, you can do something like the following:
tar -czf - /path/to/couchbase/archive | ./mc pipe backups/couchbase/archive-`date +%Y-%m-%d`.tar.gz
The above line will TAR the backup archive and pipe it to the Minio server. Each pushed backup will have a date stamp so you can keep a collection of backups. This process can be done on a schedule by creating a cron task on your client operating system.
More information on creating cluster backups of Couchbase can be found in the Couchbase Developer Portal.
Published at DZone with permission of Nic Raboy, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.