The Guts 'n' Glory of Database Internals: Backup, Restore, and the Environment
The series continues with an overview of backing up and restoring your data as well as how to foster a good environment for your system.
Join the DZone community and get the full member experience.Join For Free
A lot of the complexities involved in actually building a database engine aren't related to the core features that you want to have. They are related to what looks like peripheral concerns. Backup/restore is an excellent example of that.
Obviously, you want your database engine to have support for backup and restore. But at the same time, actually implementing that (efficiently and easily) is not something that can be done trivially. Let us consider Redis as a good example. In order to backup its in-memory state, Redis will fork a new process and use the OS' support for copy on write to have a stable snapshot of the in-memory state that it can then write to disk. It is a very elegant solution, with very little code, and it can be done with the assistance of the operation system (almost always a good thing).
It also exposes you to memory bloat if you are backing up to a slow disk (for example, a remote machine) at the same time that you have a lot of incoming writes. Because the OS will create a copy of every memory page that is touched as long as the backup process is running (on its own copy of the data), the amount of memory actually being used is non-trivial. This can lead to swapping and, in certain cases, the OS can decide that it's out of memory and just start killing random processes (most likely the actual Redis server that is being used, as it's the consumer of all this memory).
Another consideration to have is exactly what kind of work do you have to do when you restore the data. Ideally, you want to be up and running as soon as possible. Given database sizes today, even reading the entire file can be prohibitively expensive, so you want to be able to read just enough to start doing the work and then complete the process of finishing up the restore later (while being online). The admin will appreciate it much more than some sort of a spinning circle or a progress bar measuring how long the system is going to be down.
The problem with implementing such features that you need to consider the operating environment in which you are working. The ideal case is if you can control such behaviors. For example, have dedicated backup and restore commands that the admin will use exclusively. But in many cases, you have admins that do all sort of various things — from shutting down the database and zipping into a shared folder on a nightly basis to taking a snapshot, or just running a script with copy/rsync on the files on some schedule.
Some backup products have support for taking a snapshot of the disk state at a particular point in time, but this goes back to the issues we raised in a previous post about low-level hooks. You need to be aware of the relevant costs and implications of those things. In particular, most databases are pretty sensitive to the order in which you back up certain files. If you take a snapshot of the journal file at time T1, but another one of the data file at time T2, you are likely to have data corruption when you restore (the data file contain data that isn't in the journal and there's no way to recover it).
The really bad thing about this is that it is pretty easy for this to mostly work, so even if you have a diligent admin who tests the restore, it might actually work. Then, it will fail when you really need it.
And don't get me started on cloud providers/virtual hosts that offer snapshots. The kind of snapshotting capabilities that a database requires is very specific (all changes that were committed to disk, in the order they were sent, without any future things that might be in flight) in order to be able to successfully backup and restore. From experience, those are not the kind of promises that you get from those tools.
Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.