The backup industry has always had a compelling hypothesis: First, integrate the “control plane” and the “data plane” into a media server and second, optimize the data plane with variable length dedupe in the form of a backup appliance. Presto! Now, you can greatly reduce both capital and operational expenses. This hypothesis has served the traditional vendors well — they are still selling in the billions of dollars! However, the fact of the matter is that they are on the wrong side of technology trends: both in terms of the emergence of the cloud as well as cloud applications and workloads.
Let’s head back a couple of decades when tape was the king of secondary storage. The arguments brought forward against tape were the following: Although much cheaper than disk, the operational complexity was very high. For example, tapes have high seek times compared to disk, cannot handle more than one stream reliably, and have high operational maintenance overheads (e.g, temperature controlled environments). As a result, tape never graduated out of the high end enterprise, and did not have an answer to the fundamental advantages of de-duplicated disk: the capital expenditure was comparable and the operational expense was much less. In short order, a combination of Veritas media servers and Data Domain backup appliances became the de-facto standard for backing up Oracle databases.
Fast forward to the present, the same technical trends that killed tape are once again front and center, but they have a new target. The same arguments that were made against tape are now being made against disk: seek times for disks have barely improved in the last decade even though disk throughputs have improved, disks have high operational cost in large data centers because a constantly spinning disk eats up energy even when there is no I/O, disk drive failures and silent data errors are the norm rather than exception.
So what is replacing disk? In short order, it is elastic cloud storage, where there are no storage quotas, bandwidths are sufficient and scalable for all but the most latency sensitive workloads and the capital and operational expense is unmatched. The backup industry has taken the “logical” next step: let us port the backup media server-based software, purpose-built backup filesystems, and the backup appliances to the cloud and the problem is solved! After all, the cloud is just another storage target.
But this “logical” next step is the wrong one as the backup industry is ignoring two important facts.
First, porting the media server to the cloud ignores the basic fact that a cloud is geo-distributed with data flowing in and across the entire continent, if not the world. Having all data flowing in through a single media server is not only a deployment challenge, but also a severe performance bottleneck. The cloud is architected for applications to have high bandwidth access to a scalable object storage backend, not to another server sitting in one geographic location. Some companies are trying to scale out the media server, but they are solving the wrong problem: why try to create a distributed “data plane” (aka, yet more backup/storage filesystems), when the cloud has created one for you?
The similar argument works against porting backup appliances to the cloud. The key value proposition is using variable length de-dupe to reduce storage costs and operational expenses. Not only is there no need to optimize the data plane, but more importantly the era of variable length de-dupe is over! Workloads are changing and the entire spectrum of traditional to next-generation applications is becoming real-time and high throughput. This has led data to become compressed on disk to make load times quicker with the unfortunate side-effect that variable length de-dupe does not work anymore.