Diary of a Cloud Backup – part 1
Diary of a Cloud Backup – part 1
Join the DZone community and get the full member experience.Join For Free
Learn how to migrate and modernize stateless applications and run them in a Kubernetes cluster.
I have automated home backups onto a off-line disk. A combination of windows disk mirroring and Linux or Mac scheduled scripts keeps it both simple and automated. I just have to remember to transfer to disk every now and then for longer term snapshots. If you have got this far with your backups your doing well, but you have probably realised that without an off site backup your still vulnerable to theft and fire! Well lets hope for none of these things actually happen but if your backed up its one less worry!
There are now many companies providing business and home users with off site backup services. In theory its a great idea to use your broadband connection to upload the files somewhere safe. There are some issues though. Firstly for most of us in the UK ADSL is the only option. I have cable up to the house, but the fiber services have a terrible reputation.
ADSL is rate adaptive and asymmetric. Asymmetric means the upload speed is slower than the download. Adaptive means that speeds will be slower if your line is of bad quality or long. If you are more than a kilometer from the exchange the speed will drop off, but even if your closer poor line quality will drop the speed down. ADSL also comes in various flavors, ADSL, ADSL Max, ADSL 2+ to name a few. These all have their own characteristics and different speeds.
There are some things you can do to address line quality. Even changes at home like putting the router near to the phone link and improving the internal networking can make small differences. Ultimately you might need to get your ISP to work on BT to fix the line issues but as this shows that is not always easy.
Then on top of that, even if you have top quality and max speed, your ISP may cap your usage and shape your traffic to prevent you from effecting other users.
The problem I have to solve in creating an off-line backup is getting around 100Gb uploaded. If you have a small amount (less than 5Gb) it will be easy. Also there are free services for small amounts. The asymmetric nature of ADSL means that your backup is mainly restricted by up-link speeds. So lets look at some typical speeds:
|ADSL Type||Uplink speed||Gb/Day||Days to upload 100Gb|
So for 36 days probably more I will be backing up, and that also means my normal internet will be slow. This is because without spare up link capacity requests will be slow even if the download stream is not used.
As I write this I have a 0.26mbps up-link so the best estimate is 36 days to get the initial backup uploaded. But what about subsequent backups. Choose a company that gives you a system that only uploads changed files and is clever enough to only store the changes. This makes it workable because even on ordinary ADSL your subsequent backups are manageable. Some service providers make it possible to seed the initial backup using disks and post. This costs more and I am not sure how long the data would take to arrive in Italy even then!
What about the restore? Well this is less of a problem. For starters download is much faster (I hope your on an uncapped deal!) Some companies also offer a service to send you the data on a disk. I am sure I would pay for this if I had my hardware stolen.
What about switching to another supplier? Some suppliers now make it possible to switch without a huge upload again. They will take the initial data direct from your current supplier. This happens at much faster speeds than you can achieve from home.
This is by no means an extensive review of the available services. Prices change all the time. Many home services only support windows, so that rules out quite a few. Some are just storage on the cloud, and don’t provide a backup tool however simple. Others will be fine for a measly 2Gb but I have 100Gb to store and that gets costly very quickly. In the end at the time of writing only one company provided the combination I wanted: www.memopal.com. I bought a 5 year plan for £130.
This buys me 200Gb which is more than enough, I can purchase more if I require. The file system they use makes sure the data is on more than one disk. In addition they store versions. So if I overwrite something I can get back the original. I use svn source control for documents but images are stored in the raw state so its a good idea for those. In addition they have this clever feature that if your file matches one uploaded by another user, you are linked to that, meaning it does not count towards your storage. This is handy if you upload executable files as its possible someone else already has that file somewhere.
After calculating the above I decided I needed a new ISP. After much searching I decided on BE. Funnily enough that actually saved me money on the ADSL service and gave me ADSL 2+ with 1.3mbps up link speed.
I needed a new router though, and wanted an excellent one, so opted not to use the one provided by BE. The BE one is pretty good actually but its just not anywhere near as good as a DrayTek 2820. This set me back £147. A bit of a luxury perhaps but it does support cable modems should I go in that direction later.
Whilst the backup is driving this it also hugely improves my ADSL and home networking so I was happy to upgrade.
Diary – the first 16 days, 8.8Gb uploaded
I started this with my existing service on standard ADSL. Most of my data is on a windows XP machine so I will start with that. The theory said it would be slow but in addition to that my ADSL was unreliable. After signing up and downloading the memopal backup application things went well for a couple of days. Then I started to get blocks of many hours where the application would not upload files.
Contacted memopal support on day 12 sending them log output from the backup application.
To improve reliability I re-networked at home. Bought a DrayTek 2820. A non WiFi one, I have an access point for that. Plugged it in near the phone socket and experimented with different ways of connecting using the speedtester at thinkbroadband. Oddly connecting the Wan link via a surge protector was both more reliable and faster. Go figure! Noise on the line possibly?
My line is faster and more reliable (less faults) but I am still getting problems. I have managed to upload all the small files now. The big ones all fail after uploading. There is a file CRC check and that keeps failing. This indicates the file arrives at the other end corrupt. Or perhaps there is a bug in the backup software or service at memopal.
Memopal support got back to me and requested I run some tests using some web links. I sent them the resulting numbers. Directly after this the backup started working again and the following day, they confirmed that they had sorted out the issue. Not sure what that issue was though. Thanks to memopal for an excellent response.
Ordered an ADSL 2+, 1.3Mbps service from BE. Pipex are just not cutting it this service has better support, better speed and its cheaper to the tune of £3 per month. Its going to take awhile to recoup the other costs mind. During the next few days they sent over kit and instructions. I got messages delivered to my mobile at each stage. My new line will be installed in 7 days time. Very impressed.
Day 20 – 16.9Gb uploaded
At this stage with maximum speed and no outages I would be up to 56Gb. The reality is far short of that. Only very occasional outages now. Usually just resulting in 10 to 15 minute loss in service in a few days. This could be anywhere along the link between the UK and Italy or the service at memopal. I was never around to investigate. Encouragingly the backup software just deals with it and carries on when the service resumes. Looking forward to being able to shut down the machine, the fans are driving me nuts. I have two machines in the room and the Windows one is noisy.
Day 21 – 19.6Gb uploaded
The upload is still progressing well. The next milestone is likly to be when I switch from Pipex to BE, 0.62mbps to 1.3mbps in a few days time.
Published at DZone with permission of Martin Harris , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.