`cp -r folder backup` turns out folder was a symlink. Then I messed up my script and deleted all of the contents. Backup was destroyed with the original since I copied the symlink instead of the directory. Luckily I had just setup a slave server and was able to copy 95% of the files from there.
Recently I did a `rm -rf /directory/` instead of a `rm -rf /directory/directory2`. Once again luckily I had real backups.
Every-time I screw up, or a system has problems (stupid hard drives) the belief that backups the most important part of a system is reinforced. It basically doesn't matter what you do if you have proper backups you can recover.
The catch there is that no backup is truly a backup until it is tested.
Unfortunately, even that is not enough, in the long run.
You have to periodically retest your backups, and transfer them to new media as they age.
It's also a good idea to store backups off-site (preferably in multiple geographically-dispersed locations).
And, it almost goes without saying that the more frequently you do backups, the less data you'll lose when you actually have to restore from them.
Before long, it's a full time job just to keep the backup system humming along smoothly, testing and retesting backups, and transferring them from old media to new.
Of course, this problem gets a lot harder and more time consuming as the quantity of data you need to backup/restore grows.
I keep reading about the crazy amounts of data generated by projects like the LHC, and my mind boggles at what the challenges in doing backups of that amount of data must be like.
`cp -r folder backup` turns out folder was a symlink. Then I messed up my script and deleted all of the contents. Backup was destroyed with the original since I copied the symlink instead of the directory. Luckily I had just setup a slave server and was able to copy 95% of the files from there.
Recently I did a `rm -rf /directory/` instead of a `rm -rf /directory/directory2`. Once again luckily I had real backups.
Every-time I screw up, or a system has problems (stupid hard drives) the belief that backups the most important part of a system is reinforced. It basically doesn't matter what you do if you have proper backups you can recover.
The catch there is that no backup is truly a backup until it is tested.