Some time ago I thought I had lost a year’s worth of RAW files. I was using Photoshop Bridge which I rarely use (I use Lightroom to manage my photos) and it had inadvertently moved the folder when I thought the folder was deleted. It took a couple of days to find copies of all the supposedly deleted files, only to find a week or so later that they were all still there, just in a different location.
That launched an ongoing effort to come up with a better backup strategy. The solution is still evolving but it’s to a point where it’s worth sharing with you.
Configuration Then
My configuration then was to store my raw files on Western Digital network attached storage (NAS) drives. These are external drives attached to my wireless router. This works really well when you want to share files between your studio computer and your laptop. I had two 300 GB Western Digital World Book NAS drives that had two years worth of RAW files on them.
I also had a couple of external drives attached to my studio computer via USB. I kept my portfolio files on one of them (my portfolio files are the master files and print files of the images on my website, the ones I sell). Here’s a picture.
The files I though I had deleted were on the NAS drives. I tried to use a file recovery program, one you use to recover files that were accidently deleted. But none of the file recovery programs available anywhere work on NAS drives.
I went hunting everywhere for copies of the lost files and, since I had no systematic backup system, had to search all over the place. Fortunately I found them, or at least most of them, and was able to recover them. (It was later that I found out that the missing folder had simply been moved and all the lost files were hiding out there.)
This whole experience got me thinking that I really needed to get serious about backup.
Getting Serious about Backup
By now I’m suspicious of NAS drives. I mean the concept is great but the inability to recover inadvertently deleted files is a big handicap as far as I’m concerned. The WD NAS drives could only be accessed from a computer via the router. They couldn’t be directly attached. Bummer.
Also I wanted to introduce RAID storage into my solution. RAID stands for Redundant Array of Independent Devices and is basically an array of two or more drives that stores the data in such a way that if you loose one of the drives you still haven’t lost any of your data. In other words, the data is stored in a fashion so that it can be recovered if one of the drives fails. In fact, you can pull out a drive while the disk is in operation and not impact your data access at all. It’s kinda like magic.
The RAID device I selected is a Drobo. This is basically a box that contains all the RAID software. You have to add your own internal hard drives to the box. The device I selected has a capacity of four drives. I started out with two 2 TB Western Digital internal hard drives. This gave me 2 TB of storage, enough to get me started. (You only get 2TB of storage even though I added two devices at 2 TB each because of the redundancy thing.)
So now I am protected from a hardware failure. I spent weeks copying files from other sources on to the Drobo. The situation has improved, I’m protected from drive failure, but I still don’t have a backup system. (By the way, since then I’ve added a third 2 TB Western Digital drive to my Drobo and now have 4 TB of usable storage which ought to last for a while.)
Next I needed to address backup, at least for my portfolio files. If I loose my RAW files from a year or two ago it’s not a big loss; I rarely go back and find images from two years ago that I want to work on. But my portfolio files are a different matter. They each have days and even weeks and months of work in them. Losing them would be a disaster.
So to solve this problem I commandeered a 1 TB external drives I had laying around and put it back into service as a backup drive. The next issue to be resolved was the backup software I would use. I started with Genie Backup Manager which is a professional quality backup utility. After playing around with the many options it offers I finally came up with a backup job that did what I wanted it to. Here’s a picture of this new configuration.
Network Attached Storage Again
Ok, the backup issue is resolved. But I don’t have an easy way of sharing files between my laptop and studio computer. I was able to work out an awkward sharing arrangement by turning on the studio computer and sharing the Drobo drive on the network and then I can reach the Drobo from my laptop. But like I said it was awkward – I had to have the studio computer running, something I’d rather not have to do (not very green).
So the search was still on for some form of NAS. But I didn’t want to end up in the same position I was in before where if a file was inadvertently deleted there was no way to recover it. In earlier NAS research I came across a device that I though held a lot of promise. It was a device you hook to your wireless router and then can connect one or more external drives to it. The external drives could be disconnected at any time and connected to my studio or laptop computer. So if a file was deleted I could connected the drive to one of my computers and run the recovery software to get it back.
Two players in this field stood out – the Iomega and PogoPlug. They both looked very promising – just wire the device directly to your wireless router using a network cable, install the driver software and start attaching external drives. I ended up buying the PogoPlug device in spite of its gaudy magenta color because it includes the option of accessing my data via the internet from anywhere in the world. (Western Digital also had that option but the software they use to implement is, well, let me put it this way – it didn’t work well for me.) So I ordered the ‘pretty’ magenta device. And the price is definitely right – about $45. When was the last time you bought something for your computer for just $45 other than maybe a mouse pad?
So now I’m almost done. I set up the PogoPlug, installed the software and connected a couple of my external drives. It worked like a charm. And I can even access my files from my iPhone. Imagine that. I can display PowerPoint presentations on my iPhone. Why anyone would want to do that I really can’t imagine but it can be done.
But with PogoPlug drivers installed the Genie Backup Manager stopped working – on both computers. OK, so PogoPlug broke my backup solution. Back to Google. I was beginning to think that what I really wanted was not a backup utility but a synchronize utility. Synchronize is different from backup in that it looks at both sides, the source and the target, and makes the two of them look the same. In other words, if you have a file on the source that’s not on the target it will copy the file over. So far it’s no different from backup. Where sync is different is that it also works in the opposite direction – when there is a file on the target that is not on the source it gets copied.
The reason sync is so appealing is that I can sync files from my studio computer to the PogoPlug and then sync files from my laptop to the PogoPlug. This in reality will actually sync my studio computer to my laptop.
So I needed some good sync software. Microsoft has a free offering that I started to install but pulled back when I read some sort of dire warning about loosing files. So I’m in the evaluation phase with a commercial product called SyncBackSE. It’s working out and I’ll be buying the license when the evaluation period ends.
So the configuration is looking pretty robust now. Here’s a picture of my configuration as it stands today.
How Is It Working?
It looks good on paper but does it really work? Yes, although it requires a bit of patience. The connection between the PogoPlug drives and my studio computer is not lightning fast but definitely fast enough. It’s no different from having an external drive attached directly through a USB port. Meaning – the sync works well.
SyncBackSE also works well for backing up my portfolio files. In fact, it’s better than Genie Backup Manager although to be fair to the Genie I didn’t have it configured in the most optimum way.
Syncing files from the laptop can be a long, slow process but you really can’t expect any more across a wireless connection. It’s the size of the straw, not the performance of the PogoPlug that makes it slow.
I haven’t tried accessing files on the PogoPlug via the internet yet except for playing around on the iPhone. I’ll do that soon but I already know that speed will be limited by the speed of the internet and I’m not expecting that to be very impressive.
There is one aspect I’m eager to try on my iPhone and that’s to play the music stored on PogoPlug via the iPhone without having to import it into the iPod and iTunes. It looks like that’s going to work well as long as there’s a cellular connection. This means I’ll be able to have a huge library of music on the PogoPlug and listen to it in the car as I travel – again, as long as there’s a cellular connection. And since I listen to classical music my libraries will definitely be huge.
What’s Left to Do?
That remains to be done is develop a backup strategy for the current and prior year’s RAW files. I think PogoPlug and SyncBackSE will do the job just fine. Another alternative will be to use Lightroom’s backup ability to create the backup on PogoPlug. That could work well.
So I’m feeling pretty comfortable with the way this has evolved. I have NAS back without the file recovery concerns. My portfolio files are backed up and it’s easy enough to do that I can back them up every time I work on them. I have a RAID solution where my really important files are stored, giving me protection against a hard drive failure. And I can even access my files via the internet, not only with my laptop but also my iPhone! Cool.
If you have backup experiences of your own please share them with the rest of us. This is but one of many possible solutions and there are others that will work just as well if not better. Don’t’ be shy.
Join me on an upcoming workshop. Click here for more details.
To see more of my photographs click here.
(1819)
I use a $28 software program for the Mac called SuperDuper and regularly back up my entire hard drive to 2 external drives. One is always connected to the computer, the other I keep in my car in case my house burns down when I’m away. Every so often I burn the master files of my new photos onto CDs, which I keep in a bank safe deposit box 200 miles from where I live. No RAID or other complicated arrangements, and since I only have one computer, no syncing between devices. I still use film, so I also keep the transparencies from which I make my scans, even though I expect never to rescan them. I’m not constantly shooting thousands of photos, and I scan only a fraction of the ones I do shoot, so my backup strategy is pretty simple. My philosophy regarding both photography and computer technology is, in fact, to keep it all as simple as possible, using a minimum of equipment. Since entering the world of computers in 1994, I’ve never had a drive failure or lost data.
Me shy? I’ve been meaning to add my 2 cents.
It’s funny how we hear stories and come to completely opposite positions. Having a career in the IT industry, the IT part that deals with data backup and retention, I have always had a keen interest in backups. For years I relied on one or more tape solutions and happily those days are gone. Now everything is to disk.
When I first started using disk I installed 2 extra drives into my computer and would run full backups weekly with incremental backups nightly alternating the drives every other week. This protected me from a full disk failure as well as inadvertently deleting or somehow destroying a particular file. That is as long as I discovered the problem within 2 weeks after which time the old file would have been written over.
Low and behold over time your disk usage grows and grows and you outgrow yourself. Then the question of external storage comes in. I first investigated the Drobo. The Drobo had become popular with photographers so it was worth some research. What I discovered were some horror stories about the proprietary algorithms used in the Drobo and that scared me away. Industry standard RAID technology has been around for decades and is well an industry standard so I decided to stick with what my industry knows best. So what form of RAID. The Drobo has the advantage of being able to continue adding disks up to the capacity of the cabinet and it will utilize different sizes of disks very effectively. But I really wasn’t interested in a spanned data configuration. What interested me was the more expensive RAID 1 solution. RAID 1 gives me 2 complete and independent copies of the same data. If one disk fails the other survives and the rebuild process is simple. If both disk fail then you have the same problem as with other raid technologies.
Well bottom line I went with RAID 1 and an 8 drive external ESATA enclosure. My OS sits on an internal 500 GB RAID 1 config with my data files utilizing 2 external RAID 1 setups, one 1.5tb and the other 1tb. I also have a RAID 0 pair for my Lightroom catalog and temp file storage.
In addition to that I also have offsite storage. I use a Microsoft program called Synctoy. I was introduced to Synctoy by John Shaw. It’s free and simple. Synctoy compared the source and target folders. One of the options is to copy from source to target new, changed and deleted files, this is what I use. No fancy file structures just in the end a duplicate copy of the source file. I’d like to get into the habit of doing this backup every other week or so and to keep 2 copies of each disk but I don’t in either case. But the backup copy lives in a ‘fire proof’ safe bolted to the floor of my house to give it some added protection.
All in all I feel very protected with really no special software involved in the RAID solution, the backup nor the recovery of any file. It doesn’t happen often but I have gone back to a backup and recovered a file.
Whew, I’ll get off the soapbox.
First off I have no need to make my images available to more than just my studio computer so that takes some confusion out of my configuration.
After much trial and error I have settled on an industry standard backup methodology.
Don’t you just hate it when you find some descent software and you’re all ready to download and try it and then the dreaded “files could be lost” words appear.Thanks for the tip on SyncBackSE will give it a trial run and see how it pans out.
i use a software program called Allways Sync to sync my digital directory to 2 external harddrives. one i keep off site, the other is attached via usb to my desk top. it now takes just a few minutes to sych my desktop to the external backup drive. it works very well.