As a short addendum to yesterdays hard drive rant I wanted to talk a tiny bit about offsite backup. RAID 1, and physical prints, and the original negatives are all well and good until your house or, in my case, apartment burns down. I'd rather not lose everything I've ever shot, so I've been looking for off site backup solutions.
Of course, I'm not going to backup 500GB of data over the net, but I can save full resolution jpegs of my final work as a last-ditch lifeboat. So when I'm done working on an image, I've started saving these jpegs into a folder called 'Ark', as in Noah's. Then it's just a matter of where to put them.
I could just buy more space on my webserver and ftp them up there, but that seemed clunky.
For a while during the 365 portraits project I was using Mozy to back-up final jpegs just in case. You install this little app that runs in the background and keeps an eye on certain folders you designate, uploading any changes to the mozy servers. This system is great and at $5 a month for unlimited space, seemed pretty good. But I decided to do a little more research...
And I found an app called Jungle Disk, which is a front-end for Amazon's S3 (simple storage service). Now, S3 is really designed for developers and such, but Jungle Disk makes it easy and it works just like Mozy does.
So why is it better? Cost and Longevity. First, you pay for each GB of storage and transfer, but the costs are so low.. 15 cents a GB/mo that unless you're putting TONS of stuff up there, it'll be cheaper than Mozy. I've transfered a few GB up there already and so far my bill is 38 cents.
And I'm sure the Mozy people are secure and what-have-you, but I'd bet on Amazon being around in 10 years more than them.. but maybe that's my own bias. According to what I've read, Amazon keeps redundant copies at multiple data centers, and jungle disk encrypts the data between you and the server, so it's secure. Oh, and they're both cross platform, so the can be used on both Mac and Windows.
I've only been using it for a couple of days, but it looks pretty good so far.
Of course, I'm not going to backup 500GB of data over the net, but I can save full resolution jpegs of my final work as a last-ditch lifeboat. So when I'm done working on an image, I've started saving these jpegs into a folder called 'Ark', as in Noah's. Then it's just a matter of where to put them.
I could just buy more space on my webserver and ftp them up there, but that seemed clunky.
For a while during the 365 portraits project I was using Mozy to back-up final jpegs just in case. You install this little app that runs in the background and keeps an eye on certain folders you designate, uploading any changes to the mozy servers. This system is great and at $5 a month for unlimited space, seemed pretty good. But I decided to do a little more research...
And I found an app called Jungle Disk, which is a front-end for Amazon's S3 (simple storage service). Now, S3 is really designed for developers and such, but Jungle Disk makes it easy and it works just like Mozy does.
So why is it better? Cost and Longevity. First, you pay for each GB of storage and transfer, but the costs are so low.. 15 cents a GB/mo that unless you're putting TONS of stuff up there, it'll be cheaper than Mozy. I've transfered a few GB up there already and so far my bill is 38 cents.
And I'm sure the Mozy people are secure and what-have-you, but I'd bet on Amazon being around in 10 years more than them.. but maybe that's my own bias. According to what I've read, Amazon keeps redundant copies at multiple data centers, and jungle disk encrypts the data between you and the server, so it's secure. Oh, and they're both cross platform, so the can be used on both Mac and Windows.
I've only been using it for a couple of days, but it looks pretty good so far.
Great idea! My question is: how can you identify a certain picture? Does it require a disk catalog program? What do you think?
Both of them setup a network drive on your computer so you can navigate the stuff on the server as it were a drive on your local network.. inside of explorer or finder. It's pretty cool.
Bill, I'm in the same boat you are (on a smaller scale). I have outgrown my internal hard drive (I use a 24" iMac so I can't expand internally.) I'll be interested to see what you end up with. One place you may want to check is the DAM Forum (http://thedambook.com/smf/index.php) which is by the arthur of "The DAM Book - Digital Asset Management for Photographers" by Peter Krogh. Reading his book and forum messages, he seems to favor a JBOD system (Just a Bunch Of Drives) instead of a RAID. I'm not sure which way I will go but please keep us posted on your ultimate choice.
a JBOD sounds like a really bad idea to me.. if one of those fails he loses data.. he must be backing up somewhere else...
Ooo. this thing looks interesting. Kinda like the Sonnet stuff.. I wonder how much it'll cost.
http://www.macpower.com.tw/products/hddmulti/hydra/hydra_s2
I use JungleDisk to back up my photos (while still keeping the originals on external drives at home) and love it. The peace of mind that comes from having an offsite backup is priceless.
Set the automatic backup feature to run so that all new/changed photos are backed up. It has version control, too. And I've even used it to upload photos taken with a friend's camera and then download at home.
Mozy uses Amazon S3 as the backend, too, but just charges a lot more than the one time $20 charge for JungleDisk and the Amazon charges.
listen- you know how low tech I am. however- while I can't help you wih your terabytes or whatever- I keep my actual pieces of paper that i need in an actual fireproof box made of insulated steel. It's not big and i keep it on the ground level on the floor so if my house burns down I have copies of stuff like my SS card and license and passport.
i get that you're looking for a electronic box to store electronic info but my useless 2 cents is to also get a metal box to store stuff in. your photos while important will be less valuable than your personal information when the commies come and raid the city.
you know what i mean.
I tried and it work brilliantly. How do you manage these pictures? You just upload in a certain folder structure or you use some software to tag/group etc. you images?
I'll admit that I'm old school when it comes to organizing.
I don't trust lightroom or any other app to keep me organized at the bottom level. One bad corruption and everything kapooey. Screw that.
So, I have folders on my photo drive. "people", "trips", "events", "client shoots" etc.
Inside of them I have a folder for each person in 'people' for example. So taking yesterdays pics of Lindsay, there is a folder called "Lindsay Goranson" inside of which are folders named by date.. so yesterday's pics are inside "032608".
So, PhotoDrive > PEOPLE > Lindsay Goranson > 032608 > images.
Images are renamed similarly. So one of them might be called
LindsayGoranson_032608_245 and such. I do the renaming in Lightroom. Usually after I've gone through and culled the junk out.
Basically, I like the idea that I'm not relying on the lightroom catalog. So I personally don't tag every photo.
In regards to offsite backup...
I am of a different stripe of end user altogether and my solution of choice may not be for you...it depends entirely on whether or not your life's income is based on photography (mine is not and I don't even take my own advice here for my own images).
There is a significant issue with using mechanical media backups. By mechanical media I mean disk drives of various types. Raidsets can fail in spite of their redundancy. Offsite backups to other peoples servers place your proprietary data in other folks hands as well as most people don't have the type of network connectivity necessary to move terabytes of data from home to elsewhere. I prefer to keep my files RAW (or DNG).
In professional data centers (I run one) we have all the necessary primary and secondary forms of data protection in place. All volumes are Raidsets, critical raidsets are mirrored to remote locations, nightly warm backups are run every evening and once a week a cold backup is performed. Cold meaning all applications are shutdown, i/o quiesced and a backup then performed. Warm backups are done while i/o is still in progress.
Now then...all this is not required for the average home user or pro photographer. Nor is it feasible (remote mirroring for instance would be too costly). However, a subset of these operations might be feasible/desirable.
Are you ready for this??...tape backup. Tape drives were common on home computers 10 years ago but pretty much went away from the home computer market in favor of cheap disk drives. Other reasons that drove this change were the expansion of cheap disk drive capacity (hundreds of GB on one drive vs. home tape drives which could only handle 80-100MB. Too many tapes to perform a single backup and too long).
However, in professional data world tape is always a part of the protection strategy. Tapes offer the ability to easily port the data elsewhere (safe deposit box, your moms attic, a remote desert isle, etc). Todays tapes are high capacity and high speed.
LTO-4 format tapes are a square cartridge apprx 4 inch by 4 inch and one inch tall. They can store 1.6 Terabytes of data and can write data at a rate of 576GB per hour. Now the down side. Cost...
LTO-4 drives of this sort new are about 3200.00. Quite an investment. You might be able to find LTO-2 on the 2nd hand market for significantly cheaper. Alternatively, you could use LTO-2 where the drives are new cost about 1500.00 dollars - again you might save money by going to the secondary market. The diff in performance would be that each tape would carry about 400GB of data and the write rate would be about 174GB per hour. There is of course LTO-3 which would offer a compromise between the two.
Tape offers the ability to move your data around easily. Once it is removed from the tape drive it is immune to electrical surges of any sort. With some common sense it can be protected from most magnetic interferences.
I can write more on this topic if you are interested. It still may not be cost feasible. A lot will depend on how much you are making from photography today, how involved you want to be with the data management,...and then there's the cost of the tapes themselves.
John
...and one more thing. I'm not a salesman of any type. I know this stuff simply because I've been the customer and am responsible for disaster recovery in my own business environment.
Hi John.
Considering the fact that hard drives are so cheap now, is there a big advantage to the tape versus just buying a couple extra drives that you would swap back and forth off-site somewhere?
Also, I doubt that anybody is keeping off-site backups of everything they've got. I just upload high-res jpegs of finished stuff as a "just in case". I agree with you however that using flickr and such for backup is scary because of their usage policies, but straight data backup like S3 keeps things encrypted along the way, you're just renting space. In fact I'd be surprised if even the admins could open your data.
Thanks for the info.
You're question is valid and is a reasonable response to the current state of the industry today. Largely, this is a matter of cost vs. safety and relates to why I disclaimed from the first that my solution is not necessarily for everyone.
What you propose works in theory. However, continually shuttling disk drives between locations is not generally a good idea. External disk drives are not really designed to be moved around a lot. You will decrease the MTBF (mean time between failures) if these electro-mechanical devices are being moved from place to place frequently. Further, disk drives receive wear and tear just by being plugged in. Tapes only receive wear and tear during the actual read/write operations.
Drop a tape on the floor from desktop height and its probably still readable. A one time occurrence of this with a disk drive is likely to leave you in a problem situation. The chances of this occurring are significantly more likely if you're constantly moving the disk device around.
Tapes are significantly more sturdy than disk drives when it comes to transportation. Forget dropping a tape - you can probably throw it across the room and providing the case didn't shatter (which would require a baseball pitch type toss) the tape is most likely still readable.
Tapes represent removable media. If your tape drive fails you get it repaired or replaced and your tape media and data are unaffected by this particular mechanical failure. If your disk drive fails then thats it, you've lost any unique data on the drive.
A bullet pointed list looks like this:
Tapes advantages:
1. Offline tapes isolated from power failures.
2. There are now tapes with 1.6TB capacities
3. Tapes are more easily portable.
4. Tapes are more durable from a being moved/jostled/dropped perspective.
5. Tapes only receive wear and tear when actually being used.
6. Ability to keep RAW files for backup rather than networked hi-res JPEGs.
Disk drive advantages:
1. Requires no special SW beyond the O/S.
2. Easy to use.
3. Cheap...until you lose your data.
4. Receive wear and tear whenever turned on.
5. Ability to keep RAW files.
Let me be clear that I don't recommend tape in place of disk drive storage expansion. Meaning I wouldn't generally speaking backup disk drive to tape and then initialize the drive back to empty to start storing stuff anew. I see tape as a supplement to your online storage. Keep enough drive storage to keep live copies of everything and use tape to get your data offsite.
All that being said. If you were inclined to buy additional drives and felt that the shuttling operation would only need to occur once a month (i.e. that you could stand to lose up to a months worth of data in the event of a true disaster - what we refer to as RPO - recovery point objective) - then perhaps a disk only solution is the way to go.
What I look forward to is the eventual development of a really high capacity DVD burn of some sort. This would be very comforting. The images could be burned, moved anywhere you wanted, and would not be subject to magnetic or electrical disasters. Alas, it is not here...yet. The current 8GB (+compression) capacity on a double layer DVD is just completely inadequate.
BTW, I do admit that JungleDisk product looks very cool. My only concern is that you have to downgrade from raw to jpeg to use it.
John
Briefer this time. I swear.
I'm glad to see your stand on software mirrors. I'm oppposed to software mirrors of any sort. They use up i/o bandwidth and CPU capacity that are better used for applications.
Further, anytime you have an unexpected shutdown (blue screen, system hang, core-dump, etc) you will break your mirrors. When the system comes back up again you'll need to await the reconstruction of the mirrors which will use up even more bandwidth and CPU during the re-mirroring operation. Further, during this period you'll be exposed because all your disk sets will be in a degraded RAID state (i.e. one volume down).
The remirroring process is slow.
There are HW mirroring shelves available where the storage controller and mirroring operations are onboard to the controller and completely independent of the O/S. I don't know pricing offhand though because these days I mostly work with SANs - Storage Area Networks...vaguely related to the NAS devices you referred to earlier.