Hard Disks. Good for offline storage over the years?

Silencing hard drives, optical drives and other storage devices

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

DG
Posts: 424
Joined: Thu Feb 26, 2004 3:40 pm
Location: EU

Hard Disks. Good for offline storage over the years?

Post by DG » Sat Sep 27, 2008 7:45 am

With hdds going cheaper and bigger by the day, i found myself that it's not a good solution anymore to keep burning dvds with my data (not necessarily backup data) because they occupy more space then a single hdd (the height of a 3.5" hdd equals 14 dvds, or 63GB for single layer dvds, or ~120GB for dual layer dvds). By the time blu-ray writers and discs become affordable, we will have 2-3TB 3.5" hdds. So you get the picture. :)

As i have a lot of data that i must keep for storage (a box :p), i was thinking of bulding a hdd storage space, where i keep my hdds (offline) for when i need them. I will be using a hard drive docking station for easy acces.

My main concern is if a hdd can keep safely the data on it, when not in use for months (or 1 year or so). I've heard that the oil on the bearings might get damaged, if not regulary used. Or the disks might get demagnetized in time (?!). Maybe silly, but i want some expert opinion on this matter.

Thanks.
Last edited by DG on Sat Sep 27, 2008 9:08 am, edited 2 times in total.

replay0
Posts: 56
Joined: Wed Feb 18, 2004 2:59 pm
Contact:

Post by replay0 » Sat Sep 27, 2008 9:03 am

I went from CD, to DVD, to HDD backups years ago. It was becoming too cumbersome to back up 100+ GBs of various data onto dozens of single-layer DVDs. Now, I have an external USB 2.0 500 GB HDD that I plug in when I want to backup data. I've installed a 3.5" HDD removeable rack to an empty 5.25" bay of my case. I duplicate the data I originally backed up onto the external HDD onto these internal 3.5" HDDs and then move these drives offsite (in a secure locker at my work).

I have a total of 2 offsite HDs. When I want to sync offsite HDs with newer data from my external HDD backup, I bring in 1 offsite HD at a time, update it, secure it at work, and bring in the next 1.

DG
Posts: 424
Joined: Thu Feb 26, 2004 3:40 pm
Location: EU

Post by DG » Sat Sep 27, 2008 10:39 am

Thx for the reply, but you didn't answered my question. :D
Can hdds be a reliable data storage medium, if not used a couple of years? For example if i write all my sensitive data on a hdd, and then leave it in the drawer for a couple of years. Will it be functional after these years and my data will be intact? This is an extreme exemple offcourse, but i'm curious to know.

Lawrence Lee
SPCR Reviewer
Posts: 1115
Joined: Fri Mar 04, 2005 9:07 pm
Location: Vancouver

Post by Lawrence Lee » Sat Sep 27, 2008 11:30 am

As long as you keep it within its environmental tolerances, it should be good for years. It's safer than optical media as they degrade whether they're used or not.

jaganath
Posts: 5085
Joined: Tue Sep 20, 2005 6:55 am
Location: UK

Post by jaganath » Sat Sep 27, 2008 12:26 pm

Can hdds be a reliable data storage medium, if not used a couple of years? For example if i write all my sensitive data on a hdd, and then leave it in the drawer for a couple of years. Will it be functional after these years and my data will be intact? This is an extreme exemple offcourse, but i'm curious to know.
the true answer is, nobody knows. like any "failsafe" system, you need multiple points of redundancy. the main point of failure in an HDD is the read/write head, if this fails your data can still be recovered, although at great cost, however if there is a head crash all your data can be lost. really, who has gigabytes and gigabytes of "sensitive" data? most people just have pr0n and downloaded movies, TV shows, etc.

fundamentally, as a storage medium HDDs are not ideal, as they have many moving parts which are prone to failure. however HDD storage is now sufficiently cheap per GB that you can afford to have perhaps several HDDs, all with identical information on; it is very unlikely that they will all fail at exactly the same time (provided they are from different production batches) before you can copy them to whatever storage medium is best in the future.

MoJo
Posts: 773
Joined: Mon Jan 13, 2003 9:20 am
Location: UK

Post by MoJo » Sat Sep 27, 2008 2:45 pm

Aside from the danger of mechanical problems, which should be a avoidable if stored correctly, I think HDDs should be okay.

I have HDDs from the early 90s that still work fine, and some of those went unused for years (maybe a decade or more).

Think about this for a moment though - if it's your family home videos, fair enough, but if it's just downloaded stuff then why bother? Internet connections are getting faster all the time and you can just re-download stuff you want. I stopped hoarding downloads, because now if I want something I can generally download it in less than a day or two at most, even an entire TV series.

DG
Posts: 424
Joined: Thu Feb 26, 2004 3:40 pm
Location: EU

Post by DG » Sat Sep 27, 2008 2:55 pm

Thanks for the replies. It's for my collection of music (only lossless files, ~400MB per album), and i have all sorts of hard to find music there, and it's getting bigger and bigger by the day, so i need to store it someplace (some for backup only, some not). CDs won't do, at hundreds of albums, they take alot of space, some of the CDs i prefer to rip directly to the hdd in lossless format (like *.flac), others i download them from...well, you know. :oops:

Anyway, i prefer to keep them outside my pc, as i don't need it every day. I just copy what i need, when i need, and then keep the HDDs in a safe place.

sjoukew
Posts: 401
Joined: Mon Nov 27, 2006 6:51 am
Location: The Netherlands (NL)
Contact:

Post by sjoukew » Sun Sep 28, 2008 11:02 am

I read an article from seagate once about this topic. They claimed hard disks where no good as backup. Of course I can't find that article again :(

erkan
Posts: 51
Joined: Fri Sep 26, 2008 1:40 pm

Post by erkan » Sun Sep 28, 2008 11:48 am

I have read a thread on another forum about the same topic (forgot which forum).

Anyway, they contacted Seagate for info about this matter and Seagate told them it was a bad idea. They even told them that a disk has only a couple of months of life if they are stored offline and never powered up.

Now, I find that silly since I have quite a lot of disks still working 8+ years now that are not connected to any computer at all. But there may be some truth to it, I think it is more dangerous when the disk has been spinning and suddenly it is stored offline for a couple of months. Thats why I have the habbit of occasionally spin up disks that have rested for a couple of weeks.

Emyr
Posts: 91
Joined: Wed Jun 06, 2007 10:48 am
Location: Cardiff, UK

Post by Emyr » Sun Sep 28, 2008 12:18 pm

Since there's a lubricated bearing at the middle of them, I'd assume the lube flows down and the bearing dries up... and we have no way to re-lubricate it!

So balance spinning it regularly for bearing health vs. wear and tear of circuitry from all that motor power passing through: sync your stuff once a week and the drive should be pretty healthy.

erkan
Posts: 51
Joined: Fri Sep 26, 2008 1:40 pm

Post by erkan » Sun Sep 28, 2008 12:44 pm

I think you are 100% right.

There were lots of talk about bearings drying out IIRC.

MoJo
Posts: 773
Joined: Mon Jan 13, 2003 9:20 am
Location: UK

Post by MoJo » Sun Sep 28, 2008 12:48 pm

Get a Blu-Ray burner and user PAR2 files maybe?

Or just upload your collection The Pirate Bay and be assured you will always be able to re-download ;)

Lawrence Lee
SPCR Reviewer
Posts: 1115
Joined: Fri Mar 04, 2005 9:07 pm
Location: Vancouver

Post by Lawrence Lee » Sun Sep 28, 2008 1:22 pm

I don't think I've ever heard of a fluid dynamic bearing in a hard drive drying out. I'd only be concerned if it was an older drive with ball bearings.

MoJo
Posts: 773
Joined: Mon Jan 13, 2003 9:20 am
Location: UK

Post by MoJo » Sun Sep 28, 2008 2:25 pm

Why not get a cheap driver and try it? Fill it up with data and some kind of verification system (PAR2, or just MD5/CRC sums in an .sfv file), store it for a year and then try it.

At worst you get to RMA the drive...

tksh
Posts: 9
Joined: Sat Jun 09, 2007 6:15 am

Post by tksh » Sun Sep 28, 2008 5:20 pm

I've actually had older 4G to 20G drives not start up properly anymore after leaving them in the closet in good (and working) condition so I'm a believer of the lubricant claim.

Considering the cost of and maintainability of flash drives, I would put more faith in them than regular hard drives for multi-year archival but even then, you'll have to worry the capacitors eventually going bad.

My own strategy is to keep things fluid and make sure a copy of my important stuff gets on to new medium/hardware every couple of years.

erkan
Posts: 51
Joined: Fri Sep 26, 2008 1:40 pm

Post by erkan » Mon Sep 29, 2008 1:09 am

Maybe not the same, but we had a server running for 2+ years 24/7. Pulled the power and let it rest for two days becuase of a move. After two days the harddrive refused to work again.

That was a 10k SCSI HD.

sjoukew
Posts: 401
Joined: Mon Nov 27, 2006 6:51 am
Location: The Netherlands (NL)
Contact:

Post by sjoukew » Mon Sep 29, 2008 12:09 pm

Tapestreamers are meant for backing up large amounts of data on shelves, but can be quite costly.

Strid
Posts: 397
Joined: Tue Jan 08, 2008 5:09 am
Location: Copenhagen, Denmark

Post by Strid » Mon Sep 29, 2008 1:29 pm

I have a couple of old hard drives from around 98-99, which I play around with from time to time. Still works perfectly, although they are NOISY as Hell!!

I think a hard drive is good for more than you need unless you need like a life time or something. But who needs a drive when dead??

Only precaution I can think of are:
- Keep away from moist
- Store at room temperature
- Anti-static bag just for safety

EDIT: By the way .. off-line storage is probably the ultimate silent storage, am I right, huh?? :lol:

biatche
Posts: 33
Joined: Wed Feb 01, 2006 3:18 pm

Post by biatche » Mon Sep 29, 2008 10:37 pm

my concern, and well, paranoia for that matter is that these drives lose magnetism after some time... but whether it is the actual case im not quite sure

NyteOwl
Posts: 536
Joined: Wed Aug 23, 2006 7:09 pm
Location: Nova Scotia, Canada

Post by NyteOwl » Tue Sep 30, 2008 4:13 pm

All magnetic media gradually degrade. If you are comparing specs you want to look for the rententivity (magnetic retention) figures, and perhaps depending on the storage environment the coercivity (resistance to change from magnetic fields). Humididty will have little effect on the magnetic retention however elevated temperatures can degrade it over time. Similarly, physical shock can have an adverse effect on retentivity.

These characteristics apply solely to the platters themselves. Of more concern in long term storage is the problem of bearing lubrication and stiction as already mentioned. Temperature and humidity control can help here, but there are no guarantees.

That said, I fired up an old 486 last week that hadn't been used since 1999 and the 250MB Maxtor spun right up and booted into Windows 95 :)

Unless you have very large amounts of data to store, optical should be fine. The lifespan of good quality media under proper storage is 50-100 years. For large amounts of archival storage is needed magnetic tape is a good option. If the retention is going to be for more than 5-10 years I'd also store a spare tape drive along with the tapes - just in case. :)

matt_garman
*Lifetime Patron*
Posts: 541
Joined: Sun Jan 04, 2004 11:35 am
Location: Chicago, Ill., USA
Contact:

Post by matt_garman » Wed Oct 01, 2008 5:58 am

Here is an interesting article: "How To Choose CD/DVD Archival Media". It doesn't talk about offline hard drives though. :)

I too have started down the road of using lots of offline HDs with a dock as a backup storage solution. Assuming this isn't bad for the hard drive, it seems like a no-brainer, as internal hard drives are the cheapest means of bulk storage.

Anyway... another idea I've been toying with is building a "backup server". I'd start with a case like this, and fill it with the cheapest components I could find. Then I'd power it up once a week or so, do a sync, then power it down. Might be a nice compromise between letting the drives sit completely idle and running them full time.

Might be worthwhile to post this question to the StorageReview forums.

MoJo
Posts: 773
Joined: Mon Jan 13, 2003 9:20 am
Location: UK

Post by MoJo » Wed Oct 01, 2008 9:53 am

For archival DVDs, the best are Tao Yuden. They invented the original CD-R. Hard to get outside of Japan (where they are sold under the "That's" brand), but I know SVP do them in the UK.

Backup machine wise, why not just build a low power NAS machine and use HDD power saving to spin the drives down when not in use. That way you have instant access and the drives get a regular work-out. You could even schedule a full defrag/virus scan/md5 check once a week to case the drives to spin up and be used. With HDDs increasing in size so fast, you will be able to replace them drives every few years cheaply, even if they are not failing.

Most drives don't die suddenly, they go slowly. SMART monitoring software will warn you of this. In that respect, keeping the drives running 24/7 is actually probably the best way to preserve them. Seagate do a five year warranty anyway. Really, the biggest threat to drives sitting in a server is power spikes, so a good UPS might be an idea. I unplug my machines from the wall when not in use.

QuietOC
Posts: 1407
Joined: Tue Dec 13, 2005 1:08 pm
Location: Michigan
Contact:

Post by QuietOC » Wed Oct 01, 2008 11:04 am

I have several old 40GB and even a dirt cheap 6.5 GB Samsung 3.5" hard drives that are in perfect working condition. I just sold a crashed 120GB WD1200JB and a working 200GB Seagate 7200.7 on eBay to people looking to restore bad drives. I've only seen drives go bad running in a system (generally power/heat problems) or by being dropped. A hard drive sitting on a shelf isn't going to go bad--just keep it dry and away from static and shock.

I have several CD-Rs that will no longer read, and have had backup tapes with errors. My Zip disks/drives haven't had the click of death, yet--not that I have any reason to use them. I have plenty of 15 year old 1.44 MB floppies that still work fine. My 20+ year old Coleco ADAM 256kB tapes were just fine when I sold that machine two years ago. While my year old Corsair 2GB Flash USB stick has a short (need to send that in).

Commodity magnetic hard disks are probably the most reliable offline storage medium available.

MoJo
Posts: 773
Joined: Mon Jan 13, 2003 9:20 am
Location: UK

Post by MoJo » Wed Oct 01, 2008 1:56 pm

I guess the lesson is that for really valuable stuff, use multiple backup systems.

Have you considered slightly more exotic methods? Magneto Optical, while not popular in the west was/is big in Japan and is supposed to be very reliable for a very long time. Only downside is the cost of discs and the fact that they are 9.1GB each max.

There is always online backup too. Do it overnight or something. Mozy is supposed to be okay, and fairly cheap for an unlimited account. Backup is good anyway. To be honest, with DVD-Rs (which are recoverable even if damaged), PAR2 files and a few gmail accounts you could create a pretty robust system I think.

inti
Posts: 118
Joined: Thu Feb 10, 2005 7:09 am
Location: here

Post by inti » Thu Oct 02, 2008 1:38 am

I use HDDs for archival and backup, and I have some that are 5 years old with no problem at all. I probably only spin them up once every 3 months or so.

This can be a sensible approach to backup or archiving, but think about the following.

Cost: good quality DVD blanks (Taiyo Yuden) cost around $0.25 each. Realistically you can fit 4GB of data on each (sometimes less, depending on how your filesizes fit). An inexpensive hard drive (e.g. 640GB model) is about $70, although cost is coming down all the time. Factor in shipping and a good quality storage case for your DVDs and a few coasters and raw material cost is about the same.

Time: it would take many laborious days to burn 160 DVD blanks, also careful record-keeping (which files are backed up) and careful indexing. A 640GB hard drive can be fully backed up in 1-2 hours.

Risk of failure: in my experience, around 2% of DVD burns go bad after a few months even though they verify OK straight after the burn. I've never had a hard drive fail completely - see notes below - but the failure rate is supposed to be around 0.33% per year.

Type of failure (important): Often, if a DVD goes bad, the whole disk seems to become unreadable. While there are many ways a hard drive can fail, the most common failure is a bad sector: if that happens, unless it is the Master Boot Record or something, then the disk can still be read so 99.9% of your data may be recoverable. If you have very large files, you can use a parity/error recovery system (e.g. PAR2) at the time of writing the data to ensure 100% recoverability even if there is sector damage. If the hard drive electronics fail, your data can in theory be recovered cost-effectively by using a controller board from the same model of drive (so it's worth buying your archive drives in pairs). If the bearings fail, etc, then in theory your data can be recovered by a data recovery service for a few hundred dollars - most types of hard drive failure other than actual physical damage which shatters the glass platters are recoverable.

Convenience of backup: you are much more likely to backup your important data if the backup process is quick and easy - if you want to make redundant backups the same applies

Convenience of access: it is much easier to access your backed-up data in future by slotting in a hard drive that has it all on, than by having to locate the right DVD-R. If you need to restore the data, it will be much easier in future to restore it from a plug-in hard drive.

Off-site backup: because hard drives are small and portable, it would be easy to store some at your friend's house or your folks

Robustness: DVDs can be damaged by moisture, sunlight, environmental chemicals, physical damage (scratching). In contrast, a hard drive with the heads parked is surprisingly robust unless you drop it from the roof or something. DVDs are highly flammable, hard drives are not. I had a friend who had a fire in her office/studio, and her Mac burned completely, but the hard drive inside survived the fire and was still readable afterwards just by plugging into another computer.

Encryption: if your data is sensitive, you could use whole-volume encryption on a hard drive

Reusability: if your data becomes unwanted, you could wipe the hard drive and use it to back up something else instead.

Idiot factor: it is possible to accidentally erase a hard drive by a few stupid mouse clicks, but it is not possible accidentally to erase a DVD-R.

Out of 25 or so offline hard drives over the years, I have had:
* one that starting "clicking" (the head recalibration noise, indicating physical damage to the drive surface) - I stopped using the drive, pulled all the data off it (only one file was unreadable)
* one where a bad sector has developed in data, so again one file has become unreadable - I took all the important data off it but the rest of the drive is still fine so I now use that drive daily as a "scratch" drive for Windows Temp folder, etc
* one where the plastic surround of the SATA connector on the drive PCB snapped - it is still usable but the connector is not secure now

Obviously usual data backup rules apply: "unless it is backed up, it is not data you want to keep". I have the following backup policies:
* work and emails: one copy in my laptop, one daily backup onsite at my office, one weekly offsite backup, and a copy a few months old at home
* photos: three backups on different drives at home, updated in rotation
* music: one backup (and the original CDs)
* recorded TV shows: no backup

didi
Posts: 62
Joined: Tue Aug 28, 2007 7:44 am

Post by didi » Thu Oct 02, 2008 10:47 am

Hi, I didn't read the entire thread, maybe a few things were answered already.
I don't know where in the EU you're from, but in the current october issue of the Dutch C'T magazine (http://www.fnl.nl/ct/), they have an article about long term storage. Maybe you'll find the same article in the German C't, usually same content.

Their conclusion is, that hard drives can actually be quite good for long term storage, as long as they are stored properly:
- offline
- dry
- room temp
- stored in a cabinet and never moved
- in a sealed vibration absorbing shell (the plastic one the drives are shipped in seem to be excellent)
- treat them like "Write once, read only if necessary" media. The more you use and handle the drive, the more likely it will fail (or fall on the floor).

Also, the myth about oil bearings stiffening when not used for a while, doesn't seem to apply anymore to modern drives.

You'll have to keep in mind that within 10 years or so, storage interfaces may have changed, so always make sure you can connect your "old" backup media if you buy new hardware. SATA will probably be around for a couple more years, so should be ok (for now).
Apart from disappearing interfaces, another thing to be aware of are disappearing operating systems, software. You may also want to backup the software you use to play the files you want to backup, or even complete virtual machine images (complete with old OS).
example: WordPerfect files on floppies, created in the MSDOS era. You can't use them anymore because you may not have a floppy drive anymore, let alone a MSDOS machine running WordPerfect ;-)

If you fear the magnetic media losing its bits, just make a complete new copy on a new drive every 1-2 years (and in the meantime, get the newest interface available). The cost of a new hard drive every 2 years really is negligible.
And as said by someone else: make multiple copies of the same backup.

bkh
Posts: 93
Joined: Tue Jan 28, 2003 10:20 am

Post by bkh » Thu Oct 02, 2008 10:52 am

A hard disk is not an archival storage medium. If you write it, then put it away and hope for the best you may get a very bad surprise.

But as inti points out, you can use hard drives with multiple copies to get reliable long-term storage, and inti points out several good tips on how to do this.

But experience in large-scale data storage indicates that even when disks appear to be working, "bit rot" does occur. That is, you can read the data successfully but sometimes (rare, but true), the data are now wrong. The way to guard against this is to compute and store the md5 checksum with each file that you write. From time to time read all the files to recompute the md5sum and ensure that the value matches the one originally written. When there is a mismatch, you must retrieve another copy of the file where the md5sum is still good, and duplicate that file instance to replace the instance that went bad.

So it is not just keeping a few copies of each file, it is also verifying those copies via strong checksums to detect when one copy goes bad (not just when a physical device fails) so that there is still time to rescue the data from another copy.

didi
Posts: 62
Joined: Tue Aug 28, 2007 7:44 am

Post by didi » Thu Oct 02, 2008 11:34 am

Actually, even if a few bits on a hard drive have changed, the data can be read from it as usual, since every hard drive has error correcting features nowadays.

As to "bit rot", I tend to think that common backup media - like tapes, who are almost all magnetic media - would suffer more from bit rot, as they are much "weaker" media. Magnetic bits are stored on thin plastic sheets, wrapped around each other, only encased in a plastic container (opposed to hard drives who all have metal casing).

In the end however, "refreshing" the data regularly is necessary on all media, I guess.

bkh
Posts: 93
Joined: Tue Jan 28, 2003 10:20 am

Post by bkh » Thu Oct 02, 2008 8:44 pm

>Actually, even if a few bits on a hard drive have changed, the data can be read from it as usual, since every hard drive has error correcting features nowadays.

Perhaps you miss my point. Despite ECC, sometimes the data that is read is not identical to the data that was earlier written. This sort of "silent" failure is not common, but it does happen, and you likely won't know that some of your data have been degraded unless you periodically verify strong checksums over the file data. The hardware's own ECC doesn't completely repair or even detect 100% of such failures.

>As to "bit rot", I tend to think that common backup media - like tapes, who are almost all magnetic media - would suffer more from bit rot, as they are much "weaker" media.

You are telling us some plausible-sounding thoughts you have, but recall that on tape the areal density is much lower than on a modern disk: each bit on the tape is stored over a relatively huge area of magnetic material, which provides a kind of robustness. Properly stored archival-quality data tape is designed to preserve the data for decades, and it does so in practice. Unfortunately the cost of data storage on tape is prohibitive except for large organizations. Moreover, as you suggest, tapes and disks differ in the kinds of physical damage to which they are vulnerable.

didi
Posts: 62
Joined: Tue Aug 28, 2007 7:44 am

Post by didi » Fri Oct 03, 2008 3:49 am

I get your point. ECC is one thing, consistency checking is another. But that applies to all backup media.

You are correct on arael density. But the density is mainly chosen for the specific speed of the medium.
Still I think my bits are safer on a solid carrier (drive platter) than on a flexible material (tape), if both are stored as they should be and left untouched. That's the biggest "danger" for hard drives as backup media, since they are so easy to connect, read from and write to.The more one does, the less reliable it gets.

Post Reply