Backup to disk storage in a business environment

dhanson865 · Post by **dhanson865** » Fri Sep 25, 2009 10:51 am

A RAID 0 array died today. It's primary use was for storage of backuptodisk files for Symantec Backup Exec. Since we do daily backup to tape the backup to disk is merely for convenience of not having to mount a tape to restore.

The drive that actually failed is a Maxtor Atlas 10,000 RPM Serial Attached SCSI (aka SAS) manufactured 27Jul2006. Its 8J300S0 which is a 300GB drive with 16mb cache.

Since the backup restore process is mostly sustained linear writes/reads I was considering taking advantage of modern higher density platters and going with something cheaper like a WD or Samsung drive in the 500GB to 1TB range with preference towards the small end as less platters has been more reliable in past experience (and drive specs if you read them carefully).

The box they would sit in is on 24/7 and has a LSI based raid controller (I think it's a PERC5).

The bottom of the barrel pricewise would be Samsung EcoGreen F2 HD502HI 500GB 3.5" Hard Drive (SATA, 5,400 RPM, 16MB) (single 500GB platter) at about $50 each.

next step up is WD Caviar Blue or Black 640GB Hard Drive (Serial ATA, 7,200 RPM, 16MB or 32MB) (320 GB platters) at about $70 each.

newly decoded SAMSUNG EcoGreen F2 HD103SI 1TB 5400 RPM 32MB Cache SATA(500 GB Platters) at about $90 each.

newly suggested SEAGATE 1tb ST31000520AS SATA 5900rpm 32mb (500 GB Platters) at about $90 each.

going away from consumer grade drives and looking at raid editions we have

WD RE2 500GB WD5000YS (SATA, 7,200 RPM, 16MB) (250GB per platter) at about $50 each.

WD RE3 WD5002ABYS 500 GB (SATA - 7200 rpm - 16 MB) (333GB per platter) at about $100 each

There are similar options on the Samsung F1 RAID drives and based on newegg comments the Samsung drive seem to have less errors vs the WD RE drives.

If the drives are big enough I'll probably just do a RAID 1 this time instead of RAID 0 and call it good.

Toss your suggestions here.

Wibla · Post by **Wibla** » Fri Sep 25, 2009 2:26 pm

Yes, I have abit of a problem understanding why RAID0 was chosen for this workload, RAID1 is more suitable.

I'd go with Samsung tbh, they seem quite reliable, atleast in my experience.

HFat · Post by **HFat** » Sat Sep 26, 2009 4:50 am

dhanson865 wrote:Since we do daily backup to tape the backup to disk is merely for convenience of not having to mount a tape to restore.
[...]
If the drives are big enough I'll probably just do a RAID 1 this time instead of RAID 0 and call it good.

Why are you planning to use RAID at all? Either you're not telling everything or there's no reason to buy more than a single inexpensive drive.

In most cases (it depends on the kind of data), when backing up to two drives which are large enough to hold a backup in RAID1, you're better off with JBOD.

The number of heads apparently matters for reliability by the way, not only the number of platters.

Otter · Post by **Otter** » Sat Sep 26, 2009 6:17 am

How often do you back up to disk and to tape, and how critical is that data? Would loosing half a day's worth be a major problem or just a minor annoyance?

dhanson865 · Post by **dhanson865** » Sat Sep 26, 2009 6:27 am

HFat wrote:
dhanson865 wrote:Since we do daily backup to tape the backup to disk is merely for convenience of not having to mount a tape to restore.
[...]
If the drives are big enough I'll probably just do a RAID 1 this time instead of RAID 0 and call it good.
Why are you planning to use RAID at all?

The highest risk of failure is in the first 3 months of use.

I really don't want to open the box up and replace drives more than I have to. The physical part of that is inconvenient but the management of past backups and scheduled backups is a hassle as well. If I'm using a single drive and it fails I have to purchase/warranty replace that drive and deal with all the configuration hassles with backup exec. It would cost me a significant amount of man hours. If I'm using a raid 1 array and a single drive fails I can replace the single drive and I don't have to reconfigure anything at all in backup exec.

dhanson865 · Post by **dhanson865** » Sat Sep 26, 2009 6:28 am

Otter wrote:How often do you back up to disk and to tape, and how critical is that data? Would loosing half a day's worth be a major problem or just a minor annoyance?

The backup to disk and backup to tape are offset by several hours but the real world difference vs everyone going home at 5PM is that the two backups have little difference in data. There are those few users that work late and the production web server is being hit by people in other time zones but the difference in time of day isn't the concern between the types of backups.

1. If a backup to disk exists and the equivalent tape is off site then I don't have to have someone drive to get the tape. Backup to disk could be reduce the downtime by the time needed to get to that tape. The worst case scenario is the vault tapes are stored in couldn't be opened for a day or two. (bank is closed, technical difficulties getting the vault open, etc).

2. If the tape unit ever fails or the media fails I can still recover from the backup to disk. That redundancy is crucial to business continuity. If you can't imagine how a failure on a production system around the same time your tape drive fails then you just haven't worked IT long enough to be bitten. I've never had that particular scenario but I learned a long time ago to avoid putting all my eggs in one basket.

As to the tapes I do test recovery on a schedule but that doesn't prevent that combination from occurring it just reduces the odds that you'll go extended periods with no valid tape recovery.

Again this is about keeping the business running, flexibility to handle more combinations of issues than one method alone.

HFat · Post by **HFat** » Sat Sep 26, 2009 7:19 am

dhanson865 wrote:The highest risk of failure is in the first 3 months of use.

Only if you don't keep drives for very long.

dhanson865 wrote:deal with all the configuration hassles with backup exec. It would cost me a significant amount of man hours.

Really? I'll avoid the software at all costs then. Thanks for the tip.

HFat · Post by **HFat** » Sat Sep 26, 2009 7:28 am

dhanson865 wrote:1. If a backup to disk exists and the equivalent tape is off site then I don't have to have someone drive to get the tape. Backup to disk could be reduce the downtime by the time needed to get to that tape. The worst case scenario is the vault tapes are stored in couldn't be opened for a day or two. (bank is closed, technical difficulties getting the vault open, etc).

2. If the tape unit ever fails or the media fails I can still recover from the backup to disk. That redundancy is crucial to business continuity.

Then what you said above ("merely for convenience") is not true. Especially if you're concerned about downtime costs, by all means spend $50 and get RAID1 for this thing. Best use different drives too (if at all possible).

What I said above (and what Otter may be hinting at) is still valid though: depending on the particulars, it may be better to rotate between two drives than to have the same backup on two drives (which is what RAID1 nets you). But it looks like your software isn't going to allow that...

Otter · Post by **Otter** » Sat Sep 26, 2009 9:01 am

Hinting is when you say something other than what you mean in hopes that the person you are talking to will ignore the meaning of what you actually said and take the meaning of what you didn't say, but were thinking, as opposed to the millions of things you didn't say but weren't thinking. Communication is hard enough without that. I never hint.

I do however, ask lots of questions.

dhanson865 wrote:I really don't want to open the box up and replace drives more than I have to. The physical part of that is inconvenient but the management of past backups and scheduled backups is a hassle as well. If I'm using a single drive and it fails I have to purchase/warranty replace that drive and deal with all the configuration hassles with backup exec. It would cost me a significant amount of man hours. If I'm using a raid 1 array and a single drive fails I can replace the single drive and I don't have to reconfigure anything at all in backup exec.

I think your strategy makes sense for what you are doing. It sounds like RAID 1 is a good match. If you need more space, or you were using RAID 0 before because the performance boost was helpful, then consider RAID 10.

dhanson865 · Post by **dhanson865** » Sat Sep 26, 2009 9:44 am

The point of the thread isn't about what raid level to use. The point of the thread is what drives to purchase.

I see no reason to pay for SAS 10K or 15K drives for near line storage. I'm thinking raid edition SATA or even consumer grade SATA drives would be a better value.

The question is how far down the food chain would you go in price before you'd be concerned about it being too cheap a drive for using with a LSI based RAID card?

HFat · Post by **HFat** » Sat Sep 26, 2009 9:58 am

The I in RAID stands for inexpensive, your card is a sunk cost unfortunately and it doesn't sound like performance would be an issue for your application.

Am I hinting now?

Otter · Post by **Otter** » Sat Sep 26, 2009 12:56 pm

I'm not convinced the enterprise/raid versions of the drives are worth the higher cost. I suspect that these drives are often often the same as the consumer versions, but with different firmware and more hype. But I don't run a server farm, so what do I know?

Post by **MikeC** » Sat Sep 26, 2009 4:49 pm

2 Samsung ecogreen 500s would be my choice.

dhanson865 · Post by **dhanson865** » Sat Sep 26, 2009 9:14 pm

MikeC wrote:2 Samsung ecogreen 500s would be my choice.

I haven't mastered the Samsung part numbers, can you clue me in if there is a 2 platter 1 TB ecogreen that is equivalent to the HD502HI (500GB)?

I'm honestly not as worried about noise in this usage case as it will be in a server room full of noisier equipment. These drives will however sit idle 2/3 of the day or better so low idle power draw is a plus, just not my first priority.

Post by **MikeC** » Sun Sep 27, 2009 9:14 am

dhanson865 wrote:
MikeC wrote:2 Samsung ecogreen 500s would be my choice.
I haven't mastered the Samsung part numbers, can you clue me in if there is a 2 platter 1 TB ecogreen that is equivalent to the HD502HI (500GB)?

I'm honestly not as worried about noise in this usage case as it will be in a server room full of noisier equipment. These drives will however sit idle 2/3 of the day or better so low idle power draw is a plus, just not my first priority.

In that case, a Seagate Barracuda LP 1tb might be a better choice. Better warranty, I think, and avg idle power is 3W.
http://www.seagate.com/docs/pdf/datashe ... uda_lp.pdf

dhanson865 · Post by **dhanson865** » Sun Sep 27, 2009 7:05 pm

Looks like the Samsung and Seagate drives would both have a 3 year warranty. It's a shame there isn't a 500GB platter green drive from WD to consider in the mix here.

newly decoded SAMSUNG EcoGreen F2 HD103SI 1TB 5400 RPM 32MB Cache SATA(500 GB Platters) at about $90 each.

newly suggested SEAGATE 1tb ST31000520AS SATA 5900rpm 32mb (500 GB Platters) at about $90 each.

500 GB platters at 5x00 RPM or 333 GB platters at 7200 RPM both give me the same basic sustained R/W that the 10,000 RPM atlas gave me.

I doubt the difference between 3 and 4.x watts will matter when choosing between the Samsung and Seagate eco options. Reliability is more important to me than a few watts at idle.

barophobia · Post by **barophobia** » Mon Sep 28, 2009 6:42 pm

dhanson865 wrote:The question is how far down the food chain would you go in price before you'd be concerned about it being too cheap a drive for using with a LSI based RAID card?

If you don't really care about performance why even bother with a dedicated raid card. Just run Linux and do software raid. And all you are doing with a dedicated raid card is adding a new point of failure. Seriously a raid card failure especially a old one is a pain in the neck.

Anyways if you care about your data you be using a raid 1 or 5 or 5+1. And get used to replacing hard drives; it is your job. X year warranty means nothing when the drive failure takes your array down.

dhanson865 · Post by **dhanson865** » Tue Sep 29, 2009 5:28 am

barophobia wrote:
dhanson865 wrote:The question is how far down the food chain would you go in price before you'd be concerned about it being too cheap a drive for using with a LSI based RAID card?
If you don't really care about performance why even bother with a dedicated raid card. Just run Linux and do software raid. And all you are doing with a dedicated raid card is adding a new point of failure. Seriously a raid card failure especially a old one is a pain in the neck.

Anyways if you care about your data you be using a raid 1 or 5 or 5+1. And get used to replacing hard drives; it is your job. X year warranty means nothing when the drive failure takes your array down.

1. You don't have to run linux to do software raid. Windows Server does RAID 0, 1, 5 just fine in software.

2. I'm using Backup Exec for windows servers. While it does a linux agent I don't think I could have the main application work on a linux box. Switching away from Backup Exec would mean legacy issues would have to be resolved considering I have LTO tapes archived.

3. I find it seriously funny you would suggest RAID 5. In this particular case I only have 2 bays for this array unless I break up the other two drive array in this machine but even if I did I wouldn't consider raid 5. Even then if I buy new drives larger than 500GB my stripe size will be limited by the 500GB drives I have as the other two drives in the server. RAID 10 is the closest I'd ever get to RAID 5 in my current environment. I'd do RAID 0, JBOD, unRAID, etc before I'd ever consider RAID 5 for this use.

I might suggest you read something like http://www.baarf.com/ and learn to stop recommending raid 5.

Jay_S · Post by **Jay_S** » Tue Sep 29, 2009 8:17 am

I set up a backup to disk where I work for same reason you list above - having our data in "more than one basket".

IMO, there's no reason for any business NOT to do this, since any old PC collecting dust can run FreeNAS. FreeNAS can do whatever RAID you want in software, can be a rsync server OR client, and is totally free. So with very little investment a company can deploy a backup backup server. "Too much redundancy" is oxymoronic.

It's been a long time since we gave up on Backup Exec. We now use BrightStor ARCserve. You can create a what it calls a "File System Device", which is just a local or network storage location which it can use as backup media. We do full nightly backups to the FreeNAS server, with full weekly & daily incrementals going off-site each night.

How big is your backup job? Can it fit comfortably on a 500GB drive? If so, I would purchase three single-platter 500GB drives. Connect two in RAID1 for live redundancy, and keep 1 on-hand for a spare. That way your backup target isn't without redundancy while waiting for RMA, etc.

Finally, in your original post you mentioned raid 0 ... is this a type-o? Raid zero essentially halves mtbf of any single drive. This is not a good idea for backup of business-critical data!

dhanson865 · Post by **dhanson865** » Tue Sep 29, 2009 9:11 am

Jay_S wrote:Finally, in your original post you mentioned raid 0 ... is this a type-o? Raid zero essentially halves mtbf of any single drive. This is not a good idea for backup of business-critical data!

I mentioned that the old array was RAID 0 because the drives were 300GB each. It was setup before I started working here and was better than nothing. Assuming the drives that replace it are 1TB I'd have no reason to use RAID 0 with drives that large.

I have a pair of 500GB drives in the PC also in RAID 0 but I plan to rebuild that array after the new larger drives are in place.

Unfortunately Backup Exec isn't friendly about moving its data files around after a backup is made. It will take work to move the data off the raid 0 array to the new RAID 1 array.

barophobia · Post by **barophobia** » Tue Sep 29, 2009 9:12 am

dhanson865 wrote: 1. You don't have to run linux to do software raid. Windows Server does RAID 0, 1, 5 just fine in software.

2. I'm using Backup Exec for windows servers. While it does a linux agent I don't think I could have the main application work on a linux box. Switching away from Backup Exec would mean legacy issues would have to be resolved considering I have LTO tapes archived.

Ok if I understand your requirements are correctly I would just put in two large consumer drives in raid 1 and call it a day.

Worrying about which brand will fail first is just silly to me.

I might just check if there is some other better use for that raid card first though.

dhanson865 · Post by **dhanson865** » Tue Sep 29, 2009 10:50 am

barophobia wrote:I might just check if there is some other better use for that raid card first though.

The boot drive is a 73GB SAS and the motherboard only has 3 SATA plugs. I'd have to leave the RAID card in to run 5 drives as I have been. Taking out the RAID card would mean adding in another drive controller to replace it.

However if I knew the motherboard SATA plug would work with the SAS drive I could take it off the RAID controller and move all the RAID drives onto the LSI controller. Or just for redundancy I could put one drive of each RAID array on a different drive controller and do software RAID instead of letting the LSI RAID controller do it all.

Jay_S · Post by **Jay_S** » Tue Sep 29, 2009 12:46 pm

OK. Five drives? 73GB SAS, 2x500GB, 2x300GB.

I'm thinking you could just mirror a single pair of 1TB drives and ditch all the rest of those drives. I assume this is some sort of dell poweredge? Can server 2008 mirror the primary OS drive(s) through the onboard SATA controller?

That would also free up your PERC5, which is a little overkill for this backup application.

dhanson865 · Post by **dhanson865** » Tue Sep 29, 2009 1:14 pm

Jay_S wrote:OK. Five drives? 73GB SAS, 2x500GB, 2x300GB.

I'm thinking you could just mirror a single pair of 1TB drives and ditch all the rest of those drives. I assume this is some sort of dell poweredge? Can server 2008 mirror the primary OS drive(s) through the onboard SATA controller?

That would also free up your PERC5, which is a little overkill for this backup application.

1. I have the 73GB SAS drive already configured and operational. I'd rather not take it out of the loop as rebuilding the server and dealing with the implications of backup exec logs, indexes, scheduled jobs, etc is a mess I'd rather not deal with especially since I'd have to do it within a specific timeframe to not interfere with my existing backup schedule.

2. A single pair of 1 TB drives mirrored plus a single boot drive is still less storage than I did have. I'd rather not reduce the storage capacity on that server.

This server is on Server 2003 standard but yes Server 200x can mirror any drive. That isn't an issue. The issue is having room for a 6th drive or dropping to 4 drives if I mirror the boot (by the traditional method of not overlapping arrays).

Considering that I will probably go ahead and Mirror the boot to one of the 1TB or 500GB drives for redundancy and eat the 70GB loss in space on the other mirror.

3. Will the SAS drive work on the motherboard SATA controller? I assumed it wouldn't but If so I could consider lowering the capacity of the backup solution to free up the RAID controller for another server. But even then I'm not sure where I'd use it off the top of my head.

Doing this would lower my total capacity to 1TB on 3 drives or 1.5 TB on 5 drives.

dhanson865 · Post by **dhanson865** » Tue Oct 06, 2009 9:14 am

Jay_S wrote:OK. Five drives? 73GB SAS, 2x500GB, 2x300GB.

I'm thinking you could just mirror a single pair of 1TB drives and ditch all the rest of those drives. I assume this is some sort of dell poweredge? Can server 2008 mirror the primary OS drive(s) through the onboard SATA controller?

That would also free up your PERC5, which is a little overkill for this backup application.

Followup post here:

The SATA and the SAS Perc5 are both onboard controllers. The RAID controller isn't on a discrete card it is integrated.

The old 300GB drives are sitting on a desk right now. One of them is still viable and might go in another server as a hot spare or may just sit around and go unused. That is a project for another time.

I tried booting the 73GB SAS drive from the non raid controller and the SATA controller won't see the SAS drive. I expected as much but I wanted to confirm before I setup the new RAID arrays.

I went with 3 RAID 1 arrays

Array 1: ~68GB C: (Server 2003 software RAID 1)
73GB SAS (on the LSI controller)
500GB SATA (on the Intel controller) same drive as in array 2

Array 2: ~397GB D: (Server 2003 software RAID 1)
500GB SATA (on the LSI controller)
500GB SATA (on the Intel controller) same drive as in array 1

Array 3: ~931GB E: (LSI hardware RAID 1)
1TB SATA (on the LSI controller)
1TB SATA (on the LSI controller)

If I had a non SAS boot drive I would have done hardware RAID 1 and software RAID 0 to make a RAID 10 volume but I really didn't want to have to go through the hassle of setting up this server from scratch on short notice. I'll save that for down the road as well.

So far the first Full backup to disk hasn't finished. I'll have to let it run for a week or so to be sure how the performance compares. I was doing backup to LTO3 and to disk overlapping last night to get the backup rotation on track so performance was degraded as each job was contending for some of the same resources (CPU, C: drive, network adapter, hosts it they were backing up).

Hopefully the server will settle down before the afternoon backup starts so I can try a benchmark or two.

dhanson865 · Post by **dhanson865** » Wed Oct 07, 2009 4:42 am

Say does anyone know of a free benchmark that tests transfer rates by volume instead of by physical drive?

HDTune and the like all see drives (73G, 500G, 1TB) not drive letters (C:, D:, E: ).

HFat · Post by **HFat** » Wed Oct 07, 2009 5:08 pm

hdparm from a LiveCD?

Jay_S · Post by **Jay_S** » Fri Oct 09, 2009 12:59 pm

IOZone
How-to: "How We Test Networked Storage Devices"
You may need to map the volume (ie., give it a letter) if you want to bench it over your network.

dhanson865 · Post by **dhanson865** » Mon Oct 26, 2009 3:13 pm

OK, synthetic benchmarks aren't needed now.

I saw performance differences the first night I backed up to the new drives and I gave it a few weeks to normalize (as backup exec makes new files when needed but will eventually reuse the storage files after the drive fills I wanted to see how it behaved once the cycle completed).

The last full backup to disk before the drive failure took over 12 hours at 760 MB/min

The most recent full backup to disk took over 28 hours at 305 MB/min

I'm guessing that is why the old array was RAID 0. I knew I'd take a hit switching to lower RPM drives and RAID 1 but you never know exactly how much of a hit it will be until you try it.

Keep in mind the backups are completing with plenty of time to spare. Diffs take less than 2 hours on the weekdays. The weekend backups aren't effecting anything other than my convenience of being able to work on a server on the weekend without rescheduling a backup.

With that said, I either have to put up with the slower backup to disk rates or ditch the Seagate LP drives and put in something that will give me better transfer rates.

SEAGATE 1tb ST31000520AS SATA 5900rpm 32mb (500 GB Platters) claim 95MB/sec but that is on reads from the outer disk. Writes are slower and inner disk writes slower still.

Looking at WD Black they say 106MB/sec on the 1TB drive (3 platters of 333GB each) and 138MB/sec on the 2TB drive (4 platters of 500GB each). Not really wanting to go to a 3 or 4 platter drive for a minor speed boost.

WD doesn't offer Blue models above 750GB so that isn't an option unless I go back to RAID 0.

WD RE3 WD1002FBYS 1TB (SATA - 7200 rpm - 16 MB) (333GB per platter) is only slightly faster than the Seagate LP and is slower than the 2TB WD Black.

I guess I put up with the slower backups until I either setup an external enclosure or put a better raid controller in so I can do RAID 10. Or maybe I'll just redo that RAID 1 array as RAID 0 to double the space and double the speed.

I suppose I'll also trim some files off the disks and reduce the coverage of the so called "Full" backup to disk to reduce the load of the weekend backup to disk. Maybe renaming that set to Weekend instead of Full might be in order if I do reduce the amount it covers.

dhanson865 · Post by **dhanson865** » Sat Oct 31, 2009 12:22 pm

dhanson865 wrote: The last full backup to disk before the drive failure took over 12 hours at 760 MB/min

The most recent full backup to disk took over 28 hours at 305 MB/min

OK I tried redoing the 1TB Seagate LP drives in RAID 0 and the latest backup went over 13 hours at 733 MB/min

So same drives, same RAID controller, same backup software, etcetera and RAID 0 was 2.4 times faster than RAID 1 in this usage.

I can now also say that the 5900 RPM 1TB drive is 96% as fast as the 15000 RPM 300GB drive it replaced in this usage. Thank God for high density platters.

Oh, and the current build is

Array 1: ~68GB C: (Server 2003 software RAID 1)
73GB SAS (on the LSI controller)
500GB SATA (on the Intel controller) same drive as in array 2

Array 2: ~397GB D: (Server 2003 software RAID 1)
500GB SATA (on the LSI controller)
500GB SATA (on the Intel controller) same drive as in array 1

Array 3: ~1.81TB E: (LSI hardware RAID 0)
1TB SATA (on the LSI controller)
1TB SATA (on the LSI controller)

Backup to disk storage in a business environment

Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment

Re: Backup to disk storage in a business environment