SPCR

Previously my main fileserver, got bumped down to backup server in 2008, now rebuilt with 16x1.5TB drives, undergoing testing to be recertified as main fileserver (and thus bumping the 9.1TB box down to backup duties)

Check the last page for new pictures and more info.

Specs: - UPDATED 21.03.10

Asus P5WDG2-WS Motherboard
Intel Celeron-D 352 (3.2GHz)
4GB PC6400 RAM
Intel PCI-E 1Gbit NIC
Corsair HX520 PSU
3ware 9500S-12 PCI-X Raid controller
16x1.5TB Seagate LP drives
320GB Samsung PATA system drive
Coolermaster Stacker (Classic)
4x Coolermaster 4-in-3
Intake: 3xNexus 120mm fans, 1xScythe S-FLEX on the 4in3's. Exhaust: 1xNoctua 120mm + PSU

(There was alot of outdated text here, removed now. Leaving images for "historic" reasons, also added the major milestone images in this post)


An awful mess during testing, summer 2007..


"In production" - summer 2007, tried to tidy up the cables abit, but basically gave up. 80mm fan added to compensate for lacking performance from the Noctua 120mm I had in the top 4in3 at the time, worked well, but with more noise. Motherboard/CPU/GPU was changed out abit later due to very bad performance from the storage array.


August 2007 - Mobo changeout. (ALOT) better performance, 300+MB/s read, 120MB/s write


Summer 2009 Rebuild - this is after two drives failed inside a week of eachother and I had no choice but to scrap the current array, but thanks to 3ware I didnt loose any data.


October 2009 - Removed 8x500GB drives, added in 16x1.5TB Seagate LP drives and another 4in3. Next up is swapping the 30GB WD drive out for something abit more recent (and quieter, the bearings are whining more by the day..)

That certainly is not silent but I am impressed... That many HDD's. Man... I could fit nice part of my anime collection in those HDD's...

It's actually not very noisy at all, there is almost no vibration because the stacker is quite heavy, and the 120mm fans are slow running fans, also the 120mm's in the front are behind dust filters that suck up abit noise, albeit at the cost of some airflow.

The box actually makes less noise than my sister's desktop, which is a 2500+, 6800GT... with a crappy psu, one hdd and a couple of 80mm fans... whahahahaha.

I am even more impressed. But I still think that system would be quite noisy for me. Anyway, what's that expansion card beneath VGA card? Extra PCI sata-controller?

its the 3ware 12port SATA Raid card I use for hwraid5

.. obviously its not SILENT, but its definently alot less noisy than expected

It will not reside under my desk forever tho, I'm gonna move it down into the basement when I get home next time (5-6 weeks, summer vacation) as it is in the way and heats up the whole area.. 9 hdd's produce a healthy bit of heat...

Any Read/Write benchmarks you want to share?

I want to build something like this myself, it's just a question of money and space. So if you don't mind, I have several questions.
(1) Are you running lunix or windows there?
(2) Do you have it all in a single partition or in several smaller ones. I've read that either windows or many hardware controllers have an issue with partitions larger than 2TB so I was wondering if you've run into this issue or not.

Thanks.

jezuz!

Performance is pretty abysmal, the 3ware cards has never been strong performers with regards to write speed on raid5, and the read speed is limited because i only have pci 32bit/33mhz instead of pci-x... I'm currently considering a new mobo+cpu+ram combo to get pci-x and hopefully better performance, but its not at the top of my list as of now, the performance is more than enough for normal storage usage.

I use linux, with XFS filesystem on the raw device, which makes it super easy to grow the fs if i opt to use OCE (online capacity expansion), which the 3ware controller supports without problems...

jalridge6: this isnt an extreme setup, a friend of mine has 4 stackers full of drives now, but he's crazy

Here is an example of bonnie results with software raid0 on two HD501LJ drives...:
Needless to say, my next raid setup will be software based on a C2D box with plenty of AHCI SATA ports

Linux, its got to be, i've never seen windows not a difference between the letter "o" and the number zero. however, linux definitely makes it known.

what distro? i could see it being a Ubuntu variant, but i could easily be mistaken.

if you're shooting for PCI-X, look at used/old low-mid range servers, almost all of them have PCI-X slots...but if there are no normal PCI slots, make sure that you've got compatible cards, it'll become a large problem for you if not.

how effective are the front 120mm fans at preventing silicon pies? i'd think that many HDDs would get quite warm.

its Debian, I dont use ubuntu..

I'm looking at one of the Asus P5WDG2-WS Pro cards, one with 2xPCI-X and C2D support, with plans of doing a E4300 + 2GB ram setup on it if I decide to upgrade, then I have alot more SATA ports too, for future expansion

Sorry, but I HAVE to ask.... What do you DO with more than 2TB of space? Very nice build though.

Well, I dont like deleting... anything. So stuff tends to accumulate...

I'm not completely happy with the setup, performance is abit lacking, and I need to move the server down into the basement...

wouldnt 8x500GB be 4gb? Why does the topic say 2.8?

jaldridge6 wrote:wouldnt 8x500GB be 4gb? Why does the topic say 2.8?

I would expect a single RAID5 array of 8 x 500GB to have a total capacity of about 3500GB...

Because I have 7x500GB in the array + a hotspare, so total usable space is about 2.7-2.8TB

Wibla wrote:
Abit worried about drive 6 (twa0_5) and drive 8 (twa0_7), they shouldn't be running so hot.

Those temps are just fine, actually, they're quite low...even the hot ones. Some studies suggest longer lifespan for HDDs running at mid to high temps anyway. So, I wouldn't worry about them.

Nice setup, I've been wanting to build one similar....but with a 3ware off PCIe, so I don't have to get a server mobo for decent throughput.

Wibla wrote:also had to add a 80mm fan to the top drives, used very thick double sided tape that also picks up alot of the vibration from the fan.[/url]

Out of interest, why did you need the additional 80mm fan? Doesn't the 4-in-3 module's 120mm fan keep them cool enough on its own?

/edit: silly question, of course not, or you wouldn't have put the extra fan in. Did you break out the 4-in-3's perforated faceplates between the fan and the drives? They restrict the fan's airflow quite severely...

I removed all the perforated faceplates, but I still thought the temps was abit high for the topmost 3 drives compared to the bottom ones, so then the 80mm went in

miahallen: PCIE is a good choice, but you could just aswell get an abit aw9 with 9 SATA ports and do swraid in linux, a friend of mine is doing insane speeds on that setup, and with a C2D you wont have cpu load problems.

I'm having real throughput problems with my 3ware setup, most probably because I'm on pci2.0 (32bit/33mhz) + gigabit controller on same pci bus

Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea

Waiting abit seems like a good idea, maybe the 1TB drives will be more nicely priced next spring too?

miahallen wrote:Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea

What was your issue with software raid? I've been using SW linux raid for years at work and its been nothing but great on SMP systems. I just built a 3-HDD SW raid system for myself (with an ASUS M2N32-SLI which has 6 internal SATA ports) at home and performance is great. Even at full writing power with RAID5 the system doesn't burn much CPU. Comparable HW raid cards would cost close to 1k. I've seen even $500-$800 cards that get outperformed by SW RAID.

Perhaps you were thinking of Windows-based software RAID? Something I have zero experience with but I can easily see problems in that world. =) Last I heard you can't even install the OS on a raid volume.

Oh and as for HD encoding... I do a fair bit of it but disk performance has never been even close to a bottleneck. Writing to disk is nothing, less than 1-5MB/s for anyone encoding HD, and reading from disk is plenty fast (for my typical down-sampling encode, i.e. from the ATSC MPEG2 ~20Mbps to XviD @2-5Mbps). I get an average sequential read of 100-120MiB/s with my raid5 array and encoding does not use up even a quarter of that.

Shusher wrote:

miahallen wrote:Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea

What was your issue with software raid? I've been using SW linux raid for years at work and its been nothing but great on SMP systems. I just built a 3-HDD SW raid system for myself (with an ASUS M2N32-SLI which has 6 internal SATA ports) at home and performance is great. Even at full writing power with RAID5 the system doesn't burn much CPU. Comparable HW raid cards would cost close to 1k. I've seen even $500-$800 cards that get outperformed by SW RAID.

Perhaps you were thinking of Windows-based software RAID? Something I have zero experience with but I can easily see problems in that world. =) Last I heard you can't even install the OS on a raid volume.

Oh and as for HD encoding... I do a fair bit of it but disk performance has never been even close to a bottleneck. Writing to disk is nothing, less than 1-5MB/s for anyone encoding HD, and reading from disk is plenty fast (for my typical down-sampling encode, i.e. from the ATSC MPEG2 ~20Mbps to XviD @2-5Mbps). I get an average sequential read of 100-120MiB/s with my raid5 array and encoding does not use up even a quarter of that.

I've used RAID0 for 3 or 4 previos OS installs, and I had various problems in all instances, mainly with stability which I always concluded were due to the RAIDs. So I've given up on them. I never noticed much performance enhancement with them anyway. The benchmarks looked good, but in everyday usage, I never could tell.

In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.

Now I have my OS and apps installed on a 160GB drive, all my games on a 320GB drive....and the other 2 320GB drives I'm using for my media files. I have a second computer (my HTPC) with a single 160GB drive that doubles as a backup device for my critical files. I'm as happy as can be, great performace, and with my various files stratigically located on seperate drives, this is the best everyday performace I've ever had. reason = I can encode media (idle proccess priority) on the media drives, play a game off my gaming drive, and stream a movie from the media drives to my HTPC & laptop simultaniously, without any noticable loss in performance.

Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...

miahallen wrote:In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.

Are you doing all this with Linux RAID or Windows RAID? Even my oldest system (2x60GB) outperforms that. Are you sure you were running in UDMA mode (assuming we're talking about ATA disks)? Did you have a really slow system bus? If this is a linux setup, it's more likely to be a misconfiguration than the SW RAID.

crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...

I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.

Shusher wrote:

miahallen wrote:In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.

Are you doing all this with Linux RAID or Windows RAID? Even my oldest system (2x60GB) outperforms that. Are you sure you were running in UDMA mode (assuming we're talking about ATA disks)? Did you have a really slow system bus? If this is a linux setup, it's more likely to be a misconfiguration than the SW RAID.

I configured the RAID5 through the RAID controller's (si3114) BIOS setup program. Windows Vista automatically detected it as a single volume. Two drives were WD3200KS and the third is a WD3200AAKS. The entire system specs can be seen in my sig.
And, if you're wondering if all three drives were working, they were. That was the first thing I checked.

miahallen wrote:

crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...

I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.

Are you talking about the data cable? I'm talking about the power cable... the data cables aren't a problem if they have clips.

miahallen wrote:I configured the RAID5 through the RAID controller's (si3114) BIOS setup program. Windows Vista automatically detected it as a single volume. Two drives were WD3200KS and the third is a WD3200AAKS. The entire system specs can be seen in my sig.
And, if you're wondering if all three drives were working, they were. That was the first thing I checked.

Oh you were talking about Hybrid RAID, not pure software RAID. Performance on those is almost a shot in the dark. Hybrid RAID is just a gimmick . What I was talking about is 100% software RAID that uses no HW controller at all and works with any storage device on any interface (i.e. doesn't bind you to a single controller). Hybrid RAID uses proprietary drivers that manufacturers give little attention, and the drivers are generally shit. Linux's SW RAID drivers are [obviously] open source, have been around for nearly a decade, and don't rely on cheap controller.

crispyfish wrote:

miahallen wrote:

crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...

I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.

Are you talking about the data cable? I'm talking about the power cable... the data cables aren't a problem if they have clips.

Oh, whoops! No, I haven't had any problems with the power connections.

Shusher wrote:Oh you were talking about Hybrid RAID, not pure software RAID. Performance on those is almost a shot in the dark. Hybrid RAID is just a gimmick . What I was talking about is 100% software RAID that uses no HW controller at all and works with any storage device on any interface (i.e. doesn't bind you to a single controller). Hybrid RAID uses proprietary drivers that manufacturers give little attention, and the drivers are generally shit. Linux's SW RAID drivers are [obviously] open source, have been around for nearly a decade, and don't rely on cheap controller.

Interesting. Thanks for the info. I guess I took a "shot in the dark", and hit a brick wall Oh well....like I said, I'm very happy with the current performance I have.

when i look at this picture, i think of its opposite. A super clean apple workstation. You open the case and theres no wires. Ive always wondered who made the superior person, someone who is very clean, or someone who is messy. The clean person would imply organzation on the positive side while pettyness on the negative. A messy person would imply a willingness to focus on the most important aspects as a positive while the negative would be that they may be neglecting the total picture or are inable to organize higher concepts. Well, I'm stoned. forgive me.