Was 3.7TB - then 2.7TB, now 19.1TB "quiet" server

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Was 3.7TB - then 2.7TB, now 19.1TB "quiet" server

Post by Wibla » Mon Jun 04, 2007 9:42 am

Previously my main fileserver, got bumped down to backup server in 2008, now rebuilt with 16x1.5TB drives, undergoing testing to be recertified as main fileserver (and thus bumping the 9.1TB box down to backup duties)

Check the last page for new pictures and more info.

Specs: - UPDATED 21.03.10

Asus P5WDG2-WS Motherboard
Intel Celeron-D 352 (3.2GHz)
4GB PC6400 RAM
Intel PCI-E 1Gbit NIC
Corsair HX520 PSU
3ware 9500S-12 PCI-X Raid controller
16x1.5TB Seagate LP drives
320GB Samsung PATA system drive
Coolermaster Stacker (Classic)
4x Coolermaster 4-in-3
Intake: 3xNexus 120mm fans, 1xScythe S-FLEX on the 4in3's. Exhaust: 1xNoctua 120mm + PSU

(There was alot of outdated text here, removed now. Leaving images for "historic" reasons, also added the major milestone images in this post)

Image
An awful mess during testing, summer 2007..

Image
"In production" - summer 2007, tried to tidy up the cables abit, but basically gave up. 80mm fan added to compensate for lacking performance from the Noctua 120mm I had in the top 4in3 at the time, worked well, but with more noise. Motherboard/CPU/GPU was changed out abit later due to very bad performance from the storage array.

Image
August 2007 - Mobo changeout. (ALOT) better performance, 300+MB/s read, 120MB/s write

Image
Summer 2009 Rebuild - this is after two drives failed inside a week of eachother and I had no choice but to scrap the current array, but thanks to 3ware I didnt loose any data.

Image
October 2009 - Removed 8x500GB drives, added in 16x1.5TB Seagate LP drives and another 4in3. Next up is swapping the 30GB WD drive out for something abit more recent (and quieter, the bearings are whining more by the day..)
Last edited by Wibla on Sun Mar 21, 2010 9:50 am, edited 11 times in total.

thejamppa
Posts: 3142
Joined: Mon Feb 26, 2007 9:20 am
Location: Missing in Finnish wilderness, howling to moon with wolf brethren and walking with brother bears
Contact:

Post by thejamppa » Mon Jun 04, 2007 9:56 am

That certainly is not silent but I am impressed... That many HDD's. Man... I could fit nice part of my anime collection in those HDD's...

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon Jun 04, 2007 10:03 am

It's actually not very noisy at all, there is almost no vibration because the stacker is quite heavy, and the 120mm fans are slow running fans, also the 120mm's in the front are behind dust filters that suck up abit noise, albeit at the cost of some airflow.

The box actually makes less noise than my sister's desktop, which is a 2500+, 6800GT... with a crappy psu, one hdd and a couple of 80mm fans... whahahahaha.

thejamppa
Posts: 3142
Joined: Mon Feb 26, 2007 9:20 am
Location: Missing in Finnish wilderness, howling to moon with wolf brethren and walking with brother bears
Contact:

Post by thejamppa » Mon Jun 04, 2007 10:09 am

I am even more impressed. But I still think that system would be quite noisy for me. Anyway, what's that expansion card beneath VGA card? Extra PCI sata-controller?

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon Jun 04, 2007 10:22 am

its the 3ware 12port SATA Raid card I use for hwraid5 :)

.. obviously its not SILENT, but its definently alot less noisy than expected :)

It will not reside under my desk forever tho, I'm gonna move it down into the basement when I get home next time (5-6 weeks, summer vacation) as it is in the way and heats up the whole area.. 9 hdd's produce a healthy bit of heat...

TK421
Posts: 21
Joined: Fri May 11, 2007 5:45 am

Post by TK421 » Mon Jun 04, 2007 11:40 am

Any Read/Write benchmarks you want to share? :)

JazzJackRabbit
Posts: 1386
Joined: Fri Jun 18, 2004 6:53 pm

Post by JazzJackRabbit » Mon Jun 04, 2007 1:07 pm

I want to build something like this myself, it's just a question of money and space. :D So if you don't mind, I have several questions.
(1) Are you running lunix or windows there?
(2) Do you have it all in a single partition or in several smaller ones. I've read that either windows or many hardware controllers have an issue with partitions larger than 2TB so I was wondering if you've run into this issue or not.

Thanks.

jaldridge6
Posts: 319
Joined: Wed May 09, 2007 2:31 am
Location: Hell

Post by jaldridge6 » Mon Jun 04, 2007 5:15 pm

jezuz!

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon Jun 04, 2007 6:34 pm

Code: Select all

Version   1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP 
Oberon           3G 22128  54 22593   6 15981   5 31603  80 109254  17  89.5   0
3WARE              ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- 
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    83   0 +++++ +++    70   0    83   0 +++++ +++    87   0
Performance is pretty abysmal, the 3ware cards has never been strong performers with regards to write speed on raid5, and the read speed is limited because i only have pci 32bit/33mhz instead of pci-x... I'm currently considering a new mobo+cpu+ram combo to get pci-x and hopefully better performance, but its not at the top of my list as of now, the performance is more than enough for normal storage usage.

I use linux, with XFS filesystem on the raw device, which makes it super easy to grow the fs if i opt to use OCE (online capacity expansion), which the 3ware controller supports without problems...

jalridge6: this isnt an extreme setup, a friend of mine has 4 stackers full of drives now, but he's crazy :P

Here is an example of bonnie results with software raid0 on two HD501LJ drives...:

Code: Select all

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Oberon           3G 36283  95 116958  51 46296  21 33297  94 97197  31 196.3   0
SWRAID              ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2378  98 +++++ +++ +++++ +++  2407  98 +++++ +++  7990  99
Needless to say, my next raid setup will be software based on a C2D box with plenty of AHCI SATA ports ;)

bonestonne
Posts: 1839
Joined: Mon Feb 05, 2007 2:10 pm
Location: Northern New Jersey
Contact:

Post by bonestonne » Mon Jun 04, 2007 7:01 pm

Linux, its got to be, i've never seen windows not a difference between the letter "o" and the number zero. however, linux definitely makes it known.

what distro? i could see it being a Ubuntu variant, but i could easily be mistaken.

if you're shooting for PCI-X, look at used/old low-mid range servers, almost all of them have PCI-X slots...but if there are no normal PCI slots, make sure that you've got compatible cards, it'll become a large problem for you if not.

how effective are the front 120mm fans at preventing silicon pies? i'd think that many HDDs would get quite warm.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Tue Jun 05, 2007 3:19 am

its Debian, I dont use ubuntu..

I'm looking at one of the Asus P5WDG2-WS Pro cards, one with 2xPCI-X and C2D support, with plans of doing a E4300 + 2GB ram setup on it if I decide to upgrade, then I have alot more SATA ports too, for future expansion :)

Bigg
Posts: 154
Joined: Fri Nov 08, 2002 4:05 pm

Post by Bigg » Wed Jun 06, 2007 3:44 pm

Sorry, but I HAVE to ask.... What do you DO with more than 2TB of space? Very nice build though. :D

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sat Jun 09, 2007 8:41 am

Well, I dont like deleting... anything. So stuff tends to accumulate...

I'm not completely happy with the setup, performance is abit lacking, and I need to move the server down into the basement...

jaldridge6
Posts: 319
Joined: Wed May 09, 2007 2:31 am
Location: Hell

Post by jaldridge6 » Sat Jun 09, 2007 11:51 am

wouldnt 8x500GB be 4gb? Why does the topic say 2.8?

Emyr
Posts: 91
Joined: Wed Jun 06, 2007 10:48 am
Location: Cardiff, UK

Post by Emyr » Sat Jun 09, 2007 12:08 pm

jaldridge6 wrote:wouldnt 8x500GB be 4gb? Why does the topic say 2.8?
I would expect a single RAID5 array of 8 x 500GB to have a total capacity of about 3500GB...

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sat Jun 09, 2007 1:50 pm

Because I have 7x500GB in the array + a hotspare, so total usable space is about 2.7-2.8TB :)

miahallen
Posts: 305
Joined: Thu Jan 18, 2007 10:31 pm
Location: Japan

Re: My (not so) quiet 2.8TB filebox

Post by miahallen » Sun Jun 10, 2007 4:54 am

Wibla wrote:Image
Abit worried about drive 6 (twa0_5) and drive 8 (twa0_7), they shouldn't be running so hot.
Those temps are just fine, actually, they're quite low...even the hot ones. Some studies suggest longer lifespan for HDDs running at mid to high temps anyway. So, I wouldn't worry about them.

Nice setup, I've been wanting to build one similar....but with a 3ware off PCIe, so I don't have to get a server mobo for decent throughput.

nick705
Posts: 1162
Joined: Tue Mar 23, 2004 3:26 pm
Location: UK

Re: My (not so) quiet 2.8TB filebox

Post by nick705 » Sun Jun 10, 2007 5:20 am

Wibla wrote:also had to add a 80mm fan to the top drives, used very thick double sided tape that also picks up alot of the vibration from the fan.[/url]
Out of interest, why did you need the additional 80mm fan? Doesn't the 4-in-3 module's 120mm fan keep them cool enough on its own?

/edit: silly question, of course not, or you wouldn't have put the extra fan in. Did you break out the 4-in-3's perforated faceplates between the fan and the drives? They restrict the fan's airflow quite severely...

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun Jun 10, 2007 1:22 pm

I removed all the perforated faceplates, but I still thought the temps was abit high for the topmost 3 drives compared to the bottom ones, so then the 80mm went in ;)

miahallen: PCIE is a good choice, but you could just aswell get an abit aw9 with 9 SATA ports and do swraid in linux, a friend of mine is doing insane speeds on that setup, and with a C2D you wont have cpu load problems.

I'm having real throughput problems with my 3ware setup, most probably because I'm on pci2.0 (32bit/33mhz) + gigabit controller on same pci bus

miahallen
Posts: 305
Joined: Thu Jan 18, 2007 10:31 pm
Location: Japan

Post by miahallen » Sun Jun 10, 2007 5:36 pm

Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea 8)

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Wed Jun 13, 2007 11:08 pm

Waiting abit seems like a good idea, maybe the 1TB drives will be more nicely priced next spring too? ;)

Shusher
Posts: 34
Joined: Mon Apr 23, 2007 6:09 am

Post by Shusher » Thu Jun 14, 2007 11:54 am

miahallen wrote:Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea 8)
What was your issue with software raid? I've been using SW linux raid for years at work and its been nothing but great on SMP systems. I just built a 3-HDD SW raid system for myself (with an ASUS M2N32-SLI which has 6 internal SATA ports) at home and performance is great. Even at full writing power with RAID5 the system doesn't burn much CPU. Comparable HW raid cards would cost close to 1k. I've seen even $500-$800 cards that get outperformed by SW RAID.

Perhaps you were thinking of Windows-based software RAID? Something I have zero experience with but I can easily see problems in that world. =) Last I heard you can't even install the OS on a raid volume.

Oh and as for HD encoding... I do a fair bit of it but disk performance has never been even close to a bottleneck. Writing to disk is nothing, less than 1-5MB/s for anyone encoding HD, and reading from disk is plenty fast (for my typical down-sampling encode, i.e. from the ATSC MPEG2 ~20Mbps to XviD @2-5Mbps). I get an average sequential read of 100-120MiB/s with my raid5 array and encoding does not use up even a quarter of that.

miahallen
Posts: 305
Joined: Thu Jan 18, 2007 10:31 pm
Location: Japan

Software RAID

Post by miahallen » Thu Jun 14, 2007 2:24 pm

Shusher wrote:
miahallen wrote:Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea 8)
What was your issue with software raid? I've been using SW linux raid for years at work and its been nothing but great on SMP systems. I just built a 3-HDD SW raid system for myself (with an ASUS M2N32-SLI which has 6 internal SATA ports) at home and performance is great. Even at full writing power with RAID5 the system doesn't burn much CPU. Comparable HW raid cards would cost close to 1k. I've seen even $500-$800 cards that get outperformed by SW RAID.

Perhaps you were thinking of Windows-based software RAID? Something I have zero experience with but I can easily see problems in that world. =) Last I heard you can't even install the OS on a raid volume.

Oh and as for HD encoding... I do a fair bit of it but disk performance has never been even close to a bottleneck. Writing to disk is nothing, less than 1-5MB/s for anyone encoding HD, and reading from disk is plenty fast (for my typical down-sampling encode, i.e. from the ATSC MPEG2 ~20Mbps to XviD @2-5Mbps). I get an average sequential read of 100-120MiB/s with my raid5 array and encoding does not use up even a quarter of that.
I've used RAID0 for 3 or 4 previos OS installs, and I had various problems in all instances, mainly with stability which I always concluded were due to the RAIDs. So I've given up on them. I never noticed much performance enhancement with them anyway. The benchmarks looked good, but in everyday usage, I never could tell.

In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.

Now I have my OS and apps installed on a 160GB drive, all my games on a 320GB drive....and the other 2 320GB drives I'm using for my media files. I have a second computer (my HTPC) with a single 160GB drive that doubles as a backup device for my critical files. I'm as happy as can be, great performace, and with my various files stratigically located on seperate drives, this is the best everyday performace I've ever had. reason = I can encode media (idle proccess priority) on the media drives, play a game off my gaming drive, and stream a movie from the media drives to my HTPC & laptop simultaniously, without any noticable loss in performance.

crispyfish
Posts: 222
Joined: Tue Nov 21, 2006 10:53 am

Post by crispyfish » Thu Jun 14, 2007 3:56 pm

Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing... :(

Shusher
Posts: 34
Joined: Mon Apr 23, 2007 6:09 am

Re: Software RAID

Post by Shusher » Thu Jun 14, 2007 5:17 pm

miahallen wrote:In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.
Are you doing all this with Linux RAID or Windows RAID? Even my oldest system (2x60GB) outperforms that. Are you sure you were running in UDMA mode (assuming we're talking about ATA disks)? Did you have a really slow system bus? If this is a linux setup, it's more likely to be a misconfiguration than the SW RAID.

miahallen
Posts: 305
Joined: Thu Jan 18, 2007 10:31 pm
Location: Japan

Re: Software RAID

Post by miahallen » Thu Jun 14, 2007 10:06 pm

crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...
I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.
Shusher wrote:
miahallen wrote:In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.
Are you doing all this with Linux RAID or Windows RAID? Even my oldest system (2x60GB) outperforms that. Are you sure you were running in UDMA mode (assuming we're talking about ATA disks)? Did you have a really slow system bus? If this is a linux setup, it's more likely to be a misconfiguration than the SW RAID.
I configured the RAID5 through the RAID controller's (si3114) BIOS setup program. Windows Vista automatically detected it as a single volume. Two drives were WD3200KS and the third is a WD3200AAKS. The entire system specs can be seen in my sig.
And, if you're wondering if all three drives were working, they were. That was the first thing I checked.

crispyfish
Posts: 222
Joined: Tue Nov 21, 2006 10:53 am

Re: Software RAID

Post by crispyfish » Fri Jun 15, 2007 12:20 am

miahallen wrote:
crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...
I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.
Are you talking about the data cable? I'm talking about the power cable... the data cables aren't a problem if they have clips.

Shusher
Posts: 34
Joined: Mon Apr 23, 2007 6:09 am

Re: Software RAID

Post by Shusher » Fri Jun 15, 2007 4:53 am

miahallen wrote:I configured the RAID5 through the RAID controller's (si3114) BIOS setup program. Windows Vista automatically detected it as a single volume. Two drives were WD3200KS and the third is a WD3200AAKS. The entire system specs can be seen in my sig.
And, if you're wondering if all three drives were working, they were. That was the first thing I checked.
Oh you were talking about Hybrid RAID, not pure software RAID. Performance on those is almost a shot in the dark. Hybrid RAID is just a gimmick :D. What I was talking about is 100% software RAID that uses no HW controller at all and works with any storage device on any interface (i.e. doesn't bind you to a single controller). Hybrid RAID uses proprietary drivers that manufacturers give little attention, and the drivers are generally shit. Linux's SW RAID drivers are [obviously] open source, have been around for nearly a decade, and don't rely on cheap controller.

miahallen
Posts: 305
Joined: Thu Jan 18, 2007 10:31 pm
Location: Japan

Re: Software RAID

Post by miahallen » Sun Jun 17, 2007 12:48 am

crispyfish wrote:
miahallen wrote:
crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...
I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.
Are you talking about the data cable? I'm talking about the power cable... the data cables aren't a problem if they have clips.
Oh, whoops! No, I haven't had any problems with the power connections.
Shusher wrote:Oh you were talking about Hybrid RAID, not pure software RAID. Performance on those is almost a shot in the dark. Hybrid RAID is just a gimmick . What I was talking about is 100% software RAID that uses no HW controller at all and works with any storage device on any interface (i.e. doesn't bind you to a single controller). Hybrid RAID uses proprietary drivers that manufacturers give little attention, and the drivers are generally shit. Linux's SW RAID drivers are [obviously] open source, have been around for nearly a decade, and don't rely on cheap controller.
Interesting. Thanks for the info. I guess I took a "shot in the dark", and hit a brick wall :roll: Oh well....like I said, I'm very happy with the current performance I have.

jaldridge6
Posts: 319
Joined: Wed May 09, 2007 2:31 am
Location: Hell

Post by jaldridge6 » Wed Jun 20, 2007 1:36 am

when i look at this picture, i think of its opposite. A super clean apple workstation. You open the case and theres no wires. Ive always wondered who made the superior person, someone who is very clean, or someone who is messy. The clean person would imply organzation on the positive side while pettyness on the negative. A messy person would imply a willingness to focus on the most important aspects as a positive while the negative would be that they may be neglecting the total picture or are inable to organize higher concepts. Well, I'm stoned. forgive me.

Post Reply