Was 3.7TB - then 2.7TB, now 19.1TB "quiet" server
Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee
Was 3.7TB - then 2.7TB, now 19.1TB "quiet" server
Previously my main fileserver, got bumped down to backup server in 2008, now rebuilt with 16x1.5TB drives, undergoing testing to be recertified as main fileserver (and thus bumping the 9.1TB box down to backup duties)
Check the last page for new pictures and more info.
Specs: - UPDATED 21.03.10
Asus P5WDG2-WS Motherboard
Intel Celeron-D 352 (3.2GHz)
4GB PC6400 RAM
Intel PCI-E 1Gbit NIC
Corsair HX520 PSU
3ware 9500S-12 PCI-X Raid controller
16x1.5TB Seagate LP drives
320GB Samsung PATA system drive
Coolermaster Stacker (Classic)
4x Coolermaster 4-in-3
Intake: 3xNexus 120mm fans, 1xScythe S-FLEX on the 4in3's. Exhaust: 1xNoctua 120mm + PSU
(There was alot of outdated text here, removed now. Leaving images for "historic" reasons, also added the major milestone images in this post)
An awful mess during testing, summer 2007..
"In production" - summer 2007, tried to tidy up the cables abit, but basically gave up. 80mm fan added to compensate for lacking performance from the Noctua 120mm I had in the top 4in3 at the time, worked well, but with more noise. Motherboard/CPU/GPU was changed out abit later due to very bad performance from the storage array.
August 2007 - Mobo changeout. (ALOT) better performance, 300+MB/s read, 120MB/s write
Summer 2009 Rebuild - this is after two drives failed inside a week of eachother and I had no choice but to scrap the current array, but thanks to 3ware I didnt loose any data.
October 2009 - Removed 8x500GB drives, added in 16x1.5TB Seagate LP drives and another 4in3. Next up is swapping the 30GB WD drive out for something abit more recent (and quieter, the bearings are whining more by the day..)
Check the last page for new pictures and more info.
Specs: - UPDATED 21.03.10
Asus P5WDG2-WS Motherboard
Intel Celeron-D 352 (3.2GHz)
4GB PC6400 RAM
Intel PCI-E 1Gbit NIC
Corsair HX520 PSU
3ware 9500S-12 PCI-X Raid controller
16x1.5TB Seagate LP drives
320GB Samsung PATA system drive
Coolermaster Stacker (Classic)
4x Coolermaster 4-in-3
Intake: 3xNexus 120mm fans, 1xScythe S-FLEX on the 4in3's. Exhaust: 1xNoctua 120mm + PSU
(There was alot of outdated text here, removed now. Leaving images for "historic" reasons, also added the major milestone images in this post)
An awful mess during testing, summer 2007..
"In production" - summer 2007, tried to tidy up the cables abit, but basically gave up. 80mm fan added to compensate for lacking performance from the Noctua 120mm I had in the top 4in3 at the time, worked well, but with more noise. Motherboard/CPU/GPU was changed out abit later due to very bad performance from the storage array.
August 2007 - Mobo changeout. (ALOT) better performance, 300+MB/s read, 120MB/s write
Summer 2009 Rebuild - this is after two drives failed inside a week of eachother and I had no choice but to scrap the current array, but thanks to 3ware I didnt loose any data.
October 2009 - Removed 8x500GB drives, added in 16x1.5TB Seagate LP drives and another 4in3. Next up is swapping the 30GB WD drive out for something abit more recent (and quieter, the bearings are whining more by the day..)
Last edited by Wibla on Sun Mar 21, 2010 9:50 am, edited 11 times in total.
It's actually not very noisy at all, there is almost no vibration because the stacker is quite heavy, and the 120mm fans are slow running fans, also the 120mm's in the front are behind dust filters that suck up abit noise, albeit at the cost of some airflow.
The box actually makes less noise than my sister's desktop, which is a 2500+, 6800GT... with a crappy psu, one hdd and a couple of 80mm fans... whahahahaha.
The box actually makes less noise than my sister's desktop, which is a 2500+, 6800GT... with a crappy psu, one hdd and a couple of 80mm fans... whahahahaha.
its the 3ware 12port SATA Raid card I use for hwraid5
.. obviously its not SILENT, but its definently alot less noisy than expected
It will not reside under my desk forever tho, I'm gonna move it down into the basement when I get home next time (5-6 weeks, summer vacation) as it is in the way and heats up the whole area.. 9 hdd's produce a healthy bit of heat...
.. obviously its not SILENT, but its definently alot less noisy than expected
It will not reside under my desk forever tho, I'm gonna move it down into the basement when I get home next time (5-6 weeks, summer vacation) as it is in the way and heats up the whole area.. 9 hdd's produce a healthy bit of heat...
-
- Posts: 1386
- Joined: Fri Jun 18, 2004 6:53 pm
I want to build something like this myself, it's just a question of money and space. So if you don't mind, I have several questions.
(1) Are you running lunix or windows there?
(2) Do you have it all in a single partition or in several smaller ones. I've read that either windows or many hardware controllers have an issue with partitions larger than 2TB so I was wondering if you've run into this issue or not.
Thanks.
(1) Are you running lunix or windows there?
(2) Do you have it all in a single partition or in several smaller ones. I've read that either windows or many hardware controllers have an issue with partitions larger than 2TB so I was wondering if you've run into this issue or not.
Thanks.
Code: Select all
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Oberon 3G 22128 54 22593 6 15981 5 31603 80 109254 17 89.5 0
3WARE ------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 83 0 +++++ +++ 70 0 83 0 +++++ +++ 87 0
I use linux, with XFS filesystem on the raw device, which makes it super easy to grow the fs if i opt to use OCE (online capacity expansion), which the 3ware controller supports without problems...
jalridge6: this isnt an extreme setup, a friend of mine has 4 stackers full of drives now, but he's crazy
Here is an example of bonnie results with software raid0 on two HD501LJ drives...:
Code: Select all
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Oberon 3G 36283 95 116958 51 46296 21 33297 94 97197 31 196.3 0
SWRAID ------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2378 98 +++++ +++ +++++ +++ 2407 98 +++++ +++ 7990 99
-
- Posts: 1839
- Joined: Mon Feb 05, 2007 2:10 pm
- Location: Northern New Jersey
- Contact:
Linux, its got to be, i've never seen windows not a difference between the letter "o" and the number zero. however, linux definitely makes it known.
what distro? i could see it being a Ubuntu variant, but i could easily be mistaken.
if you're shooting for PCI-X, look at used/old low-mid range servers, almost all of them have PCI-X slots...but if there are no normal PCI slots, make sure that you've got compatible cards, it'll become a large problem for you if not.
how effective are the front 120mm fans at preventing silicon pies? i'd think that many HDDs would get quite warm.
what distro? i could see it being a Ubuntu variant, but i could easily be mistaken.
if you're shooting for PCI-X, look at used/old low-mid range servers, almost all of them have PCI-X slots...but if there are no normal PCI slots, make sure that you've got compatible cards, it'll become a large problem for you if not.
how effective are the front 120mm fans at preventing silicon pies? i'd think that many HDDs would get quite warm.
-
- Posts: 319
- Joined: Wed May 09, 2007 2:31 am
- Location: Hell
Re: My (not so) quiet 2.8TB filebox
Those temps are just fine, actually, they're quite low...even the hot ones. Some studies suggest longer lifespan for HDDs running at mid to high temps anyway. So, I wouldn't worry about them.Wibla wrote:
Abit worried about drive 6 (twa0_5) and drive 8 (twa0_7), they shouldn't be running so hot.
Nice setup, I've been wanting to build one similar....but with a 3ware off PCIe, so I don't have to get a server mobo for decent throughput.
Re: My (not so) quiet 2.8TB filebox
Out of interest, why did you need the additional 80mm fan? Doesn't the 4-in-3 module's 120mm fan keep them cool enough on its own?Wibla wrote:also had to add a 80mm fan to the top drives, used very thick double sided tape that also picks up alot of the vibration from the fan.[/url]
/edit: silly question, of course not, or you wouldn't have put the extra fan in. Did you break out the 4-in-3's perforated faceplates between the fan and the drives? They restrict the fan's airflow quite severely...
I removed all the perforated faceplates, but I still thought the temps was abit high for the topmost 3 drives compared to the bottom ones, so then the 80mm went in
miahallen: PCIE is a good choice, but you could just aswell get an abit aw9 with 9 SATA ports and do swraid in linux, a friend of mine is doing insane speeds on that setup, and with a C2D you wont have cpu load problems.
I'm having real throughput problems with my 3ware setup, most probably because I'm on pci2.0 (32bit/33mhz) + gigabit controller on same pci bus
miahallen: PCIE is a good choice, but you could just aswell get an abit aw9 with 9 SATA ports and do swraid in linux, a friend of mine is doing insane speeds on that setup, and with a C2D you wont have cpu load problems.
I'm having real throughput problems with my 3ware setup, most probably because I'm on pci2.0 (32bit/33mhz) + gigabit controller on same pci bus
Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea
What was your issue with software raid? I've been using SW linux raid for years at work and its been nothing but great on SMP systems. I just built a 3-HDD SW raid system for myself (with an ASUS M2N32-SLI which has 6 internal SATA ports) at home and performance is great. Even at full writing power with RAID5 the system doesn't burn much CPU. Comparable HW raid cards would cost close to 1k. I've seen even $500-$800 cards that get outperformed by SW RAID.miahallen wrote:Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea
Perhaps you were thinking of Windows-based software RAID? Something I have zero experience with but I can easily see problems in that world. =) Last I heard you can't even install the OS on a raid volume.
Oh and as for HD encoding... I do a fair bit of it but disk performance has never been even close to a bottleneck. Writing to disk is nothing, less than 1-5MB/s for anyone encoding HD, and reading from disk is plenty fast (for my typical down-sampling encode, i.e. from the ATSC MPEG2 ~20Mbps to XviD @2-5Mbps). I get an average sequential read of 100-120MiB/s with my raid5 array and encoding does not use up even a quarter of that.
Software RAID
I've used RAID0 for 3 or 4 previos OS installs, and I had various problems in all instances, mainly with stability which I always concluded were due to the RAIDs. So I've given up on them. I never noticed much performance enhancement with them anyway. The benchmarks looked good, but in everyday usage, I never could tell.Shusher wrote:What was your issue with software raid? I've been using SW linux raid for years at work and its been nothing but great on SMP systems. I just built a 3-HDD SW raid system for myself (with an ASUS M2N32-SLI which has 6 internal SATA ports) at home and performance is great. Even at full writing power with RAID5 the system doesn't burn much CPU. Comparable HW raid cards would cost close to 1k. I've seen even $500-$800 cards that get outperformed by SW RAID.miahallen wrote:Well, SW RAID sound good on paper, but my experiance has steered me away from another attempt. I'd be willing to give it another go, if it was a "known" good setup. (Then again, for serving my media files, performance isn't quite as importtant (at least not until I start encoding 1080p video files)
But, I'm not yet ready for this build, so I'm not going to stress it quite yet. I'm thinking closer to next spring. But thanks for the idea
Perhaps you were thinking of Windows-based software RAID? Something I have zero experience with but I can easily see problems in that world. =) Last I heard you can't even install the OS on a raid volume.
Oh and as for HD encoding... I do a fair bit of it but disk performance has never been even close to a bottleneck. Writing to disk is nothing, less than 1-5MB/s for anyone encoding HD, and reading from disk is plenty fast (for my typical down-sampling encode, i.e. from the ATSC MPEG2 ~20Mbps to XviD @2-5Mbps). I get an average sequential read of 100-120MiB/s with my raid5 array and encoding does not use up even a quarter of that.
In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.
Now I have my OS and apps installed on a 160GB drive, all my games on a 320GB drive....and the other 2 320GB drives I'm using for my media files. I have a second computer (my HTPC) with a single 160GB drive that doubles as a backup device for my critical files. I'm as happy as can be, great performace, and with my various files stratigically located on seperate drives, this is the best everyday performace I've ever had. reason = I can encode media (idle proccess priority) on the media drives, play a game off my gaming drive, and stream a movie from the media drives to my HTPC & laptop simultaniously, without any noticable loss in performance.
-
- Posts: 222
- Joined: Tue Nov 21, 2006 10:53 am
Re: Software RAID
Are you doing all this with Linux RAID or Windows RAID? Even my oldest system (2x60GB) outperforms that. Are you sure you were running in UDMA mode (assuming we're talking about ATA disks)? Did you have a really slow system bus? If this is a linux setup, it's more likely to be a misconfiguration than the SW RAID.miahallen wrote:In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.
Re: Software RAID
I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...
I configured the RAID5 through the RAID controller's (si3114) BIOS setup program. Windows Vista automatically detected it as a single volume. Two drives were WD3200KS and the third is a WD3200AAKS. The entire system specs can be seen in my sig.Shusher wrote:Are you doing all this with Linux RAID or Windows RAID? Even my oldest system (2x60GB) outperforms that. Are you sure you were running in UDMA mode (assuming we're talking about ATA disks)? Did you have a really slow system bus? If this is a linux setup, it's more likely to be a misconfiguration than the SW RAID.miahallen wrote:In my last setup I had three Wester Digital 320GB SATA2 16MB drives in a RAID5 off the Silicon Image controller on my A8N-SLI motherboard. Performance was TERRIBLE! I had the OS installed on a single 160GB drive, and everything else on the RAID5....if I was playing a game and attempting to stream a movie to my HTPC or laptop at the same time, the movie would stutter. Even the benchmards showed very poor performance - read speeds averaged about 15MB/s, write speeds averages 10MB/s.
And, if you're wondering if all three drives were working, they were. That was the first thing I checked.
-
- Posts: 222
- Joined: Tue Nov 21, 2006 10:53 am
Re: Software RAID
Are you talking about the data cable? I'm talking about the power cable... the data cables aren't a problem if they have clips.miahallen wrote:I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...
Re: Software RAID
Oh you were talking about Hybrid RAID, not pure software RAID. Performance on those is almost a shot in the dark. Hybrid RAID is just a gimmick . What I was talking about is 100% software RAID that uses no HW controller at all and works with any storage device on any interface (i.e. doesn't bind you to a single controller). Hybrid RAID uses proprietary drivers that manufacturers give little attention, and the drivers are generally shit. Linux's SW RAID drivers are [obviously] open source, have been around for nearly a decade, and don't rely on cheap controller.miahallen wrote:I configured the RAID5 through the RAID controller's (si3114) BIOS setup program. Windows Vista automatically detected it as a single volume. Two drives were WD3200KS and the third is a WD3200AAKS. The entire system specs can be seen in my sig.
And, if you're wondering if all three drives were working, they were. That was the first thing I checked.
Re: Software RAID
Oh, whoops! No, I haven't had any problems with the power connections.crispyfish wrote:Are you talking about the data cable? I'm talking about the power cable... the data cables aren't a problem if they have clips.miahallen wrote:I have had similar problems, but I've replace all my old OEM SATA cable with better quality ones,...it helps a bit.crispyfish wrote:Do you ever have any problems with SATA power cables disconnecting from the drives? I've found SATA power cables to be horribly unreliable. I prefer the solid connection you get with molex connectors, but IDE drives are gradually disappearing...
Interesting. Thanks for the info. I guess I took a "shot in the dark", and hit a brick wall Oh well....like I said, I'm very happy with the current performance I have.Shusher wrote:Oh you were talking about Hybrid RAID, not pure software RAID. Performance on those is almost a shot in the dark. Hybrid RAID is just a gimmick . What I was talking about is 100% software RAID that uses no HW controller at all and works with any storage device on any interface (i.e. doesn't bind you to a single controller). Hybrid RAID uses proprietary drivers that manufacturers give little attention, and the drivers are generally shit. Linux's SW RAID drivers are [obviously] open source, have been around for nearly a decade, and don't rely on cheap controller.
-
- Posts: 319
- Joined: Wed May 09, 2007 2:31 am
- Location: Hell
when i look at this picture, i think of its opposite. A super clean apple workstation. You open the case and theres no wires. Ive always wondered who made the superior person, someone who is very clean, or someone who is messy. The clean person would imply organzation on the positive side while pettyness on the negative. A messy person would imply a willingness to focus on the most important aspects as a positive while the negative would be that they may be neglecting the total picture or are inable to organize higher concepts. Well, I'm stoned. forgive me.