The Monster! (last updated: 25.02.10 - power usage numbers)
Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee
The Monster! (last updated: 25.02.10 - power usage numbers)
This is my new fileserver, named "Monster" because its the biggest and most expensive build of mine yet...
Specs:
Coolermaster Stacker Classic + 4in3 modules
Asus P5WDG2-WS
Intel 3GHz Celeron
Intel PCI-E Gig-E network
Xigmatek cpu cooler (coming soon)
4GB PC2-6400 Corsair ram
3ware 9500S-12 PCI-X RAID controller
12x1TB Samsung F1 (11x1TB raid5 + hotspare)
Seagate 80GB SATA 2.5" 7200rpm system drive
Nexus 120mm fans for the 4in3 modules
Corsair HX520 PSU
Pics: (the important bit)
Noise:
Honestly, the noise level IS NOT bad, im impressed so far, there was some vibration...
I swapped out the Coolermaster fan with an Antec Tricool in the back, removed the restrictive fan grill, and put in Nexus 120mm on the 4in3 modules... no fan controller as of yet, except the onboard asus stuff.
Performance numbers will be posted when the box is up and running, still initializing the raid array now.
Specs:
Coolermaster Stacker Classic + 4in3 modules
Asus P5WDG2-WS
Intel 3GHz Celeron
Intel PCI-E Gig-E network
Xigmatek cpu cooler (coming soon)
4GB PC2-6400 Corsair ram
3ware 9500S-12 PCI-X RAID controller
12x1TB Samsung F1 (11x1TB raid5 + hotspare)
Seagate 80GB SATA 2.5" 7200rpm system drive
Nexus 120mm fans for the 4in3 modules
Corsair HX520 PSU
Pics: (the important bit)
Noise:
Honestly, the noise level IS NOT bad, im impressed so far, there was some vibration...
I swapped out the Coolermaster fan with an Antec Tricool in the back, removed the restrictive fan grill, and put in Nexus 120mm on the 4in3 modules... no fan controller as of yet, except the onboard asus stuff.
Performance numbers will be posted when the box is up and running, still initializing the raid array now.
Last edited by Wibla on Sat Feb 27, 2010 9:02 am, edited 4 times in total.
Not quite..
11 drives in raid5 + hotspare, so...
Code: Select all
/dev/sdb 9.1T 1.1M 9.1T 1% /mnt/storage
-
- Posts: 1839
- Joined: Mon Feb 05, 2007 2:10 pm
- Location: Northern New Jersey
- Contact:
I dont really need the Quad core at all, and for some reason i cant run 64bit linux and get network connectivity with the onboard gigabit, so im gonna put the Q6600 in my ws, and put an E2140 in the fileserver, with HWraid its not an issue anyway
The only reason I got the Q6600 was to run folding@home on it aswell, but with no 64bit I cant run SMP, so no point in having it in there...
I was abit sceptical about the 2.5" at first, but it works fine, so no worries then...
2.5" drive:
The only reason I got the Q6600 was to run folding@home on it aswell, but with no 64bit I cant run SMP, so no point in having it in there...
I was abit sceptical about the 2.5" at first, but it works fine, so no worries then...
2.5" drive:
Code: Select all
# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 8246 MB in 2.00 seconds = 4129.83 MB/sec
Timing buffered disk reads: 146 MB in 3.00 seconds = 48.60 MB/sec
.....That's......impressive......(should I say overkill?) .......
If you apply "P.L.O.H"* rule, then that's alot of freekin porn.
* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.
.....
.......
......... .... ............
(resized)bonestonne wrote:If you apply "P.L.O.H"* rule, then that's alot of freekin porn.
* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.
If you apply "P.L.O.H"* rule, then that's alot of freekin porn.
* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.
.....
.......
......... .... ............
-
- SPCR Reviewer
- Posts: 561
- Joined: Tue May 30, 2006 8:22 pm
- Location: Vancouver, BC
Looking at your pictures, I'm guessing you have the same problem I've been having in my system - SATA power cables. I've also resorted to using molex->SATA adaptors for a few of my drives, but I really wish that there were a few 4-port SATA power cables for the Corsair PSUs.
Looks like a nice setup.
Looks like a nice setup.
-
- Posts: 1839
- Joined: Mon Feb 05, 2007 2:10 pm
- Location: Northern New Jersey
- Contact:
-
- SPCR Reviewer
- Posts: 561
- Joined: Tue May 30, 2006 8:22 pm
- Location: Vancouver, BC
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.Cistron wrote:I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
Thats why i chose the F1 drives, performance and pretty low noise... that box is very quiet for what it is... a big hunking monster...
(too bad write performance is lacking for now.. I might have to mail both samsung and AMCC/3ware about this, as its not supposed to be like this at all!)
Some lessons learned from Oberon to this box:
- Know what port is the first one, and how they're numbered...
in Oberon, drive 1 and port0 != the same, drive1 at the top = port1, drive2 = port0, etc.. big confusion, and a big headache when something breaks...
in monster, drive0 and port0 = same, and the order is kept for all the drives, cables are numbered accordingly, with drive0 at the top and drive11 at the bottom...
- Think through how the cabling is supposed to be laid, especially the power cables... Spent atleast an hour doing only the power cables, but I got some good dividends...
As a result of a few precautions and some experience, monster is alot neater built than oberon, even tho im running 13 drives and a DVDRW in monster, compared to 11 drives and no DVDRW in oberon.
(too bad write performance is lacking for now.. I might have to mail both samsung and AMCC/3ware about this, as its not supposed to be like this at all!)
Some lessons learned from Oberon to this box:
- Know what port is the first one, and how they're numbered...
in Oberon, drive 1 and port0 != the same, drive1 at the top = port1, drive2 = port0, etc.. big confusion, and a big headache when something breaks...
in monster, drive0 and port0 = same, and the order is kept for all the drives, cables are numbered accordingly, with drive0 at the top and drive11 at the bottom...
- Think through how the cabling is supposed to be laid, especially the power cables... Spent atleast an hour doing only the power cables, but I got some good dividends...
As a result of a few precautions and some experience, monster is alot neater built than oberon, even tho im running 13 drives and a DVDRW in monster, compared to 11 drives and no DVDRW in oberon.
-
- SPCR Reviewer
- Posts: 561
- Joined: Tue May 30, 2006 8:22 pm
- Location: Vancouver, BC
@wibla - Are you sure you've set the controller to StorSave-Performance mode? I found that I get slow write speeds when it's on Protection or Balance.
Do you use the 3DM2 to administer your settings or the command line utility?
What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup.
Do you use the 3DM2 to administer your settings or the command line utility?
What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup.
I'm a big fan of the KIS(S) principle for cooling, front to back cooling is what works best for servers in my experience, and with 3x120mm intake and 2x120mm (rear + psu) exhaust it should be decently balanced.porkchop wrote:have you tried running the system without the exhaust and what appears to be a top exhaust?
considering you have 3 intakes it should work fairly well- the less fans to control the better, right?
Ive tested with both bonnie++ and dd if=/dev/zero of=bigfile bs=1M count=4096 etcNick Geraedts wrote:@wibla - Are you sure you've set the controller to StorSave-Performance mode? I found that I get slow write speeds when it's on Protection or Balance.
Do you use the 3DM2 to administer your settings or the command line utility?
What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup.
There is no way to set storsave mode on the 9500S as far as I can see? This hasnt been an issue on my other fileserver.
Im using both 3DM2 and tw_cli, 3DM2 for managment, tw_cli to check status etc...
Performance numbers:
3x1TB in md raid0, exported from 3ware as single drives:
bonnie++: 37-40MB/s write, 188MB/s read
dd: 40MB/s write, 4GB file
11x1TB, raid5:
bonnie++: 37-40MB/s write, 365MB/s read
dd: 40MB/s write, 4GB file
6x1TB, raid5:
bonnie++: 37-40MB/s write, 250MB/s read
dd: 40MB/s write, 4GB file
6x1TB, raid0:
dd: 40MB/s write, 4GB file (!!!)
I'll have to test thatGator wrote:Beware - don't set that box on a slippery surface or the torque from all those HD's will make the entire chassis spin in the opposite direction the HD's rotate!!
I already have a 3.7TB array in my old fileserver, and its full... so I had to expand... I filled it up in a year, nuff said..AckeDman wrote:WOW!... thats huge fucking build. I dont really understand what u will use 12TB of space for thou. I have 1 750gb drive and i dont think i will be able to fill it up in quiet some time.
I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
And surely I'm not the only one that finds those stupid molex connectors to be the most annoying damn things to unplug? They should come with some kind of oil to make it easierThe SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.
Ahh, thank you. I'm still annoyed that all SATA power connectors are daisy-chained on my Pro82+.Nick Geraedts wrote:The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.Cistron wrote:I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
Just out of curiosity Wibla, what does one save on 11TB? Are you in the imaging business?
-
- SPCR Reviewer
- Posts: 561
- Joined: Tue May 30, 2006 8:22 pm
- Location: Vancouver, BC
Ouch... that's painfully slow, even with the 6x1TB RAID0 setup (which should be the fastest). I'll agree - a call to 3ware is definitely in order. Are you sure that the F1's are on the supported drives list? Are you reusing the same 9500s card as was in your previous build, or is it a new card?Wibla wrote:Performance numbers:
3x1TB in md raid0, exported from 3ware as single drives:
bonnie++: 37-40MB/s write, 188MB/s read
dd: 40MB/s write, 4GB file
11x1TB, raid5:
bonnie++: 37-40MB/s write, 365MB/s read
dd: 40MB/s write, 4GB file
6x1TB, raid5:
bonnie++: 37-40MB/s write, 250MB/s read
dd: 40MB/s write, 4GB file
6x1TB, raid0:
dd: 40MB/s write, 4GB file (!!!)
Under Management->Controller settings is where I find the following page:
You don't have anything similar for your card?
Yes, but I do not have the queuing and Storsave options, the controller isnt new, but one i got off fleabay as a spare for the one in my old fileserver.
I have already contacted 3ware and there's an open support ticket.
For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.
I have already contacted 3ware and there's an open support ticket.
For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.
-
- SPCR Reviewer
- Posts: 561
- Joined: Tue May 30, 2006 8:22 pm
- Location: Vancouver, BC
An easy way to test that is to put the card in a PCI (non-X) slot. You'll be limited to 133MB/s bandwidth, but you should still be able to write at higher speeds than what you're getting now.Wibla wrote:For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.
-
- Posts: 213
- Joined: Thu Mar 20, 2008 8:51 am
- Location: In the server room, playing Trackmania
WOW. I love you're whole set up. I never thought I would need more than my 80Gb Samsung Spinpoint, but after getting into DVD ripping for a streaming media server, I'm almost filling up my first 250Gb. It's nice to see what I'll be doing in 5 years.
Just curious what OS you're using and the inevitable... why might one need so much data? (and the "because-I-can" answer is perfectly OK).
Just curious what OS you're using and the inevitable... why might one need so much data? (and the "because-I-can" answer is perfectly OK).