Page 1 of 3

The Monster! (last updated: 25.02.10 - power usage numbers)

Posted: Sat May 24, 2008 2:17 pm
by Wibla
This is my new fileserver, named "Monster" because its the biggest and most expensive build of mine yet...

Specs:

Coolermaster Stacker Classic + 4in3 modules
Asus P5WDG2-WS
Intel 3GHz Celeron
Intel PCI-E Gig-E network
Xigmatek cpu cooler (coming soon)
4GB PC2-6400 Corsair ram
3ware 9500S-12 PCI-X RAID controller
12x1TB Samsung F1 (11x1TB raid5 + hotspare)
Seagate 80GB SATA 2.5" 7200rpm system drive
Nexus 120mm fans for the 4in3 modules
Corsair HX520 PSU

Pics: (the important bit)

Image

Image

Image

Image

Image

Image

Image

Image

Image

Image

Noise:

Honestly, the noise level IS NOT bad, im impressed so far, there was some vibration...

I swapped out the Coolermaster fan with an Antec Tricool in the back, removed the restrictive fan grill, and put in Nexus 120mm on the 4in3 modules... no fan controller as of yet, except the onboard asus stuff.

Image

Performance numbers will be posted when the box is up and running, still initializing the raid array now.

Posted: Sat May 24, 2008 2:27 pm
by Moogles
12TB? Are you trying to download the internet?

Good luck. :D

Looks good btw!

Posted: Sat May 24, 2008 3:08 pm
by Wibla
Not quite..

Code: Select all

/dev/sdb              9.1T  1.1M  9.1T   1% /mnt/storage
11 drives in raid5 + hotspare, so...

Posted: Sat May 24, 2008 3:15 pm
by bonestonne
If you apply "P.L.O.H"* rule, then that's alot of freekin porn.

* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.


just a Q6600?

clever hiding that 2.5" all the way up top

i'm a little surprised myself that one PSU handles 13 drives + quad.

Posted: Sat May 24, 2008 4:16 pm
by Wibla
I dont really need the Quad core at all, and for some reason i cant run 64bit linux and get network connectivity with the onboard gigabit, so im gonna put the Q6600 in my ws, and put an E2140 in the fileserver, with HWraid its not an issue anyway :-)

The only reason I got the Q6600 was to run folding@home on it aswell, but with no 64bit I cant run SMP, so no point in having it in there...
I was abit sceptical about the 2.5" at first, but it works fine, so no worries then...

2.5" drive:

Code: Select all

# hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   8246 MB in  2.00 seconds = 4129.83 MB/sec
 Timing buffered disk reads:  146 MB in  3.00 seconds =  48.60 MB/sec

Posted: Sat May 24, 2008 7:01 pm
by chahahc
:shock: .....That's......impressive......(should I say overkill?) :P .......

bonestonne wrote:If you apply "P.L.O.H"* rule, then that's alot of freekin porn.

* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.

(resized)

If you apply "P.L.O.H"* rule, then that's alot of freekin porn.

* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.



.....
.......
......... :? ....:| ............ :mrgreen:

Posted: Sat May 24, 2008 9:50 pm
by Modo
bonestonne wrote: i'm a little surprised myself that one PSU handles 13 drives + quad.
Drives pull very little power, so that's about the equivalent of the Quad plus just one high-end graphics card. Should be no problem for a high-efficiency unit. :)

Posted: Sun May 25, 2008 12:00 am
by Big SturL
Nice build there! But I'm happy to buy you some zip-ties for those SATA-cables.

Posted: Sun May 25, 2008 12:44 am
by Nick Geraedts
Looking at your pictures, I'm guessing you have the same problem I've been having in my system - SATA power cables. I've also resorted to using molex->SATA adaptors for a few of my drives, but I really wish that there were a few 4-port SATA power cables for the Corsair PSUs.

Looks like a nice setup. :)

Posted: Sun May 25, 2008 1:23 am
by m^2
Where did you put the 2.5 drive? Can't find it on the pics..

Posted: Sun May 25, 2008 2:28 am
by Cistron
Nick Geraedts wrote:Looking at your pictures, I'm guessing you have the same problem I've been having in my system - SATA power cables.
I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.

Posted: Sun May 25, 2008 3:05 am
by porkchop
sata power cables have one extra wire- surely it must have some sort of use :evil:
i think i remember something about hot-swapping sata hdds?

Posted: Sun May 25, 2008 6:38 am
by bonestonne
m^2 wrote:Where did you put the 2.5 drive? Can't find it on the pics..
it took me a minute too. its all the way up top. look for a black SATA cable coming off the motherboard, and you'll see it snake all the way up, and then over to the whereabouts of the optical drive.

Posted: Sun May 25, 2008 7:26 am
by thejamppa
Is this the SPCR version find the Waldo? Instead: Find the 2,5" HDD?

Anyway, case is awesome looking and that is indeed a monster. 11 TB's sweet! I could use such space ^^. That could take my anime collection like 2,5 times ^^

Posted: Sun May 25, 2008 11:47 am
by Nick Geraedts
Cistron wrote:I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.

Posted: Sun May 25, 2008 3:49 pm
by Wibla
I'm puzzled as to the shitty performance im getting with the 9500S on 12x1TB... 40MB/s tops write speed on 11 or 6 drives in raid5, and top read speed of about 365-370MB/s... the other box with 9x500GB will do 100+MB/s write easily.. :(

Posted: Sun May 25, 2008 4:36 pm
by wim
PLOH.. lol..

i hate hard disk noise... 12HD must sound horrible ... :(

Posted: Sun May 25, 2008 5:30 pm
by Wibla
Thats why i chose the F1 drives, performance and pretty low noise... that box is very quiet for what it is... a big hunking monster... :D

(too bad write performance is lacking for now.. I might have to mail both samsung and AMCC/3ware about this, as its not supposed to be like this at all!)

Some lessons learned from Oberon to this box:

- Know what port is the first one, and how they're numbered...
in Oberon, drive 1 and port0 != the same, drive1 at the top = port1, drive2 = port0, etc.. big confusion, and a big headache when something breaks...
in monster, drive0 and port0 = same, and the order is kept for all the drives, cables are numbered accordingly, with drive0 at the top and drive11 at the bottom...

- Think through how the cabling is supposed to be laid, especially the power cables... Spent atleast an hour doing only the power cables, but I got some good dividends...

As a result of a few precautions and some experience, monster is alot neater built than oberon, even tho im running 13 drives and a DVDRW in monster, compared to 11 drives and no DVDRW in oberon.

Posted: Sun May 25, 2008 8:33 pm
by porkchop
have you tried running the system without the exhaust and what appears to be a top exhaust?
considering you have 3 intakes it should work fairly well- the less fans to control the better, right?

Posted: Sun May 25, 2008 9:29 pm
by Nick Geraedts
@wibla - Are you sure you've set the controller to StorSave-Performance mode? I found that I get slow write speeds when it's on Protection or Balance.

Do you use the 3DM2 to administer your settings or the command line utility?

What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup. :)

Posted: Sun May 25, 2008 9:50 pm
by Gator
Beware - don't set that box on a slippery surface or the torque from all those HD's will make the entire chassis spin in the opposite direction the HD's rotate!! :D :roll:

Posted: Sun May 25, 2008 11:50 pm
by AckeDman
WOW!... thats huge fucking build. I dont really understand what u will use 12TB of space for thou. I have 1 750gb drive and i dont think i will be able to fill it up in quiet some time.

Posted: Mon May 26, 2008 12:05 am
by Wibla
porkchop wrote:have you tried running the system without the exhaust and what appears to be a top exhaust?
considering you have 3 intakes it should work fairly well- the less fans to control the better, right?
I'm a big fan of the KIS(S) principle for cooling, front to back cooling is what works best for servers in my experience, and with 3x120mm intake and 2x120mm (rear + psu) exhaust it should be decently balanced.
Nick Geraedts wrote:@wibla - Are you sure you've set the controller to StorSave-Performance mode? I found that I get slow write speeds when it's on Protection or Balance.

Do you use the 3DM2 to administer your settings or the command line utility?

What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup. :)
Ive tested with both bonnie++ and dd if=/dev/zero of=bigfile bs=1M count=4096 etc

There is no way to set storsave mode on the 9500S as far as I can see? This hasnt been an issue on my other fileserver.

Im using both 3DM2 and tw_cli, 3DM2 for managment, tw_cli to check status etc...

Performance numbers:

3x1TB in md raid0, exported from 3ware as single drives:

bonnie++: 37-40MB/s write, 188MB/s read
dd: 40MB/s write, 4GB file

11x1TB, raid5:

bonnie++: 37-40MB/s write, 365MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid5:

bonnie++: 37-40MB/s write, 250MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid0:

dd: 40MB/s write, 4GB file (!!!)
Gator wrote:Beware - don't set that box on a slippery surface or the torque from all those HD's will make the entire chassis spin in the opposite direction the HD's rotate!! :D :roll:
I'll have to test that :lol:
AckeDman wrote:WOW!... thats huge fucking build. I dont really understand what u will use 12TB of space for thou. I have 1 750gb drive and i dont think i will be able to fill it up in quiet some time.
I already have a 3.7TB array in my old fileserver, and its full... so I had to expand... I filled it up in a year, nuff said..

Posted: Mon May 26, 2008 2:49 am
by josh-j
I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.
And surely I'm not the only one that finds those stupid molex connectors to be the most annoying damn things to unplug? They should come with some kind of oil to make it easier :)

Posted: Mon May 26, 2008 6:24 am
by Cistron
Nick Geraedts wrote:
Cistron wrote:I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.
Ahh, thank you. I'm still annoyed that all SATA power connectors are daisy-chained on my Pro82+.

Just out of curiosity Wibla, what does one save on 11TB? Are you in the imaging business?

Posted: Mon May 26, 2008 8:43 am
by Nick Geraedts
Wibla wrote:Performance numbers:

3x1TB in md raid0, exported from 3ware as single drives:

bonnie++: 37-40MB/s write, 188MB/s read
dd: 40MB/s write, 4GB file

11x1TB, raid5:

bonnie++: 37-40MB/s write, 365MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid5:

bonnie++: 37-40MB/s write, 250MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid0:

dd: 40MB/s write, 4GB file (!!!)
Ouch... that's painfully slow, even with the 6x1TB RAID0 setup (which should be the fastest). I'll agree - a call to 3ware is definitely in order. Are you sure that the F1's are on the supported drives list? Are you reusing the same 9500s card as was in your previous build, or is it a new card?

Under Management->Controller settings is where I find the following page:
Image
You don't have anything similar for your card?

Posted: Mon May 26, 2008 9:06 am
by Wibla
Yes, but I do not have the queuing and Storsave options, the controller isnt new, but one i got off fleabay as a spare for the one in my old fileserver.

I have already contacted 3ware and there's an open support ticket.

For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.

Posted: Mon May 26, 2008 11:16 am
by Nick Geraedts
Wibla wrote:For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.
An easy way to test that is to put the card in a PCI (non-X) slot. You'll be limited to 133MB/s bandwidth, but you should still be able to write at higher speeds than what you're getting now.

Posted: Mon May 26, 2008 1:58 pm
by Wibla
I'm gonna test that this weekend :)

Posted: Mon May 26, 2008 2:11 pm
by that Linux guy
WOW. I love you're whole set up. I never thought I would need more than my 80Gb Samsung Spinpoint, but after getting into DVD ripping for a streaming media server, I'm almost filling up my first 250Gb. It's nice to see what I'll be doing in 5 years.

Just curious what OS you're using and the inevitable... why might one need so much data? (and the "because-I-can" answer is perfectly OK).