The Monster! (last updated: 25.02.10 - power usage numbers)

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

The Monster! (last updated: 25.02.10 - power usage numbers)

Post by Wibla » Sat May 24, 2008 2:17 pm

This is my new fileserver, named "Monster" because its the biggest and most expensive build of mine yet...

Specs:

Coolermaster Stacker Classic + 4in3 modules
Asus P5WDG2-WS
Intel 3GHz Celeron
Intel PCI-E Gig-E network
Xigmatek cpu cooler (coming soon)
4GB PC2-6400 Corsair ram
3ware 9500S-12 PCI-X RAID controller
12x1TB Samsung F1 (11x1TB raid5 + hotspare)
Seagate 80GB SATA 2.5" 7200rpm system drive
Nexus 120mm fans for the 4in3 modules
Corsair HX520 PSU

Pics: (the important bit)

Image

Image

Image

Image

Image

Image

Image

Image

Image

Image

Noise:

Honestly, the noise level IS NOT bad, im impressed so far, there was some vibration...

I swapped out the Coolermaster fan with an Antec Tricool in the back, removed the restrictive fan grill, and put in Nexus 120mm on the 4in3 modules... no fan controller as of yet, except the onboard asus stuff.

Image

Performance numbers will be posted when the box is up and running, still initializing the raid array now.
Last edited by Wibla on Sat Feb 27, 2010 9:02 am, edited 4 times in total.

Moogles
Posts: 315
Joined: Thu Mar 22, 2007 10:28 am

Post by Moogles » Sat May 24, 2008 2:27 pm

12TB? Are you trying to download the internet?

Good luck. :D

Looks good btw!

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sat May 24, 2008 3:08 pm

Not quite..

Code: Select all

/dev/sdb              9.1T  1.1M  9.1T   1% /mnt/storage
11 drives in raid5 + hotspare, so...

bonestonne
Posts: 1839
Joined: Mon Feb 05, 2007 2:10 pm
Location: Northern New Jersey
Contact:

Post by bonestonne » Sat May 24, 2008 3:15 pm

If you apply "P.L.O.H"* rule, then that's alot of freekin porn.

* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.


just a Q6600?

clever hiding that 2.5" all the way up top

i'm a little surprised myself that one PSU handles 13 drives + quad.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sat May 24, 2008 4:16 pm

I dont really need the Quad core at all, and for some reason i cant run 64bit linux and get network connectivity with the onboard gigabit, so im gonna put the Q6600 in my ws, and put an E2140 in the fileserver, with HWraid its not an issue anyway :-)

The only reason I got the Q6600 was to run folding@home on it aswell, but with no 64bit I cant run SMP, so no point in having it in there...
I was abit sceptical about the 2.5" at first, but it works fine, so no worries then...

2.5" drive:

Code: Select all

# hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   8246 MB in  2.00 seconds = 4129.83 MB/sec
 Timing buffered disk reads:  146 MB in  3.00 seconds =  48.60 MB/sec

chahahc
Posts: 75
Joined: Tue Oct 23, 2007 7:52 pm
Location: The Most Mexican Place in the U.S.A

Post by chahahc » Sat May 24, 2008 7:01 pm

:shock: .....That's......impressive......(should I say overkill?) :P .......

bonestonne wrote:If you apply "P.L.O.H"* rule, then that's alot of freekin porn.

* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.

(resized)

If you apply "P.L.O.H"* rule, then that's alot of freekin porn.

* Porno Law of Hardrive = .5 X total harddrive space = amount of porn files.



.....
.......
......... :? ....:| ............ :mrgreen:

Modo
Posts: 486
Joined: Wed Apr 16, 2008 3:32 am
Location: Poland

Post by Modo » Sat May 24, 2008 9:50 pm

bonestonne wrote: i'm a little surprised myself that one PSU handles 13 drives + quad.
Drives pull very little power, so that's about the equivalent of the Quad plus just one high-end graphics card. Should be no problem for a high-efficiency unit. :)

Big SturL
Posts: 38
Joined: Sun Mar 23, 2008 9:29 am
Location: Molde, Norway

Post by Big SturL » Sun May 25, 2008 12:00 am

Nice build there! But I'm happy to buy you some zip-ties for those SATA-cables.

Nick Geraedts
SPCR Reviewer
Posts: 561
Joined: Tue May 30, 2006 8:22 pm
Location: Vancouver, BC

Post by Nick Geraedts » Sun May 25, 2008 12:44 am

Looking at your pictures, I'm guessing you have the same problem I've been having in my system - SATA power cables. I've also resorted to using molex->SATA adaptors for a few of my drives, but I really wish that there were a few 4-port SATA power cables for the Corsair PSUs.

Looks like a nice setup. :)

m^2
Posts: 146
Joined: Mon Jan 29, 2007 2:12 am
Location: Poland
Contact:

Post by m^2 » Sun May 25, 2008 1:23 am

Where did you put the 2.5 drive? Can't find it on the pics..

Cistron
Posts: 618
Joined: Fri Mar 14, 2008 5:18 am
Location: London, UK

Post by Cistron » Sun May 25, 2008 2:28 am

Nick Geraedts wrote:Looking at your pictures, I'm guessing you have the same problem I've been having in my system - SATA power cables.
I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.

porkchop
Posts: 496
Joined: Thu May 25, 2006 1:19 am
Location: Australia

Post by porkchop » Sun May 25, 2008 3:05 am

sata power cables have one extra wire- surely it must have some sort of use :evil:
i think i remember something about hot-swapping sata hdds?

bonestonne
Posts: 1839
Joined: Mon Feb 05, 2007 2:10 pm
Location: Northern New Jersey
Contact:

Post by bonestonne » Sun May 25, 2008 6:38 am

m^2 wrote:Where did you put the 2.5 drive? Can't find it on the pics..
it took me a minute too. its all the way up top. look for a black SATA cable coming off the motherboard, and you'll see it snake all the way up, and then over to the whereabouts of the optical drive.

thejamppa
Posts: 3142
Joined: Mon Feb 26, 2007 9:20 am
Location: Missing in Finnish wilderness, howling to moon with wolf brethren and walking with brother bears
Contact:

Post by thejamppa » Sun May 25, 2008 7:26 am

Is this the SPCR version find the Waldo? Instead: Find the 2,5" HDD?

Anyway, case is awesome looking and that is indeed a monster. 11 TB's sweet! I could use such space ^^. That could take my anime collection like 2,5 times ^^

Nick Geraedts
SPCR Reviewer
Posts: 561
Joined: Tue May 30, 2006 8:22 pm
Location: Vancouver, BC

Post by Nick Geraedts » Sun May 25, 2008 11:47 am

Cistron wrote:I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun May 25, 2008 3:49 pm

I'm puzzled as to the shitty performance im getting with the 9500S on 12x1TB... 40MB/s tops write speed on 11 or 6 drives in raid5, and top read speed of about 365-370MB/s... the other box with 9x500GB will do 100+MB/s write easily.. :(

wim
Posts: 777
Joined: Wed Apr 28, 2004 5:16 am
Location: canberra, australia

Post by wim » Sun May 25, 2008 4:36 pm

PLOH.. lol..

i hate hard disk noise... 12HD must sound horrible ... :(

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun May 25, 2008 5:30 pm

Thats why i chose the F1 drives, performance and pretty low noise... that box is very quiet for what it is... a big hunking monster... :D

(too bad write performance is lacking for now.. I might have to mail both samsung and AMCC/3ware about this, as its not supposed to be like this at all!)

Some lessons learned from Oberon to this box:

- Know what port is the first one, and how they're numbered...
in Oberon, drive 1 and port0 != the same, drive1 at the top = port1, drive2 = port0, etc.. big confusion, and a big headache when something breaks...
in monster, drive0 and port0 = same, and the order is kept for all the drives, cables are numbered accordingly, with drive0 at the top and drive11 at the bottom...

- Think through how the cabling is supposed to be laid, especially the power cables... Spent atleast an hour doing only the power cables, but I got some good dividends...

As a result of a few precautions and some experience, monster is alot neater built than oberon, even tho im running 13 drives and a DVDRW in monster, compared to 11 drives and no DVDRW in oberon.

porkchop
Posts: 496
Joined: Thu May 25, 2006 1:19 am
Location: Australia

Post by porkchop » Sun May 25, 2008 8:33 pm

have you tried running the system without the exhaust and what appears to be a top exhaust?
considering you have 3 intakes it should work fairly well- the less fans to control the better, right?

Nick Geraedts
SPCR Reviewer
Posts: 561
Joined: Tue May 30, 2006 8:22 pm
Location: Vancouver, BC

Post by Nick Geraedts » Sun May 25, 2008 9:29 pm

@wibla - Are you sure you've set the controller to StorSave-Performance mode? I found that I get slow write speeds when it's on Protection or Balance.

Do you use the 3DM2 to administer your settings or the command line utility?

What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup. :)

Gator
Posts: 16
Joined: Thu Jul 26, 2007 2:16 pm
Location: Gainesville, FL - USA

Post by Gator » Sun May 25, 2008 9:50 pm

Beware - don't set that box on a slippery surface or the torque from all those HD's will make the entire chassis spin in the opposite direction the HD's rotate!! :D :roll:

AckeDman
Posts: 150
Joined: Wed Apr 02, 2008 12:20 am
Location: Sweden

Post by AckeDman » Sun May 25, 2008 11:50 pm

WOW!... thats huge fucking build. I dont really understand what u will use 12TB of space for thou. I have 1 750gb drive and i dont think i will be able to fill it up in quiet some time.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon May 26, 2008 12:05 am

porkchop wrote:have you tried running the system without the exhaust and what appears to be a top exhaust?
considering you have 3 intakes it should work fairly well- the less fans to control the better, right?
I'm a big fan of the KIS(S) principle for cooling, front to back cooling is what works best for servers in my experience, and with 3x120mm intake and 2x120mm (rear + psu) exhaust it should be decently balanced.
Nick Geraedts wrote:@wibla - Are you sure you've set the controller to StorSave-Performance mode? I found that I get slow write speeds when it's on Protection or Balance.

Do you use the 3DM2 to administer your settings or the command line utility?

What are you using to test the write performance? I've always just tried writing dummy files from /dev/zero using dd. You can easily test batch performance for small files or large files with that kind of setup. :)
Ive tested with both bonnie++ and dd if=/dev/zero of=bigfile bs=1M count=4096 etc

There is no way to set storsave mode on the 9500S as far as I can see? This hasnt been an issue on my other fileserver.

Im using both 3DM2 and tw_cli, 3DM2 for managment, tw_cli to check status etc...

Performance numbers:

3x1TB in md raid0, exported from 3ware as single drives:

bonnie++: 37-40MB/s write, 188MB/s read
dd: 40MB/s write, 4GB file

11x1TB, raid5:

bonnie++: 37-40MB/s write, 365MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid5:

bonnie++: 37-40MB/s write, 250MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid0:

dd: 40MB/s write, 4GB file (!!!)
Gator wrote:Beware - don't set that box on a slippery surface or the torque from all those HD's will make the entire chassis spin in the opposite direction the HD's rotate!! :D :roll:
I'll have to test that :lol:
AckeDman wrote:WOW!... thats huge fucking build. I dont really understand what u will use 12TB of space for thou. I have 1 750gb drive and i dont think i will be able to fill it up in quiet some time.
I already have a 3.7TB array in my old fileserver, and its full... so I had to expand... I filled it up in a year, nuff said..

josh-j
Posts: 11
Joined: Sat May 17, 2008 9:13 am
Location: UK

Post by josh-j » Mon May 26, 2008 2:49 am

I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.
And surely I'm not the only one that finds those stupid molex connectors to be the most annoying damn things to unplug? They should come with some kind of oil to make it easier :)

Cistron
Posts: 618
Joined: Fri Mar 14, 2008 5:18 am
Location: London, UK

Post by Cistron » Mon May 26, 2008 6:24 am

Nick Geraedts wrote:
Cistron wrote:I'm completely puzzled as to why the design has been changed from Molex at all. Probably just to make everybody's life a little more complicated.
The SATA power connector was actually a much needed improvement over molex, adding SATA-only capabilities (such as hot swapping), as well as improving the electrical security of the setup. Have a look here for more information.
Ahh, thank you. I'm still annoyed that all SATA power connectors are daisy-chained on my Pro82+.

Just out of curiosity Wibla, what does one save on 11TB? Are you in the imaging business?

Nick Geraedts
SPCR Reviewer
Posts: 561
Joined: Tue May 30, 2006 8:22 pm
Location: Vancouver, BC

Post by Nick Geraedts » Mon May 26, 2008 8:43 am

Wibla wrote:Performance numbers:

3x1TB in md raid0, exported from 3ware as single drives:

bonnie++: 37-40MB/s write, 188MB/s read
dd: 40MB/s write, 4GB file

11x1TB, raid5:

bonnie++: 37-40MB/s write, 365MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid5:

bonnie++: 37-40MB/s write, 250MB/s read
dd: 40MB/s write, 4GB file

6x1TB, raid0:

dd: 40MB/s write, 4GB file (!!!)
Ouch... that's painfully slow, even with the 6x1TB RAID0 setup (which should be the fastest). I'll agree - a call to 3ware is definitely in order. Are you sure that the F1's are on the supported drives list? Are you reusing the same 9500s card as was in your previous build, or is it a new card?

Under Management->Controller settings is where I find the following page:
Image
You don't have anything similar for your card?

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon May 26, 2008 9:06 am

Yes, but I do not have the queuing and Storsave options, the controller isnt new, but one i got off fleabay as a spare for the one in my old fileserver.

I have already contacted 3ware and there's an open support ticket.

For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.

Nick Geraedts
SPCR Reviewer
Posts: 561
Joined: Tue May 30, 2006 8:22 pm
Location: Vancouver, BC

Post by Nick Geraedts » Mon May 26, 2008 11:16 am

Wibla wrote:For now, I suspect the 3ware controller doesnt want to play with the motherboard PCI-X controller on the X38 based mobo, so I've ordered an open box P5WDG2-WS (same as in the old board) and I'm getting an old celeron 3GHz from a friend, I hope that will work better.
An easy way to test that is to put the card in a PCI (non-X) slot. You'll be limited to 133MB/s bandwidth, but you should still be able to write at higher speeds than what you're getting now.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon May 26, 2008 1:58 pm

I'm gonna test that this weekend :)

that Linux guy
Posts: 213
Joined: Thu Mar 20, 2008 8:51 am
Location: In the server room, playing Trackmania

Post by that Linux guy » Mon May 26, 2008 2:11 pm

WOW. I love you're whole set up. I never thought I would need more than my 80Gb Samsung Spinpoint, but after getting into DVD ripping for a streaming media server, I'm almost filling up my first 250Gb. It's nice to see what I'll be doing in 5 years.

Just curious what OS you're using and the inevitable... why might one need so much data? (and the "because-I-can" answer is perfectly OK).

Post Reply