The quiet beast: dual Xeon and Quadro 4000 workstation

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

The quiet beast: dual Xeon and Quadro 4000 workstation

Post by pm.stacker » Wed Jan 27, 2010 2:37 pm

Image

***UPDATED***

Case CM Stacker STC-T01 steel, DIY modified with original side window
CPU dual Intel Xeon E5520 processor, Nehalem-EP, total 8 cores (16 logical), 2.27 GHz, TDP 80W, 35W idle, Turbo Boost to 2.4 GHz (all cores loaded, 95W TDP)
CPUs cooling Prolimatech Samuel-17 (adapted to Xeon mount, basically treads removed and bolts reversed)
Motherboard Tyan S7025AGM2NR (no onboard SAS), dual Intel 5520 northbridges, ICH10R southbridge connected to NB#2
System memory 16384 MB, 8x2048 MB Corsair DDR3-1333 ECC registered
Graphics card Nvidia Quadro 4000 on X16 slot no.1
Onboard audio card Realtek ALC262, 2 channels plus coax SPDIF header
LAN dual Gigabit Intel 82574L controllers
Firewire card Combo Firewire800-USB2 Pci-Express, X16 slot no.2 (X1 mode)
Storage controller 1 Promise EX8650 8-port SAS controller on X16 slot no.4 (X8 mode)
Storage controller 2 Integrated AHCI 6-port ICH10 controller (for 6 optical drives)
Storage controller 3 Asus U3S6 combo USB3.0 (D720200) and SATA3 (Marvell 88SE9123) on X8 slot (X4 mode, PLX bridge)
ODD0 MATSHITA DVD-RAM SW-9576S (internal, SATA300 converter)
ODD1 NEC ND-4551A (internal, SATA300 converter)
ODD2 HL-DT-ST BD-RE BH10LS30 (external eSATA-SFF8088)
ODD3 HL-DT-ST BD-RE GGW-H20L (external eSATA-SFF8088)
ODD4 Optiarc BD-RW BD-5740H (external eSATA-SFF8088)
ODD5 Optiarc BD-RW BD-5730S (external eSATA-SFF8088)
SSD card OCZ RevodriveX2 (100 GB) on X16 slot no.3 (X4 mode), main OS
SSD drive OCZ Vertex2-E 60 GB (die size: 32Gb), virtual OS
HDD0-3 WDC WD10EADS
HDD4-7 WDC WD5003ABYX
Power supply Enermax Revolution85+ 850W (24P+8P+8P)
Storage controller fan Papst 612 (Noctua ULNA)
HDD fans Papst 8412-N2GLE x2
CPU and exhaust fans Scythe Slipstream SY1225SL12LM-P PWM x3
Chipset fan Titan TFD-7010M2B ball (Noctua ULNA)

Main OS Windows 7 Ultimate 64-bit
Virtual OS Windows XP Media Center Edition 2005
A/V Software Adobe Premiere Pro CS5 v5.0.3 x64 (Mercury playback engine, GPU acceleration enabled)

Main monitor Samsung Syncmaster 245B (DVI-D), adapted VESA-100x100 to VESA-200x100 for main desktop
Secondary monitor Eizo Flexscan S2243W (Displayport) VESA-100x100 for VM desktop and full-screen HD realtime video preview
Monitor placing Ergotron LX-SBS dual arm, modified
Main printer HP Business Inkjet 1100dtn, networked
Photos and CD-DVD printer Canon Pixma IP4300
Card reader generic Transcend/Hama reader modified and enclosed into light controller
Backup system DINIC Taurus Mini LCM external drive, USB 2.0, 2x WDC WD5000BEVT, RAID 1
Image acquisition Canon CanoScan LIDE 700F
HDD quick-docking station Icy Box IB-110StU3 PC-powered USB 3.0
Desktop speakers Creative I-Trigue 3000 side-mounted on main monitor
Aux speakers integrated Eizo monitor speakers, Displayport audio, for VM sound
Power conditioning APC Smart-UPS 1500, networked, fan grille removed

Assets volume for music, footage, pictures: 1300.01 GB RAID10
Finished products volume for finished videos ready to burn or store in HTPC: 640.00 GB RAID0
Temporary volume for BD rips, HD intermediate uncompressed video: 1858.00 GB RAID0

I built this workstation mainly for HD/bluray video editing and producing, upgrading from a single Q6600 Core2... :mrgreen: For HD video some large storage space is required, that's why I configured the four 500GB drives as a 2.0TB RAID0 volume with 400MBps sustained speed. I replaced the original Stacker front panel with a DIY modified Silverstone USB 3.0 front panel adding power-reset buttons and a SSD behind :wink:.
The Tyan S7025 motherboard is Tyan's top of the line product for dual Xeon 5500 series platforms, specifically targetted to workstations rather than servers, with its two northbridges and four full X16 PCI-E 2.0 slots, equally spaced that can accomodate four dual-slot cards. The top two X16 slots along with the X4 slot (connected to NB#1 to which also the ICH10R is connected) are dedicated to graphic cards and I/O funcionality, since the DMI connection to the ICH makes it hard to get SATA controllers to work (not enough chipset memory to allocate BIOS). The bottom two slots and the X8 (X4 link) slot are ideal for some serious storage works, not having other onboard components connected to: with the right slot placing I was able to get to work a good old Promise EX8650 SAS controller together with the SSD card (SiI3124) and the Asus U3S6 card which has the Marvell 88SE9123 BIOS, all the three BIOSes are loaded and work in addition to the ICH10R AHCI BIOS :-D
SSD card (X4 1.1) and SAS controller (X8 1.1) can work at their max bandwidth, as can the U3S6 card (2.0 X4), I still have to verify if the X4 PLX PCI-E bridge Asus used grants a performance improvement in USB 3.0 throughput when compared to a normal X1 card directly connected, given the massive bandwidth this board has to offer on its PCI-E slots :mrgreen:

Bye everyone! :-D

***UPDATED*** removed the very first pics, go to page 3 for the latest config :wink:

Average H264-to-X264 encoding speed: 72 fps (ffmpeg-mt, limited by CPU power)
Average VC1-to-X264 encoding speed: 28 fps (due to single-thread VC1 decoder)
Average MPEG2-to-X264 or RGB-to-X264 encoding speed: 70 fps (yet to test it)
Last edited by pm.stacker on Sun Jun 05, 2011 2:28 am, edited 17 times in total.

psiu
Posts: 1201
Joined: Tue Aug 23, 2005 1:53 pm
Location: SE MI

Post by psiu » Wed Jan 27, 2010 4:36 pm

That's some serious horsepower.

The 4x500GB drives, I take it you use those as work drives, not for storage of finished product?
And the northbridge blowers, would it be possible to replace them with bigger standard NB coolers (they look like they are designed to fit into the conventional rackmount cases) with no fans? Otherwise it looks pretty reasonable for all the components in the case.

bonestonne
Posts: 1839
Joined: Mon Feb 05, 2007 2:10 pm
Location: Northern New Jersey
Contact:

Post by bonestonne » Wed Jan 27, 2010 5:03 pm

that's a pretty monstrous system, only the second one i've ever seen with 16 cores (technicalities)..other one was a quad socket Xeon setup with 20gb of RAM.

I'd probably test it out with some slipstreams, maybe knock out those smaller fans, and see how air pressure keeps it cool.

What do you use for editing? Adobe? firewire decks i'm guessing too?

pretty impressive to say the least.

Ha, good to see a really powerful system still not using 1kW....I wonder if you could go even lower safely? 700W PSU?

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:p

Post by pm.stacker » Thu Jan 28, 2010 3:55 am

@psiu
About the HDDs
Since uncompressed RGB24 HDV editing (1440x1080) needs creation of temporary files (to pass them to the MPEG2 DVD and H264 BD encoders) that take up ~400 GB/hour, a 2.0TB "swapspace" is the least I can have to edit at least 2-3 hours of footage without filling up the whole system :lol:. When I don't do HD editing, that same space holds the temp. data for DVD and BD backup to Matroska, to put them into my HTPC. Of course the final HD video is written on a "mixed mode DVD", a standard DVD-Video with the SD downscaling and a DVD-ROM content, that's the MKV HD video in both 720p and 1080i using a AVCHD compliant encoding mode, so it can be used to author an AVCHD compliant DVD-ROM disc :wink:

About the chipset HSFs
Since the S7025 has the two northbridges placed where they would interfere with full length expansion cards (i.e. GTX295), originally they had passive only heatsinks which are very short in height, to accomodate these long cards. Upon testing the system I found out the chipsets heat up too much for my taste (80 °C) and to be sure to keep them cool enough I just took out the CPU fans from an old dual Opteron server and DIY stuck 'em on the NBs undervolting (stock speed was 6000 rpm). 1000 rpm is enough to keep the chipsets cool with silence (blowers are inaudible outside of the closed system). Using blowers instead of standard axial fans directs the air towards the rear, I tried an axial 60x60x20 fan but it would end up heating the RAM modules directly over NB#2. Until these recycled blowers work... so far so good :wink:. I put a small passive heatsink (taken from an ICH6 :-D) on the ICH10R too.
And besides, the northbridges footprint is fairly wide (more than X58) so I'd have to find a 60x60 chipset cooler, maybe recycle some old PIII ones :lol: but then it would interfere with the graphics card (had to bend the unused fan connector to fit) :wink:

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Jan 28, 2010 4:13 am

@bonestonne
Slipstreams are difficult to find here, Enermax Everest series (UCEV8/9/12) are among the most silent fans I was able to find, apart the rotation noise (which is very very low anyway :wink:) the good thing is they don't hum and don't produce vibrations, so silencing them with rubber mounts isn't always necessary. And they have the integrated thermal sensor on a 15" cable to put it where's needed. Maybe the rear exhaust is a little under-sized, but since UCEVs ain't available in 120x120x38 size, I had no choice. Tried the old SilenX IXP76 big fan (120x120x38) but it was way noisier at all speeds :roll:.
If you're talking about the HDD fans, the Chieftec backplanes are made to accomodate a standard 80x80 fan, so here too I had no choice that swap the original ones with silent ones. Verax blue fans are no longer made, they cost something like $70 each in 2003 when I bought them, but since then the (dual ball) bearings never failed and are still like new, they don't vibrate and don't hum, maybe a tiny whining due to the many-magnets motor but from outside they're inaudible I could of couse mount them to rubber grommets, we'll see :-D I'll need grommets for close-frame fans anyway :wink:. The storage controller 60mm Verax is the same as its 80mm siblings, inaudible and thermally controlled.

For editing I use Premiere Pro CS4 64-bit mode. Last year I equipped my Sony Z1 camera with the CF recording drive so the 1394 link is no longer necessary (stick the CF directly into the USB cardreader :-D), but since I have a consumer HDV camera too, Firewire ain't history yet :wink:

Ha, for this build constraint was not power, but connectors: Enermax Revolution85+ and Corsair HX1000 are the only non-server PSUs that have native 24P+dual 8P connectors. I could've derived the second 8P from a PEG 8P aux line (there's an adapter for that), but the PSU I had before (Enermax Infiniti720) would've needed a fan swap since the original was noisy as hell :-x so I bought a brand new 850W (the smallest Enermax makes in the Revo85+ line) and since it comes with only one Molex modular cable, and I need more to power the backplanes, I modded the Infiniti modu cables putting the Revo connector to them :-D

Cheers :wink: hey, if you want I have a lot of photos regarding the building of this thing :-D

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Jan 28, 2010 4:52 am

Here we go

First tryouts with Intel stock HSFs
Image

Wiring the SPDIF header to a BNC socket
Image

Modding the Stacker mobo tray for the USB ports
Image

Dismounting it to drill off some unused holes that prevented the DVI port from being used in slot 1
Image
Image
Image
Image

First backplane tests
Image

Fitting the NB blowers
Image

Mounting the Verax KPE sensor to the inside of the Velociraptors backplane
Image
Image

Verax KPV to the 4-bay
Image

Rear exhaust sensor inside CPU0 HSF
Image

PSU in place
Image

First clearance tests
Image
Image

Air flow entrance
Image

Promise BIOS
Image

Front audio header used to get a 2ch rear line out (integrated display speakers)
Image

Aux PEG 6-pin power used to obtain the 12V external line, 20A max
Image

SATA internal ODDs cabling
Image

Enermax UCEV8 in the middle backplane
Image

Cable management behind the Lianli panelImage
Image

Firewire400 front port cabling behind mobo tray
Image

Front panel door opened
Image

Yeah, with all those things in the way I can still mount expansion cards :-D
Image

:-)

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Jan 28, 2010 6:10 am

On with some tests

Temperatures in idle
Image

Velociraptors RAID group
Array0, RAID0, 64K stripes, 128 GB (2x64)
Image
Array1, RAID1, 73 GB
Image

2x WD10EACS RAID group
Array2, RAID1, 681 GB
Image
Array3, RAID0 in the end of disks, 128K stripe size, 500 GB
Image

4x500G RAID group (2x WD5000AACS, 2x WD5000AADS), 1 MB stripe size, tested with 8 MB chunk size, since HD files use large chunks
Image

I know, my Promise EX8650 card is not the best one, and I should exchange those two WD5000AACS from 2008 with new 1-platter AADS, since they slow down the array... And maybe drop those EACS for some EARS (64 MB cache) :wink:

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Jan 28, 2010 7:07 am

Some load tests
Image
The 60°C peak is due to the fans thermal response

Linpack output
Image

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun Jan 31, 2010 6:53 pm

wow, hardware porn...

Edit: you could try removing that fan mesh cover on the rear exhaust fan, I've done that to both my stackers, should yield significantly better airflow for the exhaust..

Image

neumein
Posts: 66
Joined: Tue May 20, 2008 5:06 am
Location: Launceston, Australia

Post by neumein » Wed Feb 03, 2010 5:23 am

...
...
...
... Yep. I just creamed my shorts :shock:

That is one hell of a sweet looking machine. I'm jealous, jealous to the point of literally turning green. :D

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Fri Feb 05, 2010 5:22 am

Here are some real-world performances :-D

System running a 2-core XP32 VM with vmware7 in addition to encode

From MPEG-2 DVD to anamorphic H264 (576p) using X264 x64 version (x264_64.exe)

1st pass turbo
Image
Image

2nd pass (hey, 1st pass wasn't that turbo :lol:)
Image
Image

derelyth
Posts: 16
Joined: Sat Nov 07, 2009 4:54 pm
Location: England (but NOT London!) UK

Post by derelyth » Mon Feb 08, 2010 12:12 pm

That is one hell of a system - some can only dream! :twisted:

Quick random Q, but on your Coretemp I see it displays the power usage TDP - is that a plugin or an option that needs enabling by default? I have downloaded CT but it does not show for my (puny) Intel E6550 - or is it only a Xeon thing?

Cheers :)

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Feb 11, 2010 10:25 am

Maybe it displays TDP only for newer processors (i.e. core5/7) which have the new speedstep tech. with turbo mode... :roll: who knows. BTW I run coretemp 64-bit version :wink:

bonestonne
Posts: 1839
Joined: Mon Feb 05, 2007 2:10 pm
Location: Northern New Jersey
Contact:

Post by bonestonne » Thu Feb 11, 2010 4:06 pm

odd, i see similar frame rates running DVDFlick on my Core 2 Quad...not HD or anything though, so maybe there is a massive difference..

funny how it only seems to use half the cores (be it every other core, or one CPU and not the other).

you could probably get away with less, but higher pressure fans...but if it's quiet enough for you, that's what matters.

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Sun Feb 14, 2010 3:45 am

It depends on which X264 encoding preset you use, and how speedy is decoding and processing of the source video. I encode in X264 2-pass mode "extra quality" which is only 1 step below the old 2-pass "insane" setting. If I use the new CQ presets speed boosts up to 2-300 fps, but I have less control over file size (which you know by specifying the bitrate directly in 2pass). That being said, X264 in its standard x86/x64 builds (the ones MeGUI uses) still ain't optimized for more than 4-8 cores, maybe lately it's been optimized for quad HT Core7 which show 8 threads, but since the max. threads it can allocate is 16, maybe with the new 6-cores i7 we'll see some improvements :wink: for now, I can live with that, bear in mind: my old Q6600 CPU in the same encoding situation gave me always no more than 10-12 fps... :lol: now a DVD backup is done in 40 minutes and since I maintain 720x576 anamorphic ratio (no cropping and no resizing, I know it tends to waste space but I tend to make my backups the most standard possible :wink:), the MPEG2 decoding phase takes up very little CPU. With HD, I resize 1920x1080 to 1440x1080 or 1280x720 so that takes up more power, but there's plenty of it :lol:
As for the fans, I'm happy with the ones I put in, remember this is no little silent desktop :mrgreen:, and I can see remotely if they run or if they've failed, except the HDD ones, for those there's front red LEDs and audible alarms :-)
Image

bonestonne
Posts: 1839
Joined: Mon Feb 05, 2007 2:10 pm
Location: Northern New Jersey
Contact:

Post by bonestonne » Sun Feb 14, 2010 6:04 am

silly question, but by looking at your internals, i wonder....

does the Thermalright HR-05 fit on your chipsets with those nasty blower fans?

If they do, you could get maybe two of the SLI models, and face them towards each other and stick one undervolted 80mm there for even better temps.

just me being curious...

thejamppa
Posts: 3142
Joined: Mon Feb 26, 2007 9:20 am
Location: Missing in Finnish wilderness, howling to moon with wolf brethren and walking with brother bears
Contact:

Post by thejamppa » Sun Feb 14, 2010 6:42 am

Uhm... wow :shock: That is sweet build! Congrats of nice rig. You've made a lot modifications that are very impressive. Thank you for sharing ^^

derelyth
Posts: 16
Joined: Sat Nov 07, 2009 4:54 pm
Location: England (but NOT London!) UK

Re: :P

Post by derelyth » Sun Feb 14, 2010 7:00 am

pm.stacker wrote:Maybe it displays TDP only for newer processors (i.e. core5/7) which have the new speedstep tech. with turbo mode... :roll: who knows. BTW I run coretemp 64-bit version :wink:
Ahh, could be - I have tried it on both my desktop and my server (E6550 and E3200 respectively) and enither shows TDP. Thanks for the reply though :)

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Mon Mar 01, 2010 3:54 am

Today some first HD encoding tests with meGUI 0.3.4.0 and X264_64.exe :-D
On H264->H264 1920x1080->1440x1080 it makes slightly more than 1X, 26 fps first pass, let's see 2nd pass... :-) big improvement over Q6600 :lol:

Avalanche
Posts: 137
Joined: Mon May 29, 2006 6:29 am
Location: IN, USA
Contact:

Post by Avalanche » Mon Mar 01, 2010 9:37 am

Sweet rig, thanks for sharing! The hardware is obviously awesome, and your DIY work is very neat. The result looks great.

Sometimes people show off their high powered rig and the only comments are, "lol, no way that things quite." Glad to see that's not the case here.

Parappaman
Posts: 419
Joined: Mon Aug 31, 2009 12:30 am
Location: Italy

Post by Parappaman » Mon Mar 01, 2010 2:27 pm

How could it be otherwise? This is a dream machine! :wink:

colm
Posts: 409
Joined: Tue Jan 31, 2006 8:22 am
Location: maine

Post by colm » Fri Mar 05, 2010 5:05 am

Wow. that is alot of stuff for video.
This interested me... but for 26 fps versus 5 or 6 on a 5 year old prescott....
there is a hardware decode in the avivo, limited, but proving we (10mbit+ video editors) are all getting railroaded.


I ain't budging. :roll:

I was going to humor an older pin grid array setup, workstation, simply two dual core cpus, and changed my mind.

This is some serious dedication you have. I assume more than hobby happening. I wouldn't recommend it after what I learned over the years, with video editing...unless business is writing off the equipment.

murtoz
Posts: 122
Joined: Tue Mar 13, 2007 12:24 pm
Location: Wiltshire, UK

Post by murtoz » Wed Mar 24, 2010 2:36 am

nice build!
Have a similar system albeit in a lian-li chassis, and am plannign to upgrade the stock intel heatsink to noctuas. I just wondered though - you mention you modified their retention bracket for your setup, could you give some more details on this? Why was it needed? Was it an easy mod? Do you have any pics of the process?

thanks!

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Wed Mar 24, 2010 2:56 am

I modified the brackets by adding a 5x10 washer on every screw to get more spring load, if mounted vertically these heatsinks tend to detach from the CPUs that's why I added the washers and the little wire you see in some images hanging from the PSU cage to CPU0 HSF, I'm planning to add a similar support system to CPU1 too. The other modification I had to do wasn't because of this, but because one of the coolers shipped with one of the four hexagonal screws that had a manufacturing defect, so I made another one using a M3x30 bolt and some nuts :-) i'll look for some more images I should have those :-)

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Wed Mar 24, 2010 2:57 am

This is the replacement bolt for the defective one
Image

And the washers used to get more spring load
Image

I explained this manufacturing issue to Noctua's technical support and they finally sent me a new set of brackets, but it was too late I had already mounted everything with my new homemade bolts :-)

murtoz
Posts: 122
Joined: Tue Mar 13, 2007 12:24 pm
Location: Wiltshire, UK

Post by murtoz » Wed Mar 24, 2010 6:29 pm

Great, thanks for the detailed explanation & pics!
Did you happen to check cpu temps with the intel stock heatsink? Would like to get an idea how much the noctua's gained you.

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Mar 25, 2010 3:26 am

Intel stock have very dense fins and a 60mm fan so to cool them down you need a fast and noisy fan. They're way smaller so less surface, from some brief tests I made before getting the Noctuas in open air condition I found out with the stock HSF it's hard to keep them below 80 °C without a very good airflow, while Noctuas barely let them raise above 60 :-)

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

:P

Post by pm.stacker » Thu Mar 25, 2010 10:58 am

These days I bought a lot of new GP HDDs :-P, two single-platter WD5000AADS to replace the two old WD5000AACS which still were in my 4x500G array, now they're 4 identical and the difference is clear... :mrgreen:

Image

:-)

Now before restoring the data inside, I wanna try some RAID5/RAID6 setups with this controller :wink:

piglover
Posts: 134
Joined: Mon Jun 08, 2009 9:32 am
Location: California

Post by piglover » Fri Mar 26, 2010 8:25 am

Have you had any trouble getting that system to sleep properly?

pm.stacker
Posts: 197
Joined: Wed May 14, 2008 2:54 am
Location: Northern Italy

;

Post by pm.stacker » Sat Mar 27, 2010 2:29 pm

I don't use sleep mode. If I don't use it, I power it off :-)

Post Reply