my not-so-loud file server

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Post Reply
trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

my not-so-loud file server

Post by trxman » Sat Sep 29, 2007 3:36 am

split from viewtopic.php?t=40962

speaking about file servers, here's mine (CM GeminiII came later with one 1200rpm Noctua on it):

Image


it's C2D E6550 with 8GB of RAM. 8 x WD7500AAKS, some partitions in linux kernel raid. here's the example of bonnie++ on raid5 (have no raid0 to show off):


Code: Select all

Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
titan-raid5     16G  1001  94 176941  27 100796  16  1862  98 330395  27 312.1   5
Latency             10488us     474ms     205ms   26093us   74405us     173ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
titan-raid5         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 64   981   3 +++++ +++   967   1   943   3 +++++ +++   875   3
Latency               495ms      82us     246ms     437ms      61us     492ms
xfs used for massive caching. small files performance would be much better with reiserfs...

trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

Post by trxman » Sat Sep 29, 2007 3:42 am

@matt_garman

Code: Select all

194 Temperature_Celsius     0x0022   114   085   000    Old_age   Always       -       38
194 Temperature_Celsius     0x0022   112   071   000    Old_age   Always       -       40
194 Temperature_Celsius     0x0022   114   086   000    Old_age   Always       -       38
194 Temperature_Celsius     0x0022   113   085   000    Old_age   Always       -       39
194 Temperature_Celsius     0x0022   111   071   000    Old_age   Always       -       41
194 Temperature_Celsius     0x0022   113   086   000    Old_age   Always       -       39
194 Temperature_Celsius     0x0022   116   102   000    Old_age   Always       -       36
194 Temperature_Celsius     0x0022   115   101   000    Old_age   Always       -       37
with room temp ~25 deg C.


on the side panel in front of lower 6 hdds, I have two 92mm fans at ~1500rpm.
in the front of hdd cage in 5.25" space, I have one Noctua 120mm fan ~600rpm.

GeminiII has one Nocuta 120mm that kicks in only when CPU goes above 40 deg C. that is maybe once or twice a day. I'll post photos with GeminiII as soon as I open that box again.

matt_garman
*Lifetime Patron*
Posts: 541
Joined: Sun Jan 04, 2004 11:35 am
Location: Chicago, Ill., USA
Contact:

Post by matt_garman » Sat Sep 29, 2007 7:46 am

Awesome, thanks!

protellect
Posts: 312
Joined: Tue Jul 24, 2007 3:57 pm
Location: Minnesota

Post by protellect » Sun Sep 30, 2007 10:40 am

Thats nuts, haha.

The drive temperatures are surprisingly low. Two thumbs up.

trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

Post by trxman » Sun Sep 30, 2007 11:57 am

protellect wrote:Thats nuts, haha.

The drive temperatures are surprisingly low. Two thumbs up.
thx.

anyway, I think we should all congratulate WD for making such a Cool'N'Quiet drive. ;)
but, it's very fast, too. :D

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun Sep 30, 2007 3:22 pm

Not really that low temps, my setup runs between 28 and 35 degrees with nexus/noctua fans at below 800rpm :)
Last edited by Wibla on Sun Sep 30, 2007 4:42 pm, edited 1 time in total.

protellect
Posts: 312
Joined: Tue Jul 24, 2007 3:57 pm
Location: Minnesota

Post by protellect » Sun Sep 30, 2007 3:33 pm

I'm used to seagate temperatures :) 35-45C, with fans on it.

trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

Post by trxman » Sun Sep 30, 2007 3:50 pm

Wibla wrote:Not really that low temps, my setup idles between 28 and 35 degrees with nexus/noctua fans at below 800rpm :)
there's no idle state for my WDs:

Code: Select all

 01:49:57 up 16 days,  6:35,  3 users,  load average: 0.10, 0.26, 0.36

eth0      Link encap:Ethernet  HWaddr 
          inet addr:
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:980847708 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1473649187 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:555029955747 (529317.8 Mb)  TX bytes:1725676289112 (1645733.1 Mb)
          Interrupt:16 Base address:0x6000

eth1      Link encap:Ethernet  HWaddr
          inet addr:
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:990827237 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1345575160 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:503080798431 (479775.2 Mb)  TX bytes:1755732249962 (1674396.7 Mb)
          Interrupt:20 Base address:0x8000

;)

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun Sep 30, 2007 4:37 pm

Image

There is a raid verify in there somewhere, try to spot it (not during the last 6 hours) :)

This is with ambient temp around 23C...

Samsung drives dont run as hot tho, and with hardware raid there is always some activity on the arrays, even tho its not streaming hdtv or doing verifies or whatnot.

Image
o.O traffic! :shock:

trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

Post by trxman » Sun Mar 21, 2010 7:10 am

I've done a small upgrade:

- mobo is Asus P5Q Premium
- 2 x Digitus PCI-E x4 SATA controllers
- Chieftec BA
- 16 x Samsung F2 1.5TB

Image

Image

Image

Image

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sun Mar 21, 2010 9:47 am

I really like your solution for the sata cables :)

What kind of fans do you have infront of the bottom drives? and are those P12 fans on the 4in3's?

I've heard mixed reviews on how the F2 1.5TB's work in raid, ranging from "dont use them, they cant handle it" from backblaze, to others having no problems at all, whats been your experience so far?
Edit: read this: http://forums.storagereview.com/index.p ... tler-cctl/

Also, I hope youre gonna use RAID6 ? :)

trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

Post by trxman » Sun Mar 21, 2010 11:06 am

I'm using couple of arrays made by linux software raid1 and raid5. raid arrays are non-uniform and they cover about 60% of disk space. everything else is not protected by redundancy.

Code: Select all

cat /proc/mdstat 
Personalities : [linear] [raid1] [raid6] [raid5] [raid4] 
md4 : active raid5 sdf1[0] sdo2[9] sdn2[8] sdm2[7] sdl2[6] sdk2[5] sdj2[4] sdi1[3] sdh1[2] sdg1[1]
      443658240 blocks super 1.1 level 5, 64k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md2 : active raid5 sda3[0] sdi3[8] sdh3[7] sdg3[6] sdf3[5] sde2[4] sdd2[3] sdc2[2] sdb3[1]
      11301533184 blocks super 1.1 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
      bitmap: 0/11 pages [0KB], 65536KB chunk

md3 : active raid5 sdj1[0] sdp1[6] sdo1[5] sdn1[4] sdm1[3] sdl1[2] sdk1[1]
      1988476416 blocks super 1.1 level 5, 64k chunk, algorithm 2 [7/7] [UUUUUUU]
      bitmap: 0/3 pages [0KB], 65536KB chunk

md1 : active raid1 sdb2[1] sda2[0]
      52339648 blocks [2/2] [UU]
      bitmap: 5/200 pages [20KB], 128KB chunk

md0 : active raid1 sdb1[1] sda1[0]
      96256 blocks [2/2] [UU]
      bitmap: 0/12 pages [0KB], 4KB chunk

unused devices: <none>
I'm doing complete check on all arrays every month

Code: Select all

#!/bin/bash

echo check >> /sys/block/md0/md/sync_action
echo check >> /sys/block/md1/md/sync_action
echo check >> /sys/block/md2/md/sync_action
echo check >> /sys/block/md3/md/sync_action
echo check >> /sys/block/md4/md/sync_action
If any unreadable sector is found, raid will try to resync it resulting with sector reallocation. so far so good.

fans used inside the case are not ideal - they are S12 and they lack pressure needed at least on CPU heatsink. I already had them. but, considering low CPU load and heat dissipation, they are doing good job. core temps top at ~58C when both cores are loaded with prime95, and core temps cannot reach 50C during normal operation (pbzip, direct connect hashing, compiling...).

fans on cages are good - they are P12. :)

fans in front of lower drives are NF-B9, three of them.

about Samsung F2...
well, it's not blazing fast. nor it's fast at all.
but, it's quiet, cool and cheap, so it feets my needs: massive storage without too much need for random access and some fancy seeks.
sequential read is good - it reaches 1.2TB/s during raid check. it's not too fast considering every of 16 drives gives 75MB/s, but it's not bad eighter.
to achive that I had to put additional controllers in PCI-E slots connected to north bridge.
my monthly average disk read including complete check of all arrays is 9.5MB/s (sum for all disks).
average write is ~700KB/s.

average temperatures for Samsung are 28 to 30C.

btw, Samsung F2 is known to have terrible multitasking algorytm. I've tried to lower impact of that by making sure two most busy arrays don't share single phisical disk (md2 and md3) and by increasing use of write buffers (commit=29, commit=30, commit=31... as mount options and vm.dirty_* tunning).

Post Reply