Was 3.7TB - then 2.7TB, now 19.1TB "quiet" server

Wibla · Post by **Wibla** » Tue Nov 03, 2009 7:51 am

8mbit

Monkeh16 · Post by **Monkeh16** » Tue Nov 03, 2009 7:57 am

Is that all? Yeesh.

Wibla · Post by **Wibla** » Tue Nov 03, 2009 11:13 am

Yeah I know.. I'm not fortunate enough to live in an area with fibre internet, so im stuck at 8mbit adsl2+ for now... but that doesnt matter really, its fast enough for my needs..

Chocolinx · Post by **Chocolinx** » Fri Nov 06, 2009 8:14 am

Dude your build inspires me! I need this so bad. I currently have 4 to 5TB worth of HDD space lying on my desk, I really need a machine like this to start stacking more HDDs. It'd be great to help others in my house stream all the media I download, which is roughly 600GB a month haha 8Mbit isn't so bad. I have 7Mbit

Wibla · Post by **Wibla** » Fri Nov 06, 2009 6:11 pm

Performance progression with tweaking etc:

Code: Select all

2.6.30-amd64 kernel on 32bit etch userland - first testing, bad results:
 
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Oberon           4G           73935  16 67499  20           498156  51 145.2   0
Oberon,4G,,,73935,16,67499,20,,,498156,51,145.2,0,,,,,,,,,,,,,
 
2.6.26.2-amd64 debian - 64K Chunk size (8192 stripe cache, 16384 readahead), 3ware write cache enabled:
 
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
oberon.lan.wibla 4G           118752  34 110140  38           491916  59 284.5   1
oberon.lan.wibla.net,4G,,,118752,34,110140,38,,,491916,59,284.5,1,,,,,,,,,,,,,
 
2.6.26-2amd64 debian - 256K chunk size (16384 readahead, 8129 stripe cache), 3ware write cache enabled:
 
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
oberon           4G 46975  85 203242  32 130643  25 49964  85 451722  51 454.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2193  16 +++++ +++  1721  13  1654  15 +++++ +++  1421  12
oberon,4G,46975,85,203242,32,130643,25,49964,85,451722,51,454.4,1,16,2193,16,+++++,+++,1721,13,1654,15,+++++,+++,1421,12

I discovered that I am cpu bound while doing FTP transfers, wasnt expecting that, maxing out at around 800mbit/s thus far. Might have to pick up a P4 3.4GHz from ebay (dual core preferably)...

After mirroring over the contents of my 9.1TB box:

Code: Select all

/dev/md0               20T  8.9T   11T  47% /mnt/raid

Chocolinx: Get a big case with lots of 5.25" bays, then toss in 4in3 modules with nexus fans, and youre on the right track.. A server like this will never be silent, maybe not even very quiet, but its feasible to make it quiet enough that you dont mind the machine being on. This server is currently in the home cinema, where it blends in quite well with the (low) fan noise from my Epson TW700...

I dont download nearly enough to justify this server in the short run, average download speeds have been less than 1mbit/s the last few months, tho much of that is because the old server has been nearly full. But it was never planned as a short-term solution either, I dont foresee upgrading untill late 2010, maybe early 2011, where monster (9.1TB in sig) will get a similar treatment

mark19891989 · Post by **mark19891989** » Sat Nov 07, 2009 1:21 pm

How do you make the graphs?

thanks markp1989

FartingBob · Post by **FartingBob** » Sat Nov 07, 2009 3:01 pm

Jaws wrote:You're gonna need a bigger boat.

Surely the next step up for you is a proper server rack?

Wibla · Post by **Wibla** » Sat Nov 07, 2009 3:02 pm

It's called munin, I use it to monitor all my machines

Very easy to setup, and easy to make plugins for it aswell. I use it on linux, dunno about windows support for the graphing engine itself.

Monkeh16 · Post by **Monkeh16** » Sat Nov 07, 2009 5:10 pm

FartingBob wrote:
Jaws wrote:You're gonna need a bigger boat.
Surely the next step up for you is a proper server rack?

They're remarkably expensive, and there are very few cases able to hold large numbers of HDDs short of full on SAN units, and they're noisy.

Wibla · Post by **Wibla** » Sun Nov 08, 2009 5:28 am

The Norco 4020 is a pretty decent 4U with 20 hotswap SATA/SAS bays, I was considering one, but decided against it seeing as I already had a stacker I could upgrade, and I didnt want to modify the norco to keep it at a reasonable noise level...

Chocolinx · Post by **Chocolinx** » Mon Nov 09, 2009 12:20 pm

Think a AMD Athlon LE-1660 will be fast enough to be in a server? My basic setup was going to be about half the size of yours... Thinking of only needing to hold 8HDDs 1.5TB each, so only 12GB. I was thinking of picking up an Antec 300, and changing the 3 5.25 with suspension for more hdds, it already has 6 spots for hdds.

Is 1GB of ram enough? or 2GB minimum? The Motherboard was going to be a 785GM board from Gigabyte since it has 6 SATA slots, and I was going to throw in a random SATA PCI-e card that adds another 4. Anything I'm missing? Or should look into? I plan on using it to stream 720p and 1080p videos through out the house, over ethernet 100mbit.

Wibla · Post by **Wibla** » Mon Nov 09, 2009 12:31 pm

Yes, that should do fine for a 100mbit setup. I am cpu limited for ftp transfers with my 3.2GHz Celeron, tops out at about 800mbit/s.

I'd get 2-4GB ram if possible, that way you dont really need to think about it. I've got 2GB in that server now, and its definently not enough.

jan_chu · Post by **jan_chu** » Wed Nov 11, 2009 3:19 am

insted of using allot of money on CPU, you could also look into a network card with build in TCP Offloading. That should take allot of the stress from your cpu.

Wibla · Post by **Wibla** » Thu Nov 12, 2009 11:00 am

I have an Intel pci-e card currently, not sure how much offloading it supports, but thanks for the idea, much appreciate it

Monkeh16 · Post by **Monkeh16** » Thu Nov 12, 2009 12:52 pm

Wibla wrote:I have an Intel pci-e card currently, not sure how much offloading it supports, but thanks for the idea, much appreciate it

Uhm, everything?

Read the ethtool man page.

Wibla · Post by **Wibla** » Thu Nov 12, 2009 1:34 pm

So i realized.. and most of it is enabled already...

Code: Select all

# ethtool -k eth3
Offload parameters for eth3:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

jan_chu · Post by **jan_chu** » Fri Nov 13, 2009 12:42 am

then your cpu usage should not be because of network traffic.

It must then be Software RAID I/O dependent, and maybe if you use FTP transfers with encryption ???

FartingBob · Post by **FartingBob** » Sun Nov 15, 2009 3:45 am

Chocolinx wrote:Think a AMD Athlon LE-1660 will be fast enough to be in a server? My basic setup was going to be about half the size of yours... Thinking of only needing to hold 8HDDs 1.5TB each, so only 12GB. I was thinking of picking up an Antec 300, and changing the 3 5.25 with suspension for more hdds, it already has 6 spots for hdds.

Is 1GB of ram enough? or 2GB minimum? The Motherboard was going to be a 785GM board from Gigabyte since it has 6 SATA slots, and I was going to throw in a random SATA PCI-e card that adds another 4. Anything I'm missing? Or should look into? I plan on using it to stream 720p and 1080p videos through out the house, over ethernet 100mbit.

I have a Le-1660 AMD in my fileserver (6TB on 4 drives, soon upgrading to 6 drives) and it does its job with 2GB of RAM.
Mind you, i dont use RAID (should have when setting it up, but didnt have enough storage elsewhere to format the drives with all my data on, so kept things how they were. set up automatic backups every other day, it does its job).

Chocolinx · Post by **Chocolinx** » Mon Nov 16, 2009 1:20 pm

Building my mini-server (compared to Wibla's) this week. I ended up going with the following parts.

Fortron 400W
AMD Sempron 140 (Unlocks into a Athlon II X2)
Antec 300 (6 HDD Slots, More with Modding)
Giga 785GM Board
1GB DDR2 Ram (NCIX didn't price match my $70 4GB

So using RAM that I found laying around the house)

HDD that I'm starting off with:
400GB Seagate as Boot Drive (Because it's not being used)
1x 1.5TB WD Green
And maybe these ones that are sitting in another box:
2x 1.5TB Seagate 7200.11
2x 500GB WD

Since I'm only storing files, I probably won't RAID anything since there's no real advantage over a 100mbit connection. Simply using this machine to share files between all the computers in the house (Total of 4). Probably just enable file sharing in WinXP and be done with it, or TVersity.

Just wanted to say thanks Wilba for your thread, inspiration is appreciated

I'll take pictures when I finish making it. Somehow I'll try and silence the drives haha

Monkeh16 · Post by **Monkeh16** » Mon Nov 16, 2009 6:28 pm

There's no advantage to RAID over a gigabit connection either.. It's not about speed.

Chocolinx · Post by **Chocolinx** » Fri Nov 27, 2009 7:35 am

Got all my parts and the system is finally built. Antec 300's fans are damn loud... and the stock cooler for the Sempron is hell loud, swapping those out when freezer64 and yateloon fans come in... Although maybe it was all in vain since it's a file server and the harddrives alone will produce so much noise I'll have to turn it off at night.

Either way, everything went smoooothly

Starting off with 5 HardDrives in the case. Notebook harddrive as the main OS drive. 3x 1.5TB and one 400GB. Then I have a box that can hold 4 more HDDs. I don't have an SATA card yet but I'll buy one off DealExtreme

The Sempron Unlocked fine and the cores are very stable. So now I have a $40 Dual Core CPU that's faster then all the other systems in my house except for my Q9450.

Wibla · Post by **Wibla** » Sat Nov 28, 2009 4:09 pm

I always swap out the stock fans (and preferably heatsinks) for something better, tends to help ALOT

AznJason · Post by **AznJason** » Sun Nov 29, 2009 11:21 pm

I don't want to hijack your thread, but if you create a partition > 2 TB (using a 64-bit OS or something that can handle it), does Windows XP 32-bit on a separate machine have issues seeing the full size? I know for 32-bit Windows, there's a 2TB limit for drives on the machine itself, but I think this might not matter for a network drive since the 32-bit OS isn't managing the partition.

Wibla · Post by **Wibla** » Mon Nov 30, 2009 3:28 am

Thats not an issue

AznJason · Post by **AznJason** » Tue Dec 01, 2009 6:11 pm

Thanks! From my understanding, the 32-bit limitation is on addressing all of the available space, thus it should only affect the machine that the hard drives are physically attached to. Seems like it's right.

Wibla · Post by **Wibla** » Sat Dec 05, 2009 12:41 pm

Seems like it

Tagan PSU that was in this box for 2 years, damn nasty...

Rushdie · Post by **Rushdie** » Wed Dec 09, 2009 7:40 pm

Wibla wrote: http://bilder.wibla.net/albums/oberon-r ... .sized.jpg
6 drives off one molex, probably not the smartest thing in the world...

Depends on if you're using staggered spin up or not. A 4-pin molex can handle about 10-12A (during normal temperatures) with standard AWG 18 wires. The drives I'm using requires about 32W at spin up, you do the math. (Or I can do them for you: 32W * 6 drives = 16A @ 12V.) These SATA-power splitters are much worse, they are rated at 1.5A per pin (and since there is 3pins for each voltage, you get 4.5A... just two drives spinning up at the same time could require more). Now personally I don't bother to much with it, but one might want to be aware of the fact

Wibla wrote:Seems like it

http://bilder.wibla.net/albums/pcstuff/ ... .sized.jpg
Tagan PSU that was in this box for 2 years, damn nasty...

Wow. 'You a smoker? Because that looks really filthy. My dust bunnies are light grey, that stuff looks like it could be dangerous to the touch

Edit: Now I know what it looks like! Semi-decomposed fur from roadkill left in the snow for a couple of months. Ehhh... Hm.

Wibla · Post by **Wibla** » Thu Dec 10, 2009 2:40 am

That server was in a room where people smoked, also got cats, so its kinda doomed...

Post by **CA_Steve** » Fri Jan 15, 2010 7:55 am

Good thing the cats don't smoke, or the server would be truly doomed.

Wibla · Post by **Wibla** » Fri Jan 15, 2010 6:31 pm

Indeed, hehe.

1900 hours runtime on the drives, one drive has 52 reallocated sectors, the rest are fine. The drive with reallocated sectors still passes self tests etc, so its not a catastrophic failure going on yet, but will keep monitoring the drive