P180 Mini - GA-MA74GM-S2 - Hot-swap Encrypted RAID server

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Post Reply
clalor
Posts: 56
Joined: Mon Jul 04, 2005 7:00 pm
Location: SLC, Utah, US

P180 Mini - GA-MA74GM-S2 - Hot-swap Encrypted RAID server

Post by clalor » Sat Nov 15, 2008 6:43 pm

First off, thanks to everyone who posts their systems in the gallery. Second, sorry for the novel. I think I've spent too much time writing engineering test reports. :D Make sure to click on the pictures for larger versions.

I decided to build this machine for two reasons: First, so that I'd have a common location to store my files that could easily be accessed by my normal desktop and my newly purchased Mac mini; Second, to setup a system that would make it easy to spit data out to a drive that would be stored off-site. My data isn't super critical right now, but I do have a few Subversion repositories that host both customer code and my own programming projects, plus a growing collection of digital photographs. In just one weekend out at the race track, I generated almost 15 GB of photos. The motorsports photography thing is just a hobby for me right now. However, I do have the goal of obtaining photographer's accreditation in the future for the "minor" events out at the race track and I wanted the system to be able to handle whatever I throw at it for at least the next year or two.

To accomplish those goals, I decided to build the system around two drives in a RAID 1 array and housed in a SATA hot-swap backplane. The two drives in the backplane plus a spare third drive would be rotated through the array with the spare drive being stored off-site. Rather than use a dedicated RAID card or the motherboard's RAID, I decided to setup a Linux software RAID. Since I'm paranoid when it comes to security, I also decided that the data on the array would be encrypted using Truecrypt to prevent anyone from accessing my data should either the server or the spare drive be stolen.


Construction & Mods

Here's the equipment that I ended up purchasing:

Case: Antec P180 Mini
Motherboard: Gigabyte GA-MA74GM-S2
CPU: AMD Athlon X2 4450e Brisbane
RAM: Crucial Ballistix 2 x 1GB DDR2 800
PSU: Seasonic S12 II 330 W
CPU heatsink: Thermalright SI-128 SE
Heatsink fan: Panaflo L1A (not sure of the exact model number)
Exhaust fan: Scythe S-FLEX SFF21D
SATA backplane: ICY DOCK MB453SPF-B
System hard drive: Seagate Momentus 5400.3 ST9120822AS 120GB SATA
RAID hard drives: Western Digital Caviar GP WD5000AACS 500GB SATA
Optical drive: Samsung SH-S223F SATA
Operating system: Ubuntu 8.04 Server

After I had found the backplane I was going to use, I wanted to find a case with a separate chamber with two 5.25" drive bays with the chamber's exhaust being handled by the PSU. The P180 Mini was perfect for this application. Airflow through the backplane in its stock form was pretty restrictive (check out the pictures in the Newegg link above for the backplane). I just had to perform some mods to the backplane chassis and drive trays to ensure that there would be adequate airflow across the drives.

The backplane mods simply consisted of removing the chassis' top, bottom, and rear panels, plus the stock fan. These mods were as easy as removing a few screws to remove the panels. The stock fan was designed to be hot-swapped, which made that part even easier. Since all of these mods are easily reversible, they'll be easy to undo should I need to RMA the backplane. Even with those panels removed, the chassis is still plenty rigid when it's installed in the case. I haven't run into any issues with the drive connectors being misaligned with the backplane connectors.

Image

The mod to the drive trays was a little more involved, but still easily reversible. In stock form, they have a very thin piece of metal grill behind the tray's intake. With a drive installed, this grill directs airflow toward the front of the drive, but there's not much room between the drive and this grill. That grill is removed by unscrewing the tray's slides and carefully pulling the grill from the front plastic piece.

Image

The picture below shows the before/after difference.

Image

After the mods to the backplane and trays were complete, I sealed the case's lower chamber intake. I was very careful to route the PSU power cables and SATA cables in such a way as to not impede airflow between the end of the backplane and the PSU.

Image

Image

The SI-128 SE is definitely overkill for this since the CPU spends probably 95% of its time idling and the stock AMD heatsink/fan combo was actually pretty quiet. However, I figured I'd grab it while it was still available. Had I been able to find an XP-120, I would have bought that instead.

The 200 mm top fan is installed, but it's not being used. In fact, I have a book sitting on top of its exhaust grill to force air to be pulled through the front grills. Someday I'll seal it properly.

Image

Image

I've removed the grill from the lower intake to give that some bias with airflow and I'll close up the upper intake sometime in the future.

Image

And since I like seeing other people's desks, here's mine:

Image


RAID & Encryption Setup

The software RAID was setup first. Setup, plus removing and adding drives in a running array, is accomplished using the 'mdadm' command with the proper options.

For encryption, I decided to use Truecrypt because I was familiar with it. Since the server is running the command-line only version of Ubuntu, I had to compile a special command-line only version of Truecrypt. I settled on AES for the encryption algorithm and SHA-512 for the hash algorithm. Testing showed that AES was much less demanding of CPU time than using one of the cascades like AES-Twofish, which should translate into better read/write performance from the array. I don't have the numbers for comparison, but the difference in CPU utilization between the two was pretty dramatic. I didn't compare AES to Twofish or Serpent or any of the other hashing algorithms to determine which had the lowest CPU utilization.

Since I wanted the mounting/decryption of the array to be automatic on boot, the Truecrypt volume was created using key files and no password. The keys are stored on a USB flash drive that's plugged into the server. That USB drive is mounted automatically on boot and then dismounted after the Truecrypt volume has been successfully mounted. I wrote a custom script that runs as part of the rcX.d scripts to take care of all of that. The mdadm and Samba service start orders were modified so that the mdadm service comes up first allowing access to the array, the Truecrypt volume on the array is mounted, and then the Samba service starts last so that the data on the Truecrypt volume is available right away.

Having the keys attached to the server on a USB flash drive isn't the ideal solution if the server is stolen, but it should work well in at least protecting the off-site drive. A better solution might be to bring up the server's wireless card at boot, which is normally turned off except for OS updates, to load the key files from a remote server. Other than through the wireless card, the server has no Internet access (the wired LAN runs only between the computers at my desk).


RAID Read/Write Performance

The network connecting all of these computers together is gigabit ethernet with a gigabit switch (Netgear GS-108T) between them. This server is connected with a Cat 5e cable while the other two machines are connected with short Cat 5 cables. Those Cat 5 cables are only about 5 feet long, but what sort of effect they have on read/write performance, if any, hasn't been tested yet. I've been too lazy to make new Cat 5e cables, even though I already have everything I need to make them. :)

Sustained RAID read/write tests were performed using a 694 MB ISO image (the Ubuntu 8.04 Live CD).

Code: Select all

Sustained write: 24 MB/s
Sustained read:  27
The most demanding use I'll have for the server will be Adobe Lightroom (1.4) reading from the array. Lightroom already felt a little slow for me when reading from a local drive. However, the decrease in read speed hasn't made it feel that much slower. Writes will be fine since Lightroom is setup to write any edits to a small XML "sidecar" file for each photo instead of the original file itself. The biggest of those sidecar files that I've found is still under 20 KB.

I don't have any read/write data from an unencrypted array for comparison.


Power Consumption & Temperatures

Sustained RAID read/write tests were performed using the same ISO image as above.

Code: Select all

Power Consumption:
Idle:            49-53 W
Sustained write: 75-87 (CPU utilization approx. 115% for Truecrypt + Samba)
Sustained read:  58-65 (CPU utilization approx. 115% for Truecrypt + Samba)
RAID resync:     74-75 (takes approximately 2 hours)
Those CPU utilization numbers are out of 200% available (100% per core) as reported by the 'top' command.

As far as temperatures go, I'm mostly concerned with the temperatures of the drives in the array. That's good since the k8temp-pci module is not working well with my CPU (reports temperatures between -7 and 15 C). A nice benefit of using the Linux software RAID is that it doesn't block access to the SMART data of these drives. With the two drives installed in the top and bottom and no drive installed in the middle slot:

Code: Select all

Drive Temperatures:
Idle:        Low 30s C
RAID resync: Upper 30s to low 40s
The drive temperatures go up when a third drive is intalled in the middle slot, but by how much depends on the drive. With another WD5000AACS in the middle slot, the idle temps will raise to the mid 30s. With a Seagate 120 GB 7200.7 drive in that middle slot, the Seagate will idle close to 40 C and the other drives in the upper 30s. Ambient temperature in the room was probably between 20-25 C at the time of testing.

Had I not been set on encrypting the array, I could have cut power consumption quite a bit and probably improved on read/write performance. At one point, when running the CPU at 1.0 GHz and 1.000 V, I had idle power consumption down to 40-45 W and the RAID resync was only pulling 54-55 W (no change in length to resync, not CPU bound). However, the read/write performance dropped to less than half of what I'm getting now.

Quietness

Pretty much everything in this system has a pretty good track record when it comes to being quiet, but I'm surprised at how quiet the modified backplane is even though the drives are essentially hard mounted. The trays fit snuggly into their slots on the backplane so there's no vibrational noise that I can discern. There's a very slight sound from the WD drives, which I think is just the drive's airborne noise, but it's very faint and not at all harsh. Seeks on the WD drives are pretty much inaudible for me and I generally have to be looking at the case to see if there's drive activity going on. This combination of backplane and drives may not be quiet enough for the hardcore, but I'm quite happy with it.

felix_w
Posts: 38
Joined: Tue Nov 21, 2006 5:09 am
Location: Athens, Greece

Post by felix_w » Sun Nov 16, 2008 7:06 am

Very nice setup you have there clalor...

Jay_S
*Lifetime Patron*
Posts: 715
Joined: Fri Feb 10, 2006 2:50 pm
Location: Milwaukee, WI

Post by Jay_S » Wed Apr 22, 2009 10:45 am

Clalor, great build. I'm looking at using the same MB for my new server. Having used it for several months, how's everything running?

Question: your 24 & 27 MB/s (write/read) numbers look pretty low. Is this because of the encryption? I have the slightly newer WD10EADS and see 46 / 56 MB/s (local write/read) for file sizes up to 2GB in a much slower system (single-core A64, ubuntu 8.10 desktop).

Question 2: what file system are you using on your array? Having used Ext3 for many years on my music server, and experienced creeping data corruption, I'm looking at alternatives. I'm testing opensolaris and ZFS, and it's a PITA on my current test PC. The GA-MA74GM-S2 is one of the cheapest boards with 6 sata ports on the Solaris hardware compatibility list, which is why I started researching it.

Thanks,
Jay

[5/1/09 - edited to clarify my read/write speeds were measured locally]
Last edited by Jay_S on Fri May 01, 2009 2:00 pm, edited 2 times in total.

bajker
Posts: 6
Joined: Tue Oct 11, 2005 12:32 pm
Location: Slovenia

Post by bajker » Thu Apr 23, 2009 2:04 am

Jay_S wrote:Question: your 24 & 27 MB/s (write/read) numbers look pretty low. Is this because of the encryption? I have the slightly newer WD10EADS and see 46 / 56 MB/s (write/read) for file sizes up to 2GB in a much slower system (single-core A64, ubuntu 8.10 desktop).
I think that 24 & 27 MB/s is quite decent result considering SMB & encrtyption and software RAID. You won't get much over 40MB/s over 1GBit network connection if you use SMB or NFS. Only if you use block level access over network, like iSCSI, you get higher speeds. I have RAID5 server with HW RAID without encryption and locally I can get 70&100 MB/s, but over network just 30-40 MB/s (both R&W). With iSCSI I get around 70MB/s (R&W).
So, clalor's numbers are OK.

Jay_S
*Lifetime Patron*
Posts: 715
Joined: Fri Feb 10, 2006 2:50 pm
Location: Milwaukee, WI

Post by Jay_S » Fri May 01, 2009 1:54 pm

I was waiting to reply until I had some new hardware & more data. With my new hardware (GA-MA74GM-S2 with AMD 4050e, on-board Realtek 8111C gigabit NIC, WD10EADS green drive) and Ubuntu 9.04 server, I'm getting 45-48 MB/s via Samba shares. I see periodic 52MB/s peaks in htop. This is on a fresh 9.04 server install, with no samba tweaks. I'm pretty sure I can improve this. I haven't tested FTP transfers yet, but expect them to be slightly faster than un-tweaked samba.

I think hardware performance has reached a point where it's raising the accepted network xfer standards. Everyone has been telling me that 30's MB/s is acceptible for samba transfers, but I'm seeing better than that (now - my older system was in the 30's).

Still, given the encryption and sw-raid, I can understand clalor's mid-20's

(Aside - so far I'm loving ubuntu 9.04 server.)

coolTechno
Posts: 4
Joined: Thu Jun 04, 2009 5:03 pm
Location: Colombo

Post by coolTechno » Thu Jun 04, 2009 5:14 pm

spec is good... how much did u pay

enginrical
Posts: 1
Joined: Sat Jun 06, 2009 3:04 pm
Location: Surrey

200mm fan

Post by enginrical » Sat Jun 06, 2009 3:17 pm

Nice build dude, Have you considered turning the 200mm fan upside down and running it on low for more forced vent to give you a little +ve pressure in the case.

Just got a P180mini myself so I'm checking out other builds. Yours is the first P180m on the forums. looks good, v.nice cable management.

clalor
Posts: 56
Joined: Mon Jul 04, 2005 7:00 pm
Location: SLC, Utah, US

Post by clalor » Fri Sep 11, 2009 8:36 pm

I took some time off from SPCR and missed all the replies, sorry about that. It may be a little late for my reply, but I'll go ahead anyway.

From what I can tell, the low read/write speeds are due to encryption and appear to be CPU bound. I haven't had the time or inclination to try overclocking to see if there's any speed increase with a faster CPU speed, but I'd guess that it would help. A test with a similar model hard drive, unencrypted, would also be worth performing if I had a spare. Lately, I've been spending more time working with photos in Lightroom and I've noticed that it's a bit slower than when I was working with files on a local disk (obviously). If Lightroom was a daily thing, then I'd start to consider what I could to to increase the read/write speed, but I think the benefits of this setup out weigh the negatives.

I am using EXT3 as the file system inside the Truecrypt volumes. So far, so good, at least as far as I can tell.

Back in October 2008 when I bought everything, I think I spent a total of about $650 including the four 500 GB Western Digital Green drives, but excluding the RAM (2 GB, came from the desktop when it was upgraded).

The idea about reversing the fan might be worth looking into. I've had a book sitting on top of the case acting as a cover for that vent. I'll have to see if there's an updated k8temp-pci module available that fixes the CPU temperature readings and run some tests in the future.

In the time since I put this thing together, I've been extremely happy with it. With that said, it's going to see some changes in the coming months after I move. The current plans are to pull the wireless card, install two TV tuners, and turn it into a MythTV backend (replacing my current MythTV machine, frontend duties will be taken care of by a Zotac ION that I'm assembling now). The new place also has a lower noise floor, so it will be interesting to see how quiet it really is.

Thanks for the compliments. :)

Post Reply