First off, thanks to everyone who posts their systems in the gallery. Second, sorry for the novel. I think I've spent too much time writing engineering test reports.
Make sure to click on the pictures for larger versions.
I decided to build this machine for two reasons: First, so that I'd have a common location to store my files that could easily be accessed by my normal desktop and my newly purchased Mac mini; Second, to setup a system that would make it easy to spit data out to a drive that would be stored off-site. My data isn't super critical right now, but I do have a few Subversion repositories that host both customer code and my own programming projects, plus a growing collection of digital photographs. In just one weekend out at the race track, I generated almost 15 GB of photos. The motorsports photography thing is just a hobby for me right now. However, I do have the goal of obtaining photographer's accreditation in the future for the "minor" events out at the race track and I wanted the system to be able to handle whatever I throw at it for at least the next year or two.
To accomplish those goals, I decided to build the system around two drives in a RAID 1 array and housed in a SATA hot-swap backplane. The two drives in the backplane plus a spare third drive would be rotated through the array with the spare drive being stored off-site. Rather than use a dedicated RAID card or the motherboard's RAID, I decided to setup a Linux software RAID. Since I'm paranoid when it comes to security, I also decided that the data on the array would be encrypted using Truecrypt to prevent anyone from accessing my data should either the server or the spare drive be stolen.
Construction & Mods
Here's the equipment that I ended up purchasing:
Case: Antec P180 Mini
Motherboard: Gigabyte GA-MA74GM-S2
CPU: AMD Athlon X2 4450e Brisbane
RAM: Crucial Ballistix 2 x 1GB DDR2 800
PSU: Seasonic S12 II 330 W
CPU heatsink: Thermalright SI-128 SE
Heatsink fan: Panaflo L1A (not sure of the exact model number)
Exhaust fan: Scythe S-FLEX SFF21D
SATA backplane: ICY DOCK MB453SPF-B
System hard drive: Seagate Momentus 5400.3 ST9120822AS 120GB SATA
RAID hard drives: Western Digital Caviar GP WD5000AACS 500GB SATA
Optical drive: Samsung SH-S223F SATA
Operating system: Ubuntu 8.04 Server
After I had found the backplane I was going to use, I wanted to find a case with a separate chamber with two 5.25" drive bays with the chamber's exhaust being handled by the PSU. The P180 Mini was perfect for this application. Airflow through the backplane in its stock form was pretty restrictive (check out the pictures in the Newegg link above for the backplane). I just had to perform some mods to the backplane chassis and drive trays to ensure that there would be adequate airflow across the drives.
The backplane mods simply consisted of removing the chassis' top, bottom, and rear panels, plus the stock fan. These mods were as easy as removing a few screws to remove the panels. The stock fan was designed to be hot-swapped, which made that part even easier. Since all of these mods are easily reversible, they'll be easy to undo should I need to RMA the backplane. Even with those panels removed, the chassis is still plenty rigid when it's installed in the case. I haven't run into any issues with the drive connectors being misaligned with the backplane connectors.
The mod to the drive trays was a little more involved, but still easily reversible. In stock form, they have a very thin piece of metal grill behind the tray's intake. With a drive installed, this grill directs airflow toward the front of the drive, but there's not much room between the drive and this grill. That grill is removed by unscrewing the tray's slides and carefully pulling the grill from the front plastic piece.
The picture below shows the before/after difference.
After the mods to the backplane and trays were complete, I sealed the case's lower chamber intake. I was very careful to route the PSU power cables and SATA cables in such a way as to not impede airflow between the end of the backplane and the PSU.
The SI-128 SE is definitely overkill for this since the CPU spends probably 95% of its time idling and the stock AMD heatsink/fan combo was actually pretty quiet. However, I figured I'd grab it while it was still available. Had I been able to find an XP-120, I would have bought that instead.
The 200 mm top fan is installed, but it's not being used. In fact, I have a book sitting on top of its exhaust grill to force air to be pulled through the front grills. Someday I'll seal it properly.
I've removed the grill from the lower intake to give that some bias with airflow and I'll close up the upper intake sometime in the future.
And since I like seeing other people's desks, here's mine:
RAID & Encryption Setup
The software RAID was setup first. Setup, plus removing and adding drives in a running array, is accomplished using the 'mdadm' command with the proper options.
For encryption, I decided to use Truecrypt because I was familiar with it. Since the server is running the command-line only version of Ubuntu, I had to compile a special command-line only version of Truecrypt. I settled on AES for the encryption algorithm and SHA-512 for the hash algorithm. Testing showed that AES was much less demanding of CPU time than using one of the cascades like AES-Twofish, which should translate into better read/write performance from the array. I don't have the numbers for comparison, but the difference in CPU utilization between the two was pretty dramatic. I didn't compare AES to Twofish or Serpent or any of the other hashing algorithms to determine which had the lowest CPU utilization.
Since I wanted the mounting/decryption of the array to be automatic on boot, the Truecrypt volume was created using key files and no password. The keys are stored on a USB flash drive that's plugged into the server. That USB drive is mounted automatically on boot and then dismounted after the Truecrypt volume has been successfully mounted. I wrote a custom script that runs as part of the rcX.d scripts to take care of all of that. The mdadm and Samba service start orders were modified so that the mdadm service comes up first allowing access to the array, the Truecrypt volume on the array is mounted, and then the Samba service starts last so that the data on the Truecrypt volume is available right away.
Having the keys attached to the server on a USB flash drive isn't the ideal solution if the server is stolen, but it should work well in at least protecting the off-site drive. A better solution might be to bring up the server's wireless card at boot, which is normally turned off except for OS updates, to load the key files from a remote server. Other than through the wireless card, the server has no Internet access (the wired LAN runs only between the computers at my desk).
RAID Read/Write Performance
The network connecting all of these computers together is gigabit ethernet with a gigabit switch (Netgear GS-108T) between them. This server is connected with a Cat 5e cable while the other two machines are connected with short Cat 5 cables. Those Cat 5 cables are only about 5 feet long, but what sort of effect they have on read/write performance, if any, hasn't been tested yet. I've been too lazy to make new Cat 5e cables, even though I already have everything I need to make them.
Sustained RAID read/write tests were performed using a 694 MB ISO image (the Ubuntu 8.04 Live CD).
Sustained write: 24 MB/s
Sustained read: 27
The most demanding use I'll have for the server will be Adobe Lightroom (1.4) reading from the array. Lightroom already felt a little slow for me when reading from a local drive. However, the decrease in read speed hasn't made it feel that much slower. Writes will be fine since Lightroom is setup to write any edits to a small XML "sidecar" file for each photo instead of the original file itself. The biggest of those sidecar files that I've found is still under 20 KB.
I don't have any read/write data from an unencrypted array for comparison.Power Consumption & Temperatures
Sustained RAID read/write tests were performed using the same ISO image as above.
Idle: 49-53 W
Sustained write: 75-87 (CPU utilization approx. 115% for Truecrypt + Samba)
Sustained read: 58-65 (CPU utilization approx. 115% for Truecrypt + Samba)
RAID resync: 74-75 (takes approximately 2 hours)
Those CPU utilization numbers are out of 200% available (100% per core) as reported by the 'top' command.
As far as temperatures go, I'm mostly concerned with the temperatures of the drives in the array. That's good since the k8temp-pci module is not working well with my CPU (reports temperatures between -7 and 15 C). A nice benefit of using the Linux software RAID is that it doesn't block access to the SMART data of these drives. With the two drives installed in the top and bottom and no drive installed in the middle slot:
Idle: Low 30s C
RAID resync: Upper 30s to low 40s
The drive temperatures go up when a third drive is intalled in the middle slot, but by how much depends on the drive. With another WD5000AACS in the middle slot, the idle temps will raise to the mid 30s. With a Seagate 120 GB 7200.7 drive in that middle slot, the Seagate will idle close to 40 C and the other drives in the upper 30s. Ambient temperature in the room was probably between 20-25 C at the time of testing.
Had I not been set on encrypting the array, I could have cut power consumption quite a bit and probably improved on read/write performance. At one point, when running the CPU at 1.0 GHz and 1.000 V, I had idle power consumption down to 40-45 W and the RAID resync was only pulling 54-55 W (no change in length to resync, not CPU bound). However, the read/write performance dropped to less than half of what I'm getting now.
Pretty much everything in this system has a pretty good track record when it comes to being quiet, but I'm surprised at how quiet the modified backplane is even though the drives are essentially hard mounted. The trays fit snuggly into their slots on the backplane so there's no vibrational noise that I can discern. There's a very slight sound from the WD drives, which I think is just the drive's airborne noise, but it's very faint and not at all harsh. Seeks on the WD drives are pretty much inaudible for me and I generally have to be looking at the case to see if there's drive activity going on. This combination of backplane and drives may not be quiet enough for the hardcore, but I'm quite happy with it.