I see PCI and PCIe cards for SATA expansion. Am I missing an alternative?

And which cards / chips do people recommend for easy linux (kernel 2.6.32or later) installation? Feel free to offer solutions if you don't know about linux drivers, too.

*Transfer speed is not hugely important (100MB/s 1-channel throughput would be nice)
*Assume no preference in interface type
*Internal expansion

Also - I'm assuming noise is not an issue here? Do these ever keep the drives spinning in spite hdparm spindown_time?

Edit: Ideally I am looking for ~4 SATA ports, but multiple cards are an option. And I will be using mdadm software RAID, if any.

If you want 100MB/s per port than PCI cards are off your list. You might get that on one port, but never on 2 or more at the same time.

A couple of cheap 2 port Silicon Image PCIe cards would do the trick just fine. They have "RAID" but linux will just ignore it and treat them as SATA controllers. Sleep works fine on them in Windows, don't see why it wouldn't in linux. There are 4 port cards out there too, but beware that some are built with PCI to PCIe bridge chips and not really PCIe cards.

Why not put your RAID drives on your MB ports? You'll get better performance that way. If you need the extra ports use the add on cards for boot drives and/or non-raid storage.

This would be primarily for expansion storage (speed-critical data will be on motherboard).

I meant 100MB/s while writing to a single HDD (I am assuming that bandwidth is fungible, or will effectively be via software RAID striping of partitions)

I see a few PCI cards which claim "150MB/s", which I assume is total bandwidth, but I don't know if this is accurate.

Per your post,
Rosewill RC-211 Silicon Image 2 port SATA II PCI Express Host Controller Card
http://www.newegg.com/Product/Product.a ... 6816132008
certainly looks attractive, if drivers are a sure thing!

Edit: I'll probably also be moving all DVD drives to these to free up motherboard connections, unless someone says this is unwise

edit2: good call on chipset.
"Silicon Image 3124/3124-2 (chip in 4-port SATA-II PCI-X cards) and 3132 (chip in 2-port SATA-II PCI Express cards) (Silicon Image, Inc., formerly CMD Technology, Inc.) — libata's sata_sil24 driver (production quality) supports Silicon Image 3124 (2005-08) and also the follow-on 2-port PCI Express SATA-II successor chip, the Silicon Image 3132. (Per 2004-07-08's libata status report, Silicon Image provided Garzik with docs and sample hardware.)"
per http://linuxmafia.com/faq/Hardware/sata.html#sil24 from 2007

Edit3: an interesting PCI card with Silicon Image controller:
http://www.newegg.com/Product/Product.a ... 6816132001
claims 1.5Gbps, this sounds like BS to me, though it is super cheap!

As you said you are likely using RAID then you will be limited by the speed of the PCI bus. PCI is limited to 133MB/sec and 100 MB/sec is more real world.

For example, your drives can write 100MB/sec. If you had one you would get 100MB/sec if you are lucky on PCI. Two would get 50MB/sec each. Four and you are down to 25MB/sec. If you were using 4 drives in RAID-5 that's only 75MB/sec of usable throughput because of the parity. If you had a pair of PCIe cards you could write to all 4 at 100 MB/sec giving you 300 MB/sec on a RAID-5.

Also, any other devices in your system that are on the PCI bus take away from the total as well. PCIe is dedicated, nothing else is sharing the bandwidth.

The "150MB/s" you see is the drive to controller interface speed, standard SATA 1 speed. If it's on PCI you'll never get it past the controller, it's impossible. Also the "1.5Gbps" speed is also technically correct, it's just the same thing as 150MB/sec expressed in bits and not counting overhead.

You don't want PCI-X cards, they are not the same thing as PCIe. You can use them in PCI slots, but you won't get any benefit from doing so.

DVD drives are probably not going to work, most of these cards support hard drives only. Doesn't hurt to try, but expect it not to work.

I'd say your best bet is a pair of 3132 based cards if you have enough PCIe slots.

Thank you for the generous helping of information. It's hard not to give a $15 card with free shipping a try . But I will probably order several of the PCIe card in the long run (I want to be forwards-compatible, anyway).

I plan to eventually move most/all of these drives to a dedicated NAS/server machine, but my networking options are currently slim (until I move in a few months).

Swapping HDDs is getting annoying, especially since I had to use /dev/sd* instead of UUIDs to get mdadm working for some reason

Get a cheap IBM M1015 or Supermicro U(2)SAS8 card off ebay, with SAS -> sata breakout cables you have 8 channels with full bandwidth via pci-e 4x/8x and the oppurtunity to use SAS expanders for more ports later.

Wibla: While those are good cards they are way overkill for what andymcca needs. I have quite a few IBM/LSI RAID cards at work and they are great but even used they are way more expensive than cheap SATA cards. Plus they are a big waste of power if one is not using the RAID functionality.

Hmm.. Yeah those look very neat, and obviously IBM hardware should have good drivers, but a quick glance at Froogle shows $150-$200 or higher (3x + price increase)
These are definitely something I will keep in mind if I ever build my dream server, though

I am taking this information from ZFSGuru.com which is geared toward FreeBSD, but I know many controllers have issues under Linux.



You probably want to check the Linux compatibility lists here. A more costly route are HBAs that will likely have good support.

SuperMicro USAS-L8i (2x Mini-SAS -> 8x SATA/300)
Intel SASUC8i

Price is why I recommend looking at ebay, these cards can be had NIB/used but verified good for way less than retail, and they are definently worth it vs buying crappy pci sata controllers.

The saying "buy cheap, buy twice" comes to mind in these cases... and by cheap, I mean pci sata controllers that are, well, crap.

andymcca, you building a HTPC/NAS? Just curious.

I went for a 3 drive Raid-5, via mdadm on Lenny, when a built my NAS a while back now. I have an unplugged spare for if/when a drive dies, but I now wish I had gone Raid-6. Raid-6 may be something you will want to look into.

I'll be watching this thread. I'm curious in what you end up trying and end up happy with.

Listen to washu, people!

By the by...
RAID5 and especially RAID6 ain't necessarily the best plan for a single small server. You get the same amount of capacity with 4 drives in RAID6 as you would with RAID1(0) or simply file-based mirroring. RAID6 only makes sense if the main threat to your data and its availability are two drive failures in a row... that's not a common situation at all!

Two drives failing in short succession is actually quite normal, especially if you have them in a RAID array which then has to be rebuilt.

A RAID array (1,10,5, and to an extent 6) will be very vulnerable to a secondary harddrive failure during rebuild, as the rebuild process is very stressfull on harddrives, an already "marginal" drive might (will) keel over from the load. If this happens with a RAID5 array, you will have dataloss. If it happens with a RAID6, it doesn't make a big difference as long as you have replacement drives handy to replace the second failed drive.

Ideally you don't even want to run drives from the same batch in the same server, especially not in RAID, as manufacturing defects in a batch generally affects a quite significant number of drives produced in the batch.

I run a RAID6 with 16x1.5TB LP drives, with two coldspares, backups of essential files offsite, and regular backup to a secondary fileserver with 11x1TB F1 drives in hwraid5 + hotspare. I feel reasonably safe that my data won't be gone tomorrow, but Mr. Murphy has a tendency of rearing his ugly head when you don't want him to...

The morale is that if you don't need RAID(5/6) for high availability of files, and you can implement a setup that will duplicate important stuff over several drives (and preferably servers, heh), you're better off without RAID for "plain storage".

If you want to go RAID, you will need proper HBAs + linux mdraid or solaris/BSD with ZFS.

I often rebuild RAID1 and I've never got a failure. I also stress drives in other circumstances such as making a full backup. I got drive failures in RAID1 arrays as well as in other contexts of course, just not while rebuilding any kind of RAID array (I have no huge array).
I got sector errors on marginal drives when backing up but never while rebuilding. Maybe that's because the drives in my arrays are tested automatically.
This is only anecdotes of course. But, based on my experience, I wouldn't trust your claim without some evidence.

I do not see you making any kind of comparison between the other risks and the risk of drive failure. In my experience, drive failure is generally not the main risk when considering the odds that one's data "won't be gone tomorrow". I've lost data due to drive failures in laptops so I know that not having any kind of redundancy is risky. But it's usually not the main risk.

Availability risks are also not limited to drive failures. So in most cases software RAID1 has more availability than any other kind of RAID (or ZFS for that matter).

If you have 16 drives RAID6 obviously makes a lot more sense than with 4.

You certainly do not need proper HBAs for RAID!
If you have any evidence for your claim that the hardware washu recommends is particularly unreliable, please post it.

As long as you use a PCIe controller that is atleast semi-recent, even a pcie1x controller will give enough throughput for "casual" use (considering pci-e 2.0 gives you quite good BW even on a 1x link).
I see no problems with Washu's recommendations, the point I was trying to get at is that if the OP wants to expand later on, getting that good HBA right off the bat will save him money in the long run.

RAID1 is good for system drives and other data that need max availability, but for any kind of bulk (10-15+TB) storage where you want decent/high availiability and decent sequential performance, RAID6 is the obvious option.

Saying that you don't need a proper HBA for RAID is ludicrous, the last thing you want is a flakey controller that makes the RAID manager drop drives hand over fist. A prime example is the supermicro AOC-SASLP-MV8 that had (and still isnt 100% cured) huge issues with SMART requests concurrent with high I/O load on linux, causing hangups, dropped drives and even kernel oops.

Edit: going over this thread again from top to bottom is amusing, have we passed 60% OT posts yet?

The cards washu recommended are considerably cheaper than the Supermicro card you had trouble with. If the recommendation is sound, this gives the lie to your "buy cheap, buy twice".

We are comparing apples and oranges. The cards I recommended are SAS controllers that support expanders, opening up the route to many more ports in that chassis, or even running a mini-SAS to a dedicated expander chassis. I would rather spend the extra $100 to get a SAS controller now, than have to do so in 6 months or year down the road anyways because the storage needs exceeded expectations.

Example: With a 4/8 port SAS controller you can hook up a HP SAS expander (MSRP: $300, but they can be found for around $200-250 from several dealers) and get 24+ usable SAS/SATA ports.

Also note that the USAS and the SASLP cards are not the same, they have different chipsets.

Wibla: There is no problem with the cards you recommend, but again they are way overkill for 99% of people building a simple NAS. There is no need for the expense and power draw of those sorts of cards. For many enthusiasts on this site the 20+ watts constant draw needed by these cards would not be acceptable. Performance is also not an issue, any of the cheap cards can easily max out a Gig network as long as you are not using the crappy built in RAID.

Also you are dead wrong about needing a high end HBA for RAID. Using an expensive HBA with ZFS or MDADM is simply a waste of both money and power. One of the big selling points of ZFS is removing the need for expensive HBAs.

For example the Sun X4500 "Thumper" is an enterprise grade storage server purpose built for ZFS with 48 drive slots. It has a bunch of cheap marvel controllers, no expensive HBAs.

The Silicon image chips are cheap but they work. Yes, the built in "RAID" is less than useless, but when used as simple controllers they are just fine. I've never seen any issues with the RAID bios getting in the way and I've used them in many builds across many OSes and file systems. Hell *EMC* uses them in some of their controllers. We have several crazy expensive EMC SANs at work and not only do they use Silicon image chips for some functions, all of the RAID is done in pure software. I don't think a simple Linux NAS is going to have any problems with them.

I never stated that you must have a high-end HBA, I stated that you should have a HBA with A) enough bandwidth to not severely hamper performance and B) stable drivers / firmware. Consider that my two main criterias for a "proper" HBA.

The SIL based controllers are fine for that purpose - as long as you know the limitations (expandability, raw throughput bottlenecketc).

Storage is quite easily divided into three camps. The "I need 4 ports and something that works" camp, the "I will need an expander chassis N months down the line" camp, and the "I'll just call EMC/lefthand" camp.

At this point, we're so far off topic and into semantics that there is no real point in continuing this discussion. I apologize for derailing the thread, but I do not apologize for my opinions on storage solutions, they have served me well for the last few years, with 0 dataloss and minimal downtime.

PS: The thumper is fairly old, and Sun is using custom tailored HW (pci-x controllers, pci-x/pci-e bridges etc) to get 6 of those marvell 8port SATA2 controllers hooked up. The Supermicro AOC-SAT2-MV8 using that exact chipset is still $100(!) retail at newegg