consumer-grade SATA expansion under linux?

Silencing hard drives, optical drives and other storage devices

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Post Reply
andymcca
Posts: 404
Joined: Mon Jan 11, 2010 8:19 am
Location: Boston, MA, USA

consumer-grade SATA expansion under linux?

Post by andymcca » Thu May 05, 2011 9:56 am

I see PCI and PCIe cards for SATA expansion. Am I missing an alternative?

And which cards / chips do people recommend for easy linux (kernel 2.6.32or later) installation? Feel free to offer solutions if you don't know about linux drivers, too.

*Transfer speed is not hugely important (100MB/s 1-channel throughput would be nice)
*Assume no preference in interface type
*Internal expansion

Also - I'm assuming noise is not an issue here? Do these ever keep the drives spinning in spite hdparm spindown_time?

Edit: Ideally I am looking for ~4 SATA ports, but multiple cards are an option. And I will be using mdadm software RAID, if any.

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: consumer-grade SATA expansion under linux?

Post by washu » Thu May 05, 2011 10:29 am

If you want 100MB/s per port than PCI cards are off your list. You might get that on one port, but never on 2 or more at the same time.

A couple of cheap 2 port Silicon Image PCIe cards would do the trick just fine. They have "RAID" but linux will just ignore it and treat them as SATA controllers. Sleep works fine on them in Windows, don't see why it wouldn't in linux. There are 4 port cards out there too, but beware that some are built with PCI to PCIe bridge chips and not really PCIe cards.

Why not put your RAID drives on your MB ports? You'll get better performance that way. If you need the extra ports use the add on cards for boot drives and/or non-raid storage.

andymcca
Posts: 404
Joined: Mon Jan 11, 2010 8:19 am
Location: Boston, MA, USA

Re: consumer-grade SATA expansion under linux?

Post by andymcca » Thu May 05, 2011 11:02 am

This would be primarily for expansion storage (speed-critical data will be on motherboard).

I meant 100MB/s while writing to a single HDD (I am assuming that bandwidth is fungible, or will effectively be via software RAID striping of partitions)

I see a few PCI cards which claim "150MB/s", which I assume is total bandwidth, but I don't know if this is accurate.

Per your post,
Rosewill RC-211 Silicon Image 2 port SATA II PCI Express Host Controller Card
http://www.newegg.com/Product/Product.a ... 6816132008
certainly looks attractive, if drivers are a sure thing!

Edit: I'll probably also be moving all DVD drives to these to free up motherboard connections, unless someone says this is unwise

edit2: good call on chipset.
"Silicon Image 3124/3124-2 (chip in 4-port SATA-II PCI-X cards) and 3132 (chip in 2-port SATA-II PCI Express cards) (Silicon Image, Inc., formerly CMD Technology, Inc.) — libata's sata_sil24 driver (production quality) supports Silicon Image 3124 (2005-08) and also the follow-on 2-port PCI Express SATA-II successor chip, the Silicon Image 3132. (Per 2004-07-08's libata status report, Silicon Image provided Garzik with docs and sample hardware.)"
per http://linuxmafia.com/faq/Hardware/sata.html#sil24 from 2007

Edit3: an interesting PCI card with Silicon Image controller:
http://www.newegg.com/Product/Product.a ... 6816132001
claims 1.5Gbps, this sounds like BS to me, though it is super cheap!

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: consumer-grade SATA expansion under linux?

Post by washu » Thu May 05, 2011 12:21 pm

As you said you are likely using RAID then you will be limited by the speed of the PCI bus. PCI is limited to 133MB/sec and 100 MB/sec is more real world.

For example, your drives can write 100MB/sec. If you had one you would get 100MB/sec if you are lucky on PCI. Two would get 50MB/sec each. Four and you are down to 25MB/sec. If you were using 4 drives in RAID-5 that's only 75MB/sec of usable throughput because of the parity. If you had a pair of PCIe cards you could write to all 4 at 100 MB/sec giving you 300 MB/sec on a RAID-5.

Also, any other devices in your system that are on the PCI bus take away from the total as well. PCIe is dedicated, nothing else is sharing the bandwidth.

The "150MB/s" you see is the drive to controller interface speed, standard SATA 1 speed. If it's on PCI you'll never get it past the controller, it's impossible. Also the "1.5Gbps" speed is also technically correct, it's just the same thing as 150MB/sec expressed in bits and not counting overhead.

You don't want PCI-X cards, they are not the same thing as PCIe. You can use them in PCI slots, but you won't get any benefit from doing so.

DVD drives are probably not going to work, most of these cards support hard drives only. Doesn't hurt to try, but expect it not to work.

I'd say your best bet is a pair of 3132 based cards if you have enough PCIe slots.

andymcca
Posts: 404
Joined: Mon Jan 11, 2010 8:19 am
Location: Boston, MA, USA

Re: consumer-grade SATA expansion under linux?

Post by andymcca » Thu May 05, 2011 12:44 pm

Thank you for the generous helping of information. It's hard not to give a $15 card with free shipping a try :). But I will probably order several of the PCIe card in the long run (I want to be forwards-compatible, anyway).

I plan to eventually move most/all of these drives to a dedicated NAS/server machine, but my networking options are currently slim (until I move in a few months).

Swapping HDDs is getting annoying, especially since I had to use /dev/sd* instead of UUIDs to get mdadm working for some reason :)

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Re: consumer-grade SATA expansion under linux?

Post by Wibla » Thu May 05, 2011 2:26 pm

Get a cheap IBM M1015 or Supermicro U(2)SAS8 card off ebay, with SAS -> sata breakout cables you have 8 channels with full bandwidth via pci-e 4x/8x and the oppurtunity to use SAS expanders for more ports later.

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: consumer-grade SATA expansion under linux?

Post by washu » Thu May 05, 2011 3:54 pm

Wibla: While those are good cards they are way overkill for what andymcca needs. I have quite a few IBM/LSI RAID cards at work and they are great but even used they are way more expensive than cheap SATA cards. Plus they are a big waste of power if one is not using the RAID functionality.

andymcca
Posts: 404
Joined: Mon Jan 11, 2010 8:19 am
Location: Boston, MA, USA

Re: consumer-grade SATA expansion under linux?

Post by andymcca » Fri May 06, 2011 4:03 am

Hmm.. Yeah those look very neat, and obviously IBM hardware should have good drivers, but a quick glance at Froogle shows $150-$200 or higher (3x + price increase) :)
These are definitely something I will keep in mind if I ever build my dream server, though :)

Dirge
Posts: 111
Joined: Sun Apr 11, 2004 8:55 pm
Location: New Zealand

Re: consumer-grade SATA expansion under linux?

Post by Dirge » Sat May 07, 2011 12:13 am

I am taking this information from ZFSGuru.com which is geared toward FreeBSD, but I know many controllers have issues under Linux.
FakeRAID add-on PCI/PCI-express controllers
These include chipsets like Silicon Image, Promise FastTrak and JMicron. These offer RAID support via Windows-only drivers. When you boot into Linux or FreeBSD, it is seen as SATA controller instead. But there's rough edges with many of these cheap controllers:

Often you cannot turn off the RAID functionality.
You may not touch the last sector on each HDD, which is used by the controllers boot firmware which use it to 'taste' the HDDs for any existing RAID configuration. If you use the entire disk under FreeBSD and write to the last sector, this may cause problems when you reboot and the controller gives you an annoying error. The workaround to this issue is to create partitions which ensure that you never write to the last sector of the drive. Also do not use geom labels on the physical drives; only GPT labels.
BIOS issues may cause boot failures, especially if you mix several controllers. I don't have much information about this issue. Try different PCI-express slots.
They are often bandwidth-limited. PCI is very bad for performance and PCI-express x1 generation 1 is only 250MB/s full-duplex; with just two HDDs you could be reaching this limit in sequential workloads.
Under Windows, the drivers would quickly detach disks and degrade/fail/split the array whenever a disk misbehaves or has a hickup; not that reliable at all!
You probably want to check the Linux compatibility lists here. A more costly route are HBAs that will likely have good support.

SuperMicro USAS-L8i (2x Mini-SAS -> 8x SATA/300)
Intel SASUC8i

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Re: consumer-grade SATA expansion under linux?

Post by Wibla » Sat May 07, 2011 11:27 am

Price is why I recommend looking at ebay, these cards can be had NIB/used but verified good for way less than retail, and they are definently worth it vs buying crappy pci sata controllers.

The saying "buy cheap, buy twice" comes to mind in these cases... and by cheap, I mean pci sata controllers that are, well, crap.

DrCR
Posts: 538
Joined: Sun Jun 13, 2004 11:55 am

Re: consumer-grade SATA expansion under linux?

Post by DrCR » Sat May 07, 2011 1:30 pm

andymcca, you building a HTPC/NAS? Just curious.

I went for a 3 drive Raid-5, via mdadm on Lenny, when a built my NAS a while back now. I have an unplugged spare for if/when a drive dies, but I now wish I had gone Raid-6. Raid-6 may be something you will want to look into.

I'll be watching this thread. I'm curious in what you end up trying and end up happy with.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: consumer-grade SATA expansion under linux?

Post by HFat » Sun May 08, 2011 11:45 am

Listen to washu, people!

By the by...
RAID5 and especially RAID6 ain't necessarily the best plan for a single small server. You get the same amount of capacity with 4 drives in RAID6 as you would with RAID1(0) or simply file-based mirroring. RAID6 only makes sense if the main threat to your data and its availability are two drive failures in a row... that's not a common situation at all!

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Re: consumer-grade SATA expansion under linux?

Post by Wibla » Mon May 09, 2011 7:32 am

Two drives failing in short succession is actually quite normal, especially if you have them in a RAID array which then has to be rebuilt.

A RAID array (1,10,5, and to an extent 6) will be very vulnerable to a secondary harddrive failure during rebuild, as the rebuild process is very stressfull on harddrives, an already "marginal" drive might (will) keel over from the load. If this happens with a RAID5 array, you will have dataloss. If it happens with a RAID6, it doesn't make a big difference as long as you have replacement drives handy to replace the second failed drive.

Ideally you don't even want to run drives from the same batch in the same server, especially not in RAID, as manufacturing defects in a batch generally affects a quite significant number of drives produced in the batch.

I run a RAID6 with 16x1.5TB LP drives, with two coldspares, backups of essential files offsite, and regular backup to a secondary fileserver with 11x1TB F1 drives in hwraid5 + hotspare. I feel reasonably safe that my data won't be gone tomorrow, but Mr. Murphy has a tendency of rearing his ugly head when you don't want him to...

The morale is that if you don't need RAID(5/6) for high availability of files, and you can implement a setup that will duplicate important stuff over several drives (and preferably servers, heh), you're better off without RAID for "plain storage".

If you want to go RAID, you will need proper HBAs + linux mdraid or solaris/BSD with ZFS.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: consumer-grade SATA expansion under linux?

Post by HFat » Mon May 09, 2011 10:09 am

I often rebuild RAID1 and I've never got a failure. I also stress drives in other circumstances such as making a full backup. I got drive failures in RAID1 arrays as well as in other contexts of course, just not while rebuilding any kind of RAID array (I have no huge array).
I got sector errors on marginal drives when backing up but never while rebuilding. Maybe that's because the drives in my arrays are tested automatically.
This is only anecdotes of course. But, based on my experience, I wouldn't trust your claim without some evidence.

I do not see you making any kind of comparison between the other risks and the risk of drive failure. In my experience, drive failure is generally not the main risk when considering the odds that one's data "won't be gone tomorrow". I've lost data due to drive failures in laptops so I know that not having any kind of redundancy is risky. But it's usually not the main risk.

Availability risks are also not limited to drive failures. So in most cases software RAID1 has more availability than any other kind of RAID (or ZFS for that matter).

If you have 16 drives RAID6 obviously makes a lot more sense than with 4.

You certainly do not need proper HBAs for RAID!
If you have any evidence for your claim that the hardware washu recommends is particularly unreliable, please post it.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Re: consumer-grade SATA expansion under linux?

Post by Wibla » Mon May 09, 2011 11:27 am

As long as you use a PCIe controller that is atleast semi-recent, even a pcie1x controller will give enough throughput for "casual" use (considering pci-e 2.0 gives you quite good BW even on a 1x link).
I see no problems with Washu's recommendations, the point I was trying to get at is that if the OP wants to expand later on, getting that good HBA right off the bat will save him money in the long run.

RAID1 is good for system drives and other data that need max availability, but for any kind of bulk (10-15+TB) storage where you want decent/high availiability and decent sequential performance, RAID6 is the obvious option.

Saying that you don't need a proper HBA for RAID is ludicrous, the last thing you want is a flakey controller that makes the RAID manager drop drives hand over fist. A prime example is the supermicro AOC-SASLP-MV8 that had (and still isnt 100% cured) huge issues with SMART requests concurrent with high I/O load on linux, causing hangups, dropped drives and even kernel oops.

Edit: going over this thread again from top to bottom is amusing, have we passed 60% OT posts yet? :D

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: consumer-grade SATA expansion under linux?

Post by HFat » Mon May 09, 2011 11:53 am

The cards washu recommended are considerably cheaper than the Supermicro card you had trouble with. If the recommendation is sound, this gives the lie to your "buy cheap, buy twice".

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Re: consumer-grade SATA expansion under linux?

Post by Wibla » Mon May 09, 2011 1:23 pm

HFat wrote:The cards washu recommended are considerably cheaper than the Supermicro card you had trouble with. If the recommendation is sound, this gives the lie to your "buy cheap, buy twice".
We are comparing apples and oranges. The cards I recommended are SAS controllers that support expanders, opening up the route to many more ports in that chassis, or even running a mini-SAS to a dedicated expander chassis. I would rather spend the extra $100 to get a SAS controller now, than have to do so in 6 months or year down the road anyways because the storage needs exceeded expectations.

Example: With a 4/8 port SAS controller you can hook up a HP SAS expander (MSRP: $300, but they can be found for around $200-250 from several dealers) and get 24+ usable SAS/SATA ports.

Also note that the USAS and the SASLP cards are not the same, they have different chipsets.

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: consumer-grade SATA expansion under linux?

Post by washu » Mon May 09, 2011 5:41 pm

Wibla: There is no problem with the cards you recommend, but again they are way overkill for 99% of people building a simple NAS. There is no need for the expense and power draw of those sorts of cards. For many enthusiasts on this site the 20+ watts constant draw needed by these cards would not be acceptable. Performance is also not an issue, any of the cheap cards can easily max out a Gig network as long as you are not using the crappy built in RAID.

Also you are dead wrong about needing a high end HBA for RAID. Using an expensive HBA with ZFS or MDADM is simply a waste of both money and power. One of the big selling points of ZFS is removing the need for expensive HBAs.

For example the Sun X4500 "Thumper" is an enterprise grade storage server purpose built for ZFS with 48 drive slots. It has a bunch of cheap marvel controllers, no expensive HBAs.

The Silicon image chips are cheap but they work. Yes, the built in "RAID" is less than useless, but when used as simple controllers they are just fine. I've never seen any issues with the RAID bios getting in the way and I've used them in many builds across many OSes and file systems. Hell *EMC* uses them in some of their controllers. We have several crazy expensive EMC SANs at work and not only do they use Silicon image chips for some functions, all of the RAID is done in pure software. I don't think a simple Linux NAS is going to have any problems with them.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Re: consumer-grade SATA expansion under linux?

Post by Wibla » Tue May 10, 2011 7:28 am

I never stated that you must have a high-end HBA, I stated that you should have a HBA with A) enough bandwidth to not severely hamper performance and B) stable drivers / firmware. Consider that my two main criterias for a "proper" HBA.

The SIL based controllers are fine for that purpose - as long as you know the limitations (expandability, raw throughput bottlenecketc).

Storage is quite easily divided into three camps. The "I need 4 ports and something that works" camp, the "I will need an expander chassis N months down the line" camp, and the "I'll just call EMC/lefthand" camp.

At this point, we're so far off topic and into semantics that there is no real point in continuing this discussion. I apologize for derailing the thread, but I do not apologize for my opinions on storage solutions, they have served me well for the last few years, with 0 dataloss and minimal downtime.

PS: The thumper is fairly old, and Sun is using custom tailored HW (pci-x controllers, pci-x/pci-e bridges etc) to get 6 of those marvell 8port SATA2 controllers hooked up. The Supermicro AOC-SAT2-MV8 using that exact chipset is still $100(!) retail at newegg

Post Reply