Hi all,

Well, with thanks to your advice, I bought the parts discussed here, including 2x1TB drives which I intend to RAID1 for data security.
http://www.silentpcreview.com/forums/viewtopic.php?t=52809

I am hoping that some of you here have done software RAID1 using Linux and can advise me? In particular with Ubuntu (chosen for it's larger community base).

I have installed Ubuntu 8.1, created 3 partitions on each drives - 15gb for /, 4gb for swap, and the 981gb rest for /home. I specified these were for RAID and then created the multiply devices as per http://kuparinen.org/martti/comp/ubuntu/en/raid.html.
On boot, Linux took some 4hours to synch the arrays, and is now happy.

Grub
Next, I need to sort grub. Currently, grub is just on the first harddisk, so on boot if the 1st harddisk is down, the system will not be able to start. Reading around, I need to add 'fallback' to grub's menu.lst, and then point it at a Kernel declaration just like http://www.howtoforge.com/software-raid1-grub-boot-debian-etch-p2.
On booting with just one harddisk, the system asks me what to do, but tells me that I can add bootdegraded=true to the kernel boot options - any idea where these go? menu.lst has something called "kopt", so I'm thinking here.

Testing
Next, I need to test this to prove to myself it is working reliably in a failure.
The scenarios are
1) Drive 1 fails during boot
2) Drive 2 fails during boot
3) Drive 1 fails whilst machine on
4) Drive 2 fails whilst machine on
I'm guessing that when I put a "new" drive back in, I can use mdadm to add the drive to the array. When I do this, I am guessing I will need to go through synching again?

Reporting
Is there a way that I can set up reporting, so if a drive fails I am notified. A silently failing drive is no good as from the outside all will appear well!


I'm just hoping one of you has done this or something similar before...

Thanks in advance,

MG

Hi,

I have raid 1 on Ubuntu 8.10, but I haven't tested failure
modes much. I did find on launchpad that grub-install now
puts grub on all disks on the array and I tried it on my
machine with the following results:

smartmontools will install the basics, smart-notifier puts a GUI on top of the reporting.

nagios can be set up to do the same, but it's like using a nuke to swat a fly.

I use gkrellm - it's always open down the left side of my desktop for monitoring system temps etc.

The neat thing is it's set up to graph read/write activity on each disk, so when I'm using RAID 5 goodness there's 4 graphs of read or write activity. If a drive fails, it will be visible.

There's alerts and stuff that can be set up using mdadm, but I haven't bothered as the RAID 5 is basically a torrent archive and backup for other data stored elsewhere - nothing critical.

adding kernel boot parameters to 'kopt' would affect every new kernel image installed (at least when using ubuntu-packaged kernel images with corresponding hooks in post-installation scripts). you still have to add this option to every 'kernel'-directive of the boot items in menu.lst you wish to be affected by the boot option.
alternatively, you could manually add any kernel boot options directly to the 'kernel' directive of every boot item in menu.lst. in this case, you would want to edit your menu.lst every time a new kernel image package is installed.

also, don't forget to make sure grub's stage1 is installed to second disk's master boot record, too.



it depends. which stage of the boot process? what kind of grub setup? bootdegraded option used or not?
basically, if a disk (it does not matter at all, if it would be the first or the second one) fails AFTER the md-subsystem is loaded, the failure should be detected by the md subsystem, MD arrays using the failed disk would get automatically degraded by removal of any physical device related to the failed disk from any running MD array.
if a disk fails before MD subsystem is started, your BIOS and your bootloader are both set up to be able to boot from either drive, and you are using the bootdegraded option, your system will come up, and any MD device using a physical device on the failed disk would run in degraded mode (of course, it depends on raid-level used. a raid0 MD-device with one of the physical devices on a failed disk won't be able to start at all, for example).



exaclty. you'll have to partition the 'new' drive accordingly, add the 'new' partitions to corresponding arrays (hot-adding works, you don't have to stop an array to add a new physical device). this would trigger a resync. you'll be able to use the array during resync, but the performance would be severely degraded.
it does not matter at all if 'disk1' or 'disk2' fails.



mdadm can be set up to send out notifications. /etc/mdadm/mdadm.conf and/or /etc/default/mdadm are the files to check & modify to set the notifications up. it should be trivial to set up a cronjob checking for md-device status and alerting you in any other way in case of a failure -- page your mobile phone, buzz the buzzer, automatically place an order for a replacement disk... whatever you need/consider useful.

smartmontools is a nice-to-have as a kind of 'early detection' system, but I've seen enough dying drives with perfect SMART-values. don't forget to set smartmontools to perform regular self-testing routines (-s option of smartctl), this makes it much more useful. you may be able to detect possible drive failures long before MD is able to tell the drive is failing, thus possibly giving you more time to prepare everything for the actual failure.

oh, and don't forget to backup your raid1 arrays! i just cannot stress it enough that no redundant raid setup could ever be a replacement for a regular backup routine. raid1 gives you an operation reliability bonus (beeing able to keep the services running while you're replacing a failed disk), but does not affect the safety of your data in any way. even the cheapest/most simple possible backup solution like rsyncing to some kind of external storage device (like a usb-connected hard disk) would contribute much more to the safety of your data than any kind of raid-setup. another issue to be aware of is a problem related to most journalling filesystems (ext3fs, for example) and modern disks using out-of-order writeback caching mechanisms (see http://lwn.net/Articles/283161/, http://en.wikipedia.org/wiki/Ext3#No_checksumming_in_journal...)

Hi all,

Firstly, I wanted to thank you all for your responses, and to apologise for not responding earlier (a mixture of being away from work and looking at this forum without logging in so not getting email alerts).

I have made a lot of progress with the RAIDing and was actually very impressed with how easy it was for a Linux novice like me to set it up.

With grub, the net offered a number of options.
One was specifying the default kernel stanza, then specifying a different one for fallback
After much tweaking, I found that I didn't need to do any of that. I just specified both drives as the default stanza (through grub), then installed both drives as (HD0,0), so if the first drive is present it gets loaded, or if the second drive is present it gets loaded. My menu list just needed to contain the below (note the bootdegraded option - without this the machine hangs on boot waiting for you to tell it what to do - not ideal for an unattended server).


I have since tested booting the machine with a drive missing. mdadm gives me [U_] or [_U] to tell me that the RAID array is degraded, and using mdadm -add I can snap the other drive back in the array (4.5hrs for the 1TB). As there aren't a lot of changes to the data it's a shame it can't recognise that this was the drive previously used (say by model number or something) then you could just 'resynchronise' them in some way, rather than it destroying all the data on the 2nd drive and rewriting (which takes the time).
However, given that this shouldn't happen all that often *crosses fingers* it's not a major problem.

The monitoring side has given me more grief - well, the monitoring was easy to set up using smartmontools (as advised), but getting the emails to be sent by this machine proved an incredible challenge for something that seems so simple (I wont bore you with the details).
Being a Java programmer (and knowing not a lot about c or Linux), I am since investigating writing my own little app to periodically check the output of mdadm and smartctl (smartmontools) and email me if something is found. Whilst this over-engineering probably appals you, it plays to my strengths moreso.

Thanks very much for all your help and advice,

MG

well, on a second glance you'd notice that it's not really an issue. in a real-world use scenario for a raid array, there would be exactly two reasons for a disk to get pulled from an array: a test, just like in your case, and a hardware failure event. in the second case, you'd never want to force the failed drive back into array, you'd just get a fresh one, pop it in and rebuild your array. if you're just testing, than it would only make sense if your test lets you recreate a real-world failure as closely as possible -- including the full rebuild of the array, so you would not want any part of the md subsystem to behave defferently somehow.
additionally, the rebuild process is not that primitive as it could seem, as your array stays online, contained partitions could be mounted and be written to, so the md subsystem has to keep track of it, too, not just duplicate the partitions blockwise. running a degraded array and having a rebuild running already does impose a heavy performance hit on your system, now imagine slowing it even further by employing some kind of more advanced and complicated resync techniques. this could slow your server to a crawl, rendering the main benefit you get from running a RAID1 array useless -- the ability to keep your system/services up and running until you replace the faulty hardware. it also adds to complexity of a software solution which is meant to be as robust as possible, which is never a good thing (which you probably know well enough, beeing a programmer yourself).
in this case, less is more.



it doesn't appal me at all, it's always great to see people having fun tinkering with stuff and creating their own solutions