ZFS under windows...

[domb]

im not sure if this is the most genius idea or most stupid idea but what do you think of this...

i want to use zfs for my media centre to store my movies etc. this is because it offers raid-z redundancy and i can use whatever drives i have lying around and add more when i need them. the other thing is i want to use it under windows which isn't possible at the moment. i also dont really want to use a seperate machine as i dont have the space. the other option is a drobo or something but they are rather expensive.

now my idea was...

how about using VirtualBox under Windows to boot an imaged version of OpenSolaris and use this to mount a ZFS volume at startup?

Means I can stuff 7 drives in the Silverstone LC10-E i was going to get to replace my antec fusion, use one or maybe 2 drives as a system volumes and use the other 5 or 6 as zfs storage!

genius or stupid?

[theycallmebruce]

Seems like a reasonable way to achieve your goal. You may find performance is an issue, but there's only one way to find out! Give it a go and let us know.

Even if the disk I/O is a bit slow, I wouldn't think that playing compressed video would required lightning disk throughput anyway.

EDIT: There is an awesome flamefest on this topic here: http://www.opensolaris.org/jive/thread. ... 3&tstart=0

[domb]

lol. doesn't seem like such a good idea now! im certainly not risking losing my data to try it. if i had a load of 1tb drives lying around i would but i can't afford that and a new case at the mo.

thats very annoying as I have a 1tb, 320gb and a 160gb i was going to stick in a raidz pool and use that until i could get myself a couple more 1tb drives... hence the point of using zfs.

like you said it doesn't have to be blazingly fast, just quick enough to play 40Mbps video.

Maybee i'll have to wait until i can get a few more drives and just use raid 5.

[DanielSmedegaardBuus]

FWIW, and I realize this is an aging topic, I just set up something similar to this.

Windows XP x64 host, E2200 @3 GHz
VirtualBox 4.0.2
Ubuntu Server 10.10 guest installed with -virtual kernel
ZFS-FUSE in RAIDZ2 accessing 7 members via VBox raw disks (writethrough mode, i.e. snapshot independent)

Performance is pretty abysmal. As this CPU is a budget one, without VT-x, I can only emulate 1 core. Also, I have to settle for the i386 version of Ubuntu.

Individually doing hdparm -Tt on each of the seven members shows pretty standard performance at around 55-60 MB/sec buffered. The bottleneck here quickly becomes the CPU. When copying from my zpool to /dev/null, the one core available is completely saturated for the duration, and transfer speeds max out at ~10 MB/sec.

This would be adequate, I guess, for just serving files to a media center application on the host, but is not really acceptable if you need to serve files quickly via Gbit ether (like for video editing).

In comparison, a reboot into my native Kubuntu 10.10 installation gives me about 150 MB/sec in the zpool to /dev/null test, and I can push files through Gbit ether at ~75 MB/sec.

I would assume using OpenSolaris as the guest OS would greatly enhance the speeds, though OpenSolaris is now abandonware, and the ZFS revision available there is pretty outdated, if I'm not mistaken. A better CPU with VT-x would also help alot.

This is just for reference

[DanielSmedegaardBuus]

Okay, for those still interested, I tried the same setup on the same rig with the same everything but using VMware Workstation 7.1 instead of VirtualBox 4.0.2. I haven't been able to access all of my disks as raw disks using VMware Server (neither 1.0.x nor 2.0.x), but Workstation will!

VMware Workstation will also use both cores on this non-VT-x CPU, so I would naturally assume - with the CPU being the bottleneck - that I'd get ~20 MB/s at least on the same test, i.e. twice the speed as I have twice the number of cores.

However, VMware suprised me by actually churning out 65 MB/s!!! That's 6.5x faster in raw numbers, and 3.25 times faster per core! I'm astonished! This is far, far enough for pretty much anything I could want.

I haven't done a native test on (K)ubuntu to compare with, so I'll do that next up.

[DanielSmedegaardBuus]

...and the final test, using ZFS-FUSE with the same disks in a real Kubuntu OS, yields at least 135 MB/s in the same test (up to 170 MB/s at times).

But still, the VMware numbers are pretty good here!

[domb]

I was just thinking about this again and stumbled upon my old thread! Didn't realise someone had actually done it!

I ended up setting up a 4 disk RAID 5 with 1TB drives... with an ICR9 on my P5E-VM HDMI.

Now I'm thinking of expanding it which is a problem and thought about ZFS again. I could buy 4 x 2TB drives, setup a ZFS volume, copy the data to it, then add the 4 x 1TB drives and voila! One 12TB ZFS volume.

[DanielSmedegaardBuus]

Now I'm thinking of expanding it which is a problem and thought about ZFS again. I could buy 4 x 2TB drives, setup a ZFS volume, copy the data to it, then add the 4 x 1TB drives and voila! One 12TB ZFS volume.

Don't forget: ZFS cannot expand "horizontally", i.e. you cannot add more members, but you can grow when members become larger. But you probably already know that since you're talking about adding 1TB drives to 2TB drives

Are you using raw disks or virtual ones? Raw disks would at least require you to use some middle layer, I'm using mdadm, but LVM would work, too. Not sure which has less overhead. Can you span virtual disks across different physical locations? Like split to 2GB segments, and the have 100 on a 2TB drive and the remaining 50 on a 1TB drive?

I'm kind of torn between which solution would be better when using ZFS members that are spanning several physical devices, using a guest-level middle layer or spanning virtual disk segment on the host. Having virtual disks in segments on the host, you'd be able to switch between different virtualized OSes, like OpenNevada, IllumOS, FreeBSD, Linux or Solaris. You could get a nice performance increase using an OS with native ZFS support. On the other hand, you would never be able to use the disks directly if you wanted to run bare-bone...

Choices It's fun though, all this, don't you agree? Please let us know what you're gonna do, and benchmarks, too!

[domb]

It seems you've done more research into it than me! I was just looking at this enclosure and want a file system I can expand over time.

http://www.google.co.uk/products/catalo ... #scoring=p

Preferably something native to windows. I have a large Silverstone enclosure so if possible I'd rather not even buy that Edgestor enclosure as I can fit 7 drives in my enclosure. I was really hoping ZFS would have taken off by now but it just never happened. Is there anything else similar to ZRAID and native?

[DanielSmedegaardBuus]

It seems you've done more research into it than me!

You have no idea how much time I've used fiddling with this array Also, using ZFS, even on FUSE, is fantastic. It's very very stable, and for a collector like me, it gives so much piece of mind to know that ZFS will know and fix file errors on-the-fly and that I can always do "zpool status" and see a list of any files that possibly couldn't be fixed even with double parity (I'm running RAIDZ2 on 7*2 TB members, of which one member is 4*½TB and two are 2*1TB, both in "linear" (i.e. JBOD) mdadm arrays for the middle layer).

I was just looking at this enclosure and want a file system I can expand over time. Preferably something native to windows now. I have a large Silverstone enclosure so if possible I'd rather not even buy that Edgestor enclosure as I can fit 7 drives in my enclosure. I was really hoping ZFS would have taken off by now but it just never happened. Is there anything else similar to ZRAID and native?

Not sure exactly what you mean by "ZRAID and native". Do you mean like, a solution where you don't use virtual disk files on the host but rather raw disks (or even a bare-metal OS not virtualized), or a filesystem that behaves like ZFS? In the latter case, btrfs for Linux is coming along very nicely, and the versions with the newest 2.6.37/8 kernels should be very stable. I'm using it on my laptop just as a regular fs, but haven't tried it as a RAID provider (yet, I'm gonna use it on my HTPC/fileserver for RAID-1'ing a couple of old ATA disks used as temporary download space).

Have you already bought the enclosure? I'm only asking because I've done a bit of case modding over time to be able to fit all my disks in them. You can really stuff quite a few disks in a regular case with a bit of creativity, a hacksaw, some metal bars, screws and lady's nylon stockings It'll even look good, you'll get better airflow, cooler operation, less noise and less dust in your case! I fitted 11 drives in a regular mid-tower case, and have a larger Lian Li case for my HTPC which holds 18 drives plus an SSD for the OS. You just have to say bye-bye to all the 5.25" bays at the front of your case

About the expandability, I'm gonna say this - with the disclaimer that I haven't followed best practices, but rather tried to find a way where I could use drives of different sizes and be able to expand the array later on, even with the ZFS limitation that you cannot add members:

Using a middle-layer like mdadm or LVM is really nifty in that you can create ZFS member drives using any combination of smaller physical drives. Some will argue that this makes an array member more likely to fail, but I don't really agree to that. Statistically, you can't say that a member comprised of three drives is three times more likely to fail than one made of just one member. Harddrives will fail eventually, that's a given, but each of the three disks should - statistically - last just as long as one, so it should be the same on average. Of course, if we accept that any drive may be "bad" from the factory and will fail quickly, the risk of getting one of those and have it affect your member would be three times as high. On the other hand, you could argue that each of the three disks is only carrying 1/3 of the load of that member (so long as you use linear/JBOD in the middle layer), so if anything they should last longer than a single disk. Also, random seeks on the filesystem level would on average be quicker, as you have roughly three times the number of drive heads and platters to move around poking at your data when compared to a single disk.

Add to that the possibility of being able to expand you array. Point in fact your situation. Not sure if you're actually running ZFS yet, but let's say you currently have a 4*1TB ZFS RAIDZ array. That's 3TB of data. Since you cannot add new members to a ZFS array, buying 4 2TB drives and following best practices you'd have to create a new 4*2TB RAIDZ vdev and add it to the pool, adding 6TB of data. That's 9TB total.

If, however, those initial 4*1TB members were actually on a middle layer like md or LVM, you could chose to add the new 2TB drives to the existing middle layer arrays instead of creating a new vdev. In that case, each member would expand to 3TB, and ZFS would automatically grow to use the new space, ending up at - also - 9TB total.

In this exact setup, you wouldn't gain any extra space using the middle layer, but given a larger number of members and a higher level of redundancy, like RAIDZ2, the difference can be much larger. Point in fact, my setup, if using best practices:
4*½TB physical disks in RAIDZ2 vdev: 1TB
4*1TB physical disks in RAIDZ2 vdev: 2TB
4*2TB physical disks in RAIDZ2 vdev: 4TB
Total, using best practices: 7TB effective. Using my approach:
7*2TB middle layer linear array members: 10TB
Total, using middle layers: 10TB effective.

That's a very noticeable difference, +3TB with the same disks!

Now, about when disks fail. I recently had 3 drives failing simultaneously (two with bad sectors, and one just going into complete convulsions). It was pretty quickly fixed by replacing each of the physical devices and mirroring the old faulty device onto it. Re-add to the middle layer arrays, and do a scrub on ZFS. As most of the data was just fine, it was mostly reading and fixing the few errors that were caused by bad sectors.

Hm. I'm rambling a bit here But it's nice to be able to pass some of this info on to someone else. I didn't find much useful info when I started on my venture back then, so if you need any info or tips, ask away - I have done a lot of thinking and testing doing this

Gotta go to the butcher and pix up some meat, do some shopping for beer, but I'll be back later and will check back.

Daniel

[DanielSmedegaardBuus]

Just gonna add one more advantage of the middle layer: Drives become larger all the time. When I bought the 500GB drives that are in my fileserver, the were top of the pops. Because I've assembled them using a middle layer, I can replace them over time with larger drives as they fail. And in replacing them, I gain the possibility of adding even more drives as they leave empty bays behind.

The reason I modded my fileserver case to be able to hold 18 drives is because I was using 18 drives until just recently. I had 8 500GB drives, 6 1TB drives, 2 2TB drives, and two old ATA drives for the OS. I replace 4 500GB and 2 1TB drives with 2 2TB drives. That's four vacant slots right there. I could remove the two ATA drives as I'm now also using an SSD for the OS. That would be 6 vacant slots. That's just one shy of being able to add a complete line of new top of the pops drives (like 3TB ones) to expand the array instantaneously up to 250% its current capacity

Oh, must stop rambling. Must buy beer and meat. Must simmer in pressure cooker for many hours...

[domb]

Not sure exactly what you mean by "ZRAID and native". Do you mean like, a solution where you don't use virtual disk files on the host but rather raw disks (or even a bare-metal OS not virtualized), or a filesystem that behaves like ZFS? In the latter case, btrfs for Linux is coming along very nicely, and the versions with the newest 2.6.37/8 kernels should be very stable. I'm using it on my laptop just as a regular fs, but haven't tried it as a RAID provider (yet, I'm gonna use it on my HTPC/fileserver for RAID-1'ing a couple of old ATA disks used as temporary download space).

I meant is there anything available that runs natively in Windows that has functionality like ZFS's RAIDZ.

I really can't be botehred with setting up a Linux file server. I don't even have the space for it. I run MediaPortal and everything is setup perfectly at the moment. Just need a decent file system that I can expand.

EDIT

Sorry didn't mean to be blunt there! I take on-board your advice and appreciate you writing it! Busy at work at the moment so not had time to read your post properly.

Just thinking again and I guess I could run a small Linux build with good ZFS support in a VM and get it to boot at startup after Windows has. Then share it using CIFS to the Windows machine.

I'll have to get some spare drives and much around with it.

I would forget the chassis I suggested, it was just a way of getting more drives without moving my current build but I bet I can fit an 8 disk raid in my current chassis plus a system drive...

http://www.silverstonetek.com/products/ ... E&area=usa

[HFat]

a few friendly warnings...

for a collector like me, it gives so much piece of mind to know that ZFS will know and fix file errors on-the-fly and that I can always do "zpool status" and see a list of any files that possibly couldn't be fixed even with double parity

As discussed elsewhere on this forum, ZFS is poorly suited to archival. Your data can be safer without messing with filesystems and the like.
Also, double parity is sometimes overkill.

Harddrives will fail eventually, that's a given, but each of the three disks should - statistically - last just as long as one, so it should be the same on average ... you could argue that each of the three disks is only carrying 1/3 of the load of that member (so long as you use linear/JBOD in the middle layer), so if anything they should last longer than a single disk.

This is wrong, and very poor risk analysis. This is not a matter of opinion. Please do not make potentially dangerous misleading claims on a public forum.
There's nothing wrong with using multiple layers in principle but you've got to know what you're doing.

I recently had 3 drives failing simultaneously (two with bad sectors, and one just going into complete convulsions). It was pretty quickly fixed by replacing each of the physical devices and mirroring the old faulty device onto it. Re-add to the middle layer arrays, and do a scrub on ZFS.

scary

[HFat]

Just thinking again and I guess I could run a small Linux build with good ZFS support in a VM and get it to boot at startup after Windows has. Then share it using CIFS to the Windows machine.

Be careful when mixing VMs and filesystems such as ZFS. Google for horror stories...
Generally be careful with VMs. Being safe will cost you performance and you will likely not be safe by default. A real server is not that expensive if you don't over-spec it...

[domb]

I live in a small flat. I don't have the space to build another file server. Not a case of money...

[DanielSmedegaardBuus]

I meant is there anything available that runs natively in Windows that has functionality like ZFS's RAIDZ.

Oh, of course I'm on the zfs-fuse mailinglist, and this question pops up every once in awhile. Unfortunately, no, not ATM, and nothing I know of is in the works. People have discussed the possibility of porting ZFS to Windows, but it seems to rely heavily on POSIX stuff IIRC, and so would be extremely difficult to port.

I really can't be botehred with setting up a Linux file server. I don't even have the space for it. I run MediaPortal and everything is setup perfectly at the moment. Just need a decent file system that I can expand.

EDIT

Sorry didn't mean to be blunt there! I take on-board your advice and appreciate you writing it! Busy at work at the moment so not had time to read your post properly.

No worries, I did go off-topic there for awhile, forgetting that this was related to virtualization in the first place I'll get back on topic!

Just thinking again and I guess I could run a small Linux build with good ZFS support in a VM and get it to boot at startup after Windows has. Then share it using CIFS to the Windows machine.

That's exactly the setup I was doing with my benchmarks posted previously.

I'll have to get some spare drives and much around with it.

I would forget the chassis I suggested, it was just a way of getting more drives without moving my current build but I bet I can fit an 8 disk raid in my current chassis plus a system drive...

In your shoes, having not created an array yet, and doing virtualization, I'd not choose raw disks, but stick with virtualized ones. At least with VMware, it's possible to distribute the segments of a "split" virtual disk across different locations on the host. As a proof-of-concept I just created one using VMware, and edited its .vmdk descriptor, which read,

# Extent description
RW 4192256 SPARSE "testttt-s001.vmdk"
RW 4192256 SPARSE "testttt-s002.vmdk"
RW 4192256 SPARSE "testttt-s003.vmdk"
RW 4192256 SPARSE "testttt-s004.vmdk"
RW 4192256 SPARSE "testttt-s005.vmdk"
RW 4192256 SPARSE "testttt-s006.vmdk"
RW 4192256 SPARSE "testttt-s007.vmdk"
RW 4192256 SPARSE "testttt-s008.vmdk"
RW 4192256 SPARSE "testttt-s009.vmdk"
RW 4192256 SPARSE "testttt-s010.vmdk"
RW 4192256 SPARSE "testttt-s011.vmdk"
RW 4192256 SPARSE "testttt-s012.vmdk"
RW 4192256 SPARSE "testttt-s013.vmdk"
RW 4192256 SPARSE "testttt-s014.vmdk"
RW 4192256 SPARSE "testttt-s015.vmdk"
RW 4192256 SPARSE "testttt-s016.vmdk"
RW 4192256 SPARSE "testttt-s017.vmdk"
RW 4192256 SPARSE "testttt-s018.vmdk"
RW 4192256 SPARSE "testttt-s019.vmdk"
RW 4192256 SPARSE "testttt-s020.vmdk"
RW 40960 SPARSE "testttt-s021.vmdk"

I simply moved half of the files to /home/daniel/test/ and the other half to /home/daniel/test2/, booted up a windows guest, formatted the disk and started filling it with data. It worked fine. So that would make out your middle layer right there, on the host without the need to do anything on the guest. You can create 3TB virtual disks and spread the segments across sets of 1 and 2 TB drives. Virtual disks can even be expanded, so basically this is a great solution when running virtualized. This also means you can move your ZFS zpool easily between different guest OSes if you want.

If you're just doing fileserving, i.e. ZFS + CIFS, I wouldn't go with Linux as ZFS is using FUSE here, so it's much slower than when running natively. Either Solaris (which is payware AFAIK), or IllumOS, *BSD, or OpenIndiana (think I misnamed it OpenNevada previously?). They all have native ZFS support.

While VMware Workstation proved much faster than Virtualbox to me, it isn't free, there's the VMware Player, which is - and which can create VMs since version 6 or 7, forget exactly which. You can also create more advanced VMs using the 30-day trial version of Workstation, and use them afterwards in Player.

[DanielSmedegaardBuus]

a few friendly warnings...

for a collector like me, it gives so much piece of mind to know that ZFS will know and fix file errors on-the-fly and that I can always do "zpool status" and see a list of any files that possibly couldn't be fixed even with double parity

As discussed elsewhere on this forum, ZFS is poorly suited to archival. Your data can be safer without messing with filesystems and the like.
Also, double parity is sometimes overkill.

Hi HFat, thanks for the warnings. A couple of things:

Yes, double parity is overkill until you loose two members. But in that logic, any parity is overkill, until you have to use it But, it also is a performance killer for writes, so one should be very careful in chosing the right amount of parity.

Harddrives will fail eventually, that's a given, but each of the three disks should - statistically - last just as long as one, so it should be the same on average ... you could argue that each of the three disks is only carrying 1/3 of the load of that member (so long as you use linear/JBOD in the middle layer), so if anything they should last longer than a single disk.

This is wrong, and very poor risk analysis. This is not a matter of opinion. Please do not make potentially dangerous misleading claims on a public forum.There's nothing wrong with using multiple layers in principle but you've got to know what you're doing.

Well, seeing as how I explained my logic through deduction, maybe you could do the same? Point in fact, why is it wrong to deduce that three disks spending more time spinning idley would live longer than one drive spending more time transferring data? I explained my reasoning, you just state that, it's "not a matter of opinion". Please clarify. What you write there, is just an opinion

I recently had 3 drives failing simultaneously (two with bad sectors, and one just going into complete convulsions). It was pretty quickly fixed by replacing each of the physical devices and mirroring the old faulty device onto it. Re-add to the middle layer arrays, and do a scrub on ZFS.

scary

Very! But luckily, I was running ZFS RAIDZ2, so 0 data was lost. Also, the members that were failing were two 500GB drives and one 1TB drive, and interestingly none of them were "horizontally paired", and by that I mean, the respective "slice" they made up of each ZFS member was matched by fully working slices in all of the other 6 members, so I could actually have survived this with even RAIDZ1.

[DanielSmedegaardBuus]

As discussed elsewhere on this forum, ZFS is poorly suited to archival.

I'm sure you meant to write something else? A filesystem is poorly suited to archival? What else would it be suited for?

Be careful when mixing VMs and filesystems such as ZFS. Google for horror stories...
Generally be careful with VMs. Being safe will cost you performance and you will likely not be safe by default. A real server is not that expensive if you don't over-spec it...

I second that. It's always a double-risk when using virtual disks without writethrough. You effectively end up with double disk caches, and in particular, you have one that the guest cannot control. A cache flush in a guest which it thinks is committed to disk, but is not, can lead to very unfortunate effects in case of a system crash, even in the guest. As in lost data, or worse: silently corrupted data. This goes for any fs, but the more complex it is (like ZFS or btrfs with snapshotting), the more devastating the effects can be.

Make sure to disable host-side caching for your virtual disks if you value your data. This will slow down writes, but if you're fileserving media files and such for personal use, that really shouldn't mean much.

ZFS in particular has a huge number of horror stories where even systems using Solaris best practices (as in raw devices only, no partitions, no nothing), find all data irrevocably lost after someone tripped over the power cord. On bare-metal hardware, mind you.

I tested this a number of years back with ZFS-FUSE 0.5.something, and could easily replicate a complete loss of data. The issue has been resolved with later zfs versions, and in my own experience with ZFS-FUSE 0.6.something versions, I've gone through extensive crash-testing to ensure that I wouldn't lose all my data. This involves starting mirrordir writes to my zfs, then pulling out the power cord and reboot a dozen times. It involves pulling out SATA cables one-by-one until all hell breaks lose. I have yet to "succeed" in killing off my zpool in this way. And this is with an mdadm middle-layer.

So, when reading horror stories, always make sure to check for the version in question. OpenSolaris, for instance, before being abandoned by Oracle, never made it to version 11, which would be installed with a "good" zfs version.

[DanielSmedegaardBuus]

Oh, and never-ever enable dedup, even though on paper it is awesome. The performance hit is insane. Future versions may prove better, but ATM, it's performance death.

[DanielSmedegaardBuus]

Was chopping onions, and it struck me, if you're going to use virtual disks anyway, you might want to just use either VMware Server 1 or 2. They're both free, they both install as services, and you can set them up to start and stop specific VMs with the host as it starts up and shuts down. It will even delay host shutdown to give guests the time needed to shut down properly, too.

[HFat]

There is objective information available on hard drive failures. There is unfortunately not nearly as much as we'd need to make rational decisions but some things are not a matter of opinion or of abstract logic. I don't have the patience to go into details here and this is not the place anyway so please use your search engine of choice, search the forum or post on an appropriate forum or mailing-list.

ZFS is suited to dynamic data, not to archival. It's not only that ZFS is complex: it's brittle. And the way you set it up is more fragile than it needs to be. In any case automatic real-time mirroring and/or generation of parity is poorly suited to archiving in general, regardless of implementation. You do not want erroneous, spurious or malicious modifications to affect your mirrors and/or parity data. This can easily be achieved if your data is not supposed to be modified.
ZFS would be more suited to email stores, databases, transient data, files which are in the process of being edited and so on (assuming the performance was adequate).
For archival, a simple and robust file system is preferable. On top of that, you want to use software suited for archival. see:
http://en.wikipedia.org/wiki/Parchive
http://en.wikipedia.org/wiki/Comparison ... _archivers
http://en.wikipedia.org/wiki/Comparison ... n_software
Please Google or search the forum for more information.

Double parity is often overkill not so much because you're not likely to ever need it but because there's usually a better use for the drive capacity.

[HFat]

I don't have the space to build another file server.

In that case I would recommend a well-tested, robust file system which has well-tested recovery tools. Ext3 would be one such file system.
But if you already use NTFS, I'd stick with that. NTFS is not the safest file system in my experience but ZFS is known to be much worse. ZFS is more suited to well-endowed environment (space, budget, time and skills). Other file systems suffer from similar issues such as XFS or btrfs.

[DanielSmedegaardBuus]

There is objective information available on hard drive failures. There is unfortunately not nearly as much as we'd need to make rational decisions but some things are not a matter of opinion or of abstract logic. I don't have the patience to go into details here and this is not the place anyway so please use your search engine of choice, search the forum or post on an appropriate forum or mailing-list.

This is interesting. You enter this discussion, quoting my writing that, "you could argue that each of the three disks is only carrying 1/3 of the load of that member (so long as you use linear/JBOD in the middle layer), so if anything they should last longer than a single disk". You correct me, saying that "it's wrong", and that I should not just post opinions.

When I point out that my argument is based on clearly expressed deductions, and reverse your request onto yourself and ask that you do what you request of me and back up your "it's wrong" opinion by some clarification, you respond with "I don't have the patience to go into details here", and ask me to "use your search engine of choice". Do you really mean to come off this arrogant? What is your business here? Are you here to contribute or are you trolling? I have spent the last two years with ZFS in a number of different setups, I've worked with systems administration professionally in my job as a systems developer, I'm actively engaged in the ZFS-FUSE community, and I'm using it ever day. My point of writing here is to help out. What is your agenda?

ZFS is suited to dynamic data, not to archival.

Opinion (strange one, too, and strangely put). Should you follow your own request and choose to try and back that up somehow, you'll be in for quite some googling to find a crazy person to actually say that.

It's not only that ZFS is complex: it's brittle.

Opinion again. Zzzzz... Gonna back any of this up? How long have you used ZFS, did you say?

And the way you set it up is more fragile than it needs to be.

The way you set it up? It's the easiest of all professional file storage solutions to set up that I've ever encountered. With what did you have difficulties?

In any case automatic real-time mirroring and/or generation of parity is poorly suited to archiving in general, regardless of implementation. You do not want erroneous, spurious or malicious modifications to affect your mirrors and/or parity data. This can easily be achieved if your data is not supposed to be modified.

What the hell are you talking about? With every nonsensical, badly formulated, sentence you write out, it becomes ever more clear that you have zero experience with ZFS in particular, and very little understanding of how it - or previous generation parity-based storage solutions - work.

ZFS would be more suited to email stores, databases, transient data, files which are in the process of being edited and so on (assuming the performance was adequate).

LOL, this is exactly where ZFS is not very good. And databases don't contain "transient data". The whole point of a database is to store data. If you have transient data, you put it in a cache. Any sane database setup would involve using an extremely fast fs (especially with regards to seeking and small IO operations) on the production site, combined with a more secure secondary mirror where data integrity is more important than performance. ZFS is NOT a performer.

For archival, a simple and robust file system is preferable. On top of that, you want to use software suited for archival. see:
http://en.wikipedia.org/wiki/Parchive
http://en.wikipedia.org/wiki/Comparison ... _archivers
http://en.wikipedia.org/wiki/Comparison ... n_software
Please Google or search the forum for more information.

You're seriously linking to file archivers and file-level parity tools?

Double parity is often overkill not so much because you're not likely to ever need it but because there's usually a better use for the drive capacity.

Thank you for finishing off with yet another ridiculous opinion.

I expect that you're going to flame back at this, but I'm not going to answer. I'll continue to provide any help with regards to the original topic, in case there's need for any real advice from someone with experience and understanding.

Have a nice weekend, though! You might want to use it - if you're really interested - reading the ZFS administrators guide from Sun (now Oracle), the Google harddrive evaluation report from a couple years back, and visiting ZFS forums on *solaris sites or ZFS-FUSE.

[domb]

NTFS isn't going to allow me to build a RAID 5/6 type setup thats expandable in any way... hence the initial idea of using ZFS in a VM.

The other option is something like a Drobo, but the 8 disk models are about 1k.

I was just thinking if there are any NAS linux distros that do a simliar thing with a web interface.

HAHA didn't realise this would start an argument. Quite amusing.

[HFat]

Amusing indeed. ZFS is argument fodder if you didn't know. There are strongly opinionated people on both sides.

Surely there must be a way to do something like RAID5 in Windows. But if you want RAID in a VM, you could use mdraid. No need to use ZFS.
If at all possible, use mirroring instead... and not real-time mirroring like RAID1 but something that would get updated nightly (for example). That would be much safer if your resources are limited.

edit: I don't use web interfaces but this is Google's first result: http://serverfault.com/questions/3680/i ... admin-tool
That website is more appropriate than SPCR for such matters by the way.

[DanielSmedegaardBuus]

NTFS isn't going to allow me to build a RAID 5/6 type setup thats expandable in any way... hence the initial idea of using ZFS in a VM.

Also, no RAID setup will guard you against silent corruption, or corruption in the FS layer. They only guard stripe integrity, and when they fail, they can't tell you which files and/or metadata got corrupted They also need full-platter rebuilds when they cannot reliably state that they're uncompromised, whereas filesystem-level checksumming and parity protection (zfs/btrfs) need only scan used space and recover where checksums fail. IMO, nothing beats this scheme, and I'm never going back. I'll switch to btrfs once it's stable enough, though, but that might take awhile

I was just thinking if there are any NAS linux distros that do a simliar thing with a web interface.

As in a dedicated box? There's coraid, for one. We've used those for our office fileserver at work, they offer AoE (ATA over Ethernet), and are quite fast. You can shove a bunch of drives in them, and configure them quite extensively. With a terminal interface, though. Not sure if they're compatible with Windows, either... But they should be cheap second-hand and are production quality. And if there are AoE drivers for Windows, it'd just be another physical hard to partition and format with NTFS as you please.

If I were you, I'd still go for ZFS with a Solaris or BSD based distro as guest, and use segmented virtual disks distributed across your drives. It's free save for the drives

PS: Depending on your Windows, there is software RAID5 either available (server versions and IIRC W7 Ultimate), or attainable through a couple of hacks (XP), but I'm pretty sure you won't be able to assemble 4*1 and 4*2 TB drives into one array, though.

[washu]

ZFS is suited to dynamic data, not to archival.

Opinion (strange one, too, and strangely put). Should you follow your own request and choose to try and back that up somehow, you'll be in for quite some googling to find a crazy person to actually say that.

ZFS is a filesystem and like most fileystems it will do exactly what you tell it to. If you accidentally delete something, you get a virus, your OS screws up, hardware failure etc, ZFS will not save you. ZFS will write whatever you tell it to, good or bad. It will be perfectly happy to write corrupt data because something else screwed up. It will even checksum and tell you your corrupt data is good when you scrub it.

True archiving with some kind of difficult to modify external verification prevents all that. ZFS could be used to store an archive separate from your main data (weather stored on ZFS or not) but it's complexity, poor OS support and lack of recovery tools (see below) make it a poor choice over simpler filesystems.

It's not only that ZFS is complex: it's brittle.

Opinion again. Zzzzz... Gonna back any of this up? How long have you used ZFS, did you say?

HFat is 100% right. ZFS is very new and does much more than other filesystems. There are lots of opportunities for something to go wrong. Have you used ZFS on a memory constrained system? Such as the likely case in a VM? If ZFS wants more memory than it can get it just crashes and takes the OS with it.

There are also no recovery tools for ZFS (fsck, chkdsk, etc). ZFS proponents claim it's unnecessary, but there are lots of horror stories on google to show that not to be true. Every filesystem can get corrupted, even ZFS.

[HFat]

Hi washu! Thanks for being less opaque than me. I was wondering how long it would take for you to show up.
FYI, I have recently updated my answer to one of your questions in the thread where we last had a ZFS argument. Don't forget to bring peanuts for the gallery in case you want to reply!

[domb]

You could look it forever and realise that nothing is really 100% fail safe... but I'm not going that far. Just something with some layer of drive failure protection. RAID 5, Raid Z... doesn't matter. Just something that will let me lose a drive but keep the data, then rebuild. I also don't want to sit at a terminal looking after it every 5 minutes.

But the main thing I need on top of the parity data is the ability to expand the volume with new drives without losing data!

As I said before I have no room for a dedicated file server... I meant a distro like freenas... Perhaps it will run well in a VM... I shall see if I get time the weekend to play with it and let you know the results!

http://freenas.org/

ALSO

http://www.openfiler.com/
http://www.nexenta.org/

[HFat]

The expanding part can be done with LVM. mdraid + LVM is a common and therefore well-tested configuration. It's not without pitfalls (everything flexible has issues) but at least they're well-understood.
You never need to look at a terminal or a web interface every 5 minutes. mdadm can be set up to email you if there is a problem. I imagine ZFS has a similar feature.
Native solutions would be preferable of course. I'm not knowledgable about Windows. But lots of people on serverfault.com and similar sites are...