Has anyone here had any experiences with ATA over Ethernet? It seems a little bleeding edge, right now. From what I've gathered from a cursory internet search, there's currently only one commercial vendor and only one open source server project (written by an employee of that commercial vendor, Coraid).

Currently Linux kernels already support AoE, but I'm not sure about Windows. But IMHO the ultimate would be if AoE support could be crammed into a mobo BIOS--then you could install Windows or Linux or whatever onto a fast networked RAID just as if it were a local drive!

With the speed of gigabit ethernet, ATA over Ethernet could even be much faster than a quiet local drive.

1000 Mbps / 8 = 125 MBps

SATA = 1.5Gbps
SATAII = 3Gbps

It doesn't seem all that impressive to me, at least not an improvement. On the flip side though, you could take the "relocate-external-hard-drive" to a new extreme.

I don't think the point is speed - it's ease of expanding SANs (storage-area-networks) and the lower overhead of data transfer over network.. what I mean is SMB is very slow, as is NFS.. but FTP has much less overhead and therefore much better throughput.. although doesn't allow proper random access to files.
Hopefully ATAoE will have low overhead and good random access features. dunno.

btw.. sthayashi your table threw me a bit.. the best comparison would be:
SATA = 1.5Gbps
SATAII = 3Gbps
GbE = 1Gbps

However, together with overheads an SMB over GbE link would only give a useful bandwidth of 400Mbps (this is a wild guess based on the delivered bandwidth I get from a SAMBA server on a 100Mbps ethernet link - I get about 5MBps).
400Mbps is 50MBps - which is about the max sustained read speed off todays 7200rpm disks. So, today at least, GbE is just about enough for ATAoE, and any more wouldn't be necessary.

Right now you're not going to get anywhere near full SATA 1.5Gbps speeds with ANY single drive, much less a single QUIET drive.

As for the limits of ATA over Ethernet, it is a high efficiency low level protocol which doesn't even use TCP/IP (it uses MAC addresses directly). It is extremely efficient with almost no latency.

However, even if it did use TCP/IP, the overhead is not necessarily that much. I use NFS on my 100mbit LAN, and it easily acheives 10+ megabytes per second transfer speeds. Before I switched to all Linux, I used SAMBA, which never acheived even half of that (similar to DonP's performance results).

The main disadvantage of TCP/IP apparently is the CPU cycles required to do TCP/IP stack processing, rather than the actual bandwidth cost.

BTW, NFS is faster than FTP. That shouldn't be surprising since NFS is about as low level as you can get with TCP/IP.



ANYWAYS...


After thinking about it, what I'd REALLY like to see is a combo gigabit ethernet NIC with AoE->SATA converter (or AoE->PATA converter) and internal RAID controller (a small embedded Linux machine). This card sticks in a PCI slot, with an ethernet connector on the backplane. On the inside, it has a SATA or PATA hard drive connector, which connects up just like any other hard drive. As far as the motherboard is concerned, it's just a really really fast hard drive. The small embedded Linux machine accesses AoE drives and performs software RAID to turn them into a single big fast virtual drive. It also has a small SSH server so the user can administer the Linux subsystem (like setting up the shares).

Erm, solid state storage like, say, Gigabyte iRAM?

ATA over Ethernet sounds like an interesting idea. Sort of like a poor man's version of iSCSI?

I don't know the details of it, but NAS (Network Attached Storage) is the latest enterprise storage solution used by many large companies. All the big vendors like IBM, EMC, Hatachi Data Systems, and Network Appliance have these solutions.

My company is moving from SAN to NAS, primarily to save money and improve reliability. The storage people say it is fast, but I don't have details on that. These systems use Fibre Channel controllers at the high end and Gigabit ethernet at the low end.

Hmm, most Flash drives aren't "super fast" in terms of sustained bandwidth, but in terms of latency (microseconds insteead of ~10 miliseconds).

@ Henk Poley

Sorry, I didn't catch why you're talking about Flash based drives. No one in this thread mentionned them...

Sorry for dragging this off topic - it's not relevant to the subject of ATA over ethernet at all, and there are several threads on this, so do a search on Gigabyte iRAM, which is a PCI card form factor, DDR SDRAM populated, SATA interface, semi affordable solid state hard disk with battery backup.

Hm. I think AoE sounds interested, but really only from a technical point of view - if it works as well as you say it does, that's quite a feat and very neat/cool. And I'm sure there are cases where it's useful. I don't see using it for silencing though, that'd just be way too involved for me.
You'd need to built a seperate enclosure, for instance. And unless you add power over ethernet, another cutting edge technology, you'd need a seperate power supply, too, although hardly a very powerful one. And since it seems like HDDs get more silent as well as larger over time, it doesn't seem to be enough of an issue - for me, anyway.

That sentence doesn't make much sense, except for the saving money part

AFAIK, SAN stands for 'storage area network' and NAS is 'network attached storage' as you said. The only difference is that in a 'SAN' you have dedicted network hardware (out of band) and in 'NAS' you use an existing network (in band) to route the storage traffic atleast part of the way.

As it happens when storage solutions providers first coined the term SAN, they used a lot of proprietary technologies with almost no interoperation between vendors. Later as more standardisation started to happen (FiberChannel, iSCSI), people began using the term NAS for simpler hardware setups often using devices from multiple vendors.

Still in it's basic meaning the simplest possible NAS is one of those harddrive enclosures with a builtin fileserver and an ethernet port. It could be argued that the same drive becomes a SAN if you buy an extra network card for your machine and plug the drive enclosure to that, in effect creating a dedicated network for storage traffic. So in a basic sense these terms are not counterpoles but different configuration options for networked storage hardware.

Note however, that device vendors are likely to put their own spin on the meaning of the terms to make their products look better

you would be able to use an ATA cd/dvd drive with this ATAoE, yes?
whilst everyone is talking about using ATA hard drive and relocate-external-hard-drive, i would rather put the whole machine in another room and keep the dvd burner on the desk attached by ethernet (with cordless keyb+mouse, long dvi cable, wake on ps/2 - would seldom need to see nor hear the mainframe)

I don't claim to be an expert, but SAN is different than NAS. As you mentioned, SAN uses proprietary interfaces and NAS uses TCP/IP. But many times the NAS networks are not using existing LAN's (at least in serious high performance installations), they are using private IP networks using Gigabit, Fibre Channel, etc.

I would agree that SAN and NAS are not counterpoles, but the point is that many major companies are using NAS for high end applications where performance is important. So maybe ATA over Ethernet (or at least SATA) is not as farfetched as it sounds. Dell is selling a SATA based NAS solution.

It's not a requirement for SAN to use proprietary technologies. Most certainly it has been often implemented that way, but the term just means 'storage area network' i.e. a network dedicated to storage traffic. What network technology you happen to use is really secondary. Everything from Fiberchannel to ATM to Ethernet could be used and the used encapsulation isn't set in stone either. You could use your own frame protocol, TCP/IP, just IP, or even the hardware layer directly.

You mentioned that your company has been transitioning from SAN to NAS to improve reliability. For that to hold true one would have to know what your companys current SAN looks like. The techniques used to achieve reliability are pretty much the same for both SAN and NAS. So what I'm really thinking is that you are transitioning to a new generation of devices for improved flexibility and that might allow for greater reliability because it allows some new configurations. Ofcourse there is the added benefit of less vendor lockin.


Actually Fibre Channel doesn't use IP (IIRC). This is the point I was trying to make. When you configure your devices like this, what you really get is a SAN and not a NAS anymore

In fact even Coraid uses the term SAN for their AoE solution.

I'm assuming they use the term SAN here because their solution doesn't make use of TCP/IP and you really want that dedicated network for storage anyway, atleast if you are planning to write any significant amounts of data during workinghours.


This is what really threw me about your original comment. Are you trying to imply that NAS is something too expensive and 'enterprisy' for a normal consumer ? Like I said, the simplest NAS is a single harddrive in a networked enclosure. Do you think this should not be called NAS because the enterprise market doesn't have a use for such a simplistic NAS implementation ?

As far as ATA over Ethernet goes, I don't think this will fly in the enterprise market. You can just as well use iSCSI to connect to your drive rack even if there are actually SATA drives in the rack. In fact such devices exist already. We are also starting to see the emergence of SAS 'Serially Attached SCSI', which is SCSI using the same cabling as SATA. The intresting side effect is that apparently some upcoming SAS raid controllers can drive SATA drives as well, so you could start with a modest setup using cheaper SATA drives and upgrade to SAS as needed.

I see ATA over Ethernet being mostly interesting in the consumer space. Normal consumers wont need all the advanced SCSI features and aren't intrested in managing their devices. In such an environment using the simplest possible transport to connect to your remote storage is an acceptable compromise to reduce cost. I wouldn't be surpriced if this technology gets picked up by home entertainment system builders, because while drive sizes have been going up, you really don't want to put more than one hdd in your htpc device. So if you really want a _lot_ of storage space to handle your entire movie and music collection you want to place the storage array away from your livingroom.

Yes, ATA over Ethernet is basically "poor man's iSCSI". However, since it uses MAC addresses directly instead of TCP/IP, you can get better performance than doing the same with iSCSI.

I don't really see it as useful for HTPC, though, since bandwidth requirements for HTPC are relatively low. For HTPC, even SMB file sharing over wireless LAN is fast enough--and these files will be readily accessable by other computers and media player appliances.

Except that by not using TCP/IP you loose the ability to use normal TCP/IP tools and hardware to do bandwidth & priority management. So far I've understood this to be a major point for iSCSI, because otherwise you could just use FiberChannel. Also I don't think that the performance issue is the deciding factor. We must not forget, that even if we are not using TCP/IP we must have our own frame protocol to ensure that data does not get lost or corrupted.

I'll have to admit that TCP/IP has not been deviced for high bandwidth traffic, but I doubt the performance difference is more than 5%. Actually I think the technologies have their sweetspots on different areas. One would choose one or the other depending on planned deployment environment. The fact that Coraid seems to be the only vendor for AoE is a big deal on the enterprise market though. Broad vendor supports can be very important for companies these days.


True, but with SMB you then need to have a more intelligent server serving out the files. With AoE you could let your HTPC be the server to other machines and add very simple (cheap) disk arrays remotely to it as your media collection grows.

We should also remember that SMB as a protocol is really old (developed by IBM in the 60ties ) and not very good. It has an artificial 2GB filesize limit and it's pretty inefficient. I think the filesize limit can be overcome, but as an HTPC vendor I wouldn't neccesarily want to use SMB as the only way for extending my product.

Just that we won't be sidetracked by the definition of SAN and NAS anymore, I looked up the thing from Wikipedia: http://en.wikipedia.org/wiki/Storage_area_network

Basically what it says is: A SAN provides block-level access to the storage medium while a NAS device is connected to with a higher level protocol like a NFS or SMB/CIFS.

That makes AoE a SAN implementation, since it exposes the block level protocol over the network. Same applies to iSCSI and Fibre Channel where usually SCSI protocol is used, but ATM, IP or really any transport protocol is possible.

Check-out www.hammer-storage.com and/or www.zetera.com

They gang a bunch of SATA drives inside a 1-U storage box with an Ethernet port to the outside. I know they don't implement RAID5 between multiple storage boxen; they let the network swtich's management features handle mirroring instead. (I need to read some more to see if they've got RAID 5 inside the box.)

They call their technology SoIP (Storage over IP), but the functionality is essentially the same as as ATAoE.

I don't see anything in either of those two which is remotely like what I was hoping for--bootable networked storage which the client computer thinks is a local (ATA) drive.

In other words, can the networked drive be "C:"? Can the client computer be diskless but think that it has a local drive connected to the primary IDE channel?

If so, then that has some interesting potential for silent computing.

If not, then it's just another way to store files on a file server. Big deal.

No one's mentioned AoE or iSCSI since 2005-2006? Thought I'd bring this one up from the dead. Unfortunately for me, I'm restricted to a Windows storage server, as I plan on running Mediaportal TVE3 server on it as well.

AoE
From what I've seen, there's no Windows AoE targets, so I can't run AoE. A free open source AoE initiator can be found here: WinAoE

iSCSI
Free iSCSI target for Windows 2003 found here: Nimbus MySAN iSCSI target
Free iSCSI initiator for Windows here: RocketDivision StarPort iSCSI initator

Actually RocketDivision has free versions of iSCSI target and initiator software. On Linux there's a whole slew of options.

As I'm running Mediaportal, I was thinking of running Mediaportal "Extenders" which would be bare bones HTPC boxes that merely stream videos, tv, music, etc. If I can run them driveless using iSCSI it may be well worth giving it a shot. Unfortunately, none of the iSCSI initiators for Windows support booting from iSCSI. Whereas WinAoE does support boot off AoE, there are no AoE targets for windows.

Sorry for necroing this thread once again, but i recently found out about this myself and i'm very interested in learning more about this.

badkarma: Maybe you can use PxE instead of AoE/iSCSI?

It basically does the same as AoE but it's integrated into the NIC and it supports any interface for Hd's you can use with your computer. I'm not exactly sure how many motherboards / network cards support this but from what i have read it is supposed to be a standard. My "Msi p965 Neo" supports it. It uses the "Realtek 8110SC" NIC controller so i'm guessing that any motherboard with that controller supports PxE/Booting over ethernet. I have not yet been able to confirm that this technology works as i dont have another computer i could boot from though.

Read more about what PxE is in general here: http://en.wikipedia.org/wiki/Preboot_Ex ... nvironment

Read more about booting windows over ethernet using PxE here: http://silent.gumph.org/content/4/7/071 ... s-pxe.html

For what it's worth, I bit the bullet and figured out how to do diskless netbooting. I wrote a how-to on doing it with Debian. It does involve a noticeable performance hit, with plain old 100mbit ethernet--about the same speed as an average laptop drive (around 10megs per second). However, it's cheap, especially if you have numerous computers. Only the server needs to have a hard drive, and that hard drive can be a cheaper 3.5" drive.

Konata Izumi,

Didn't realize PXE can boot Windows directly. Sounds like PXE has to load the entire windows image into RAM. What AoE or iSCSI does is provide a driver, that allows Windows to see an AoE/iSCSI disk as a local disk. I use PXE to chainload gPXE which then boots off AoE/iSCSI. On a Gigabit connection I get about 50MBps which is plenty of speed for a HTPC device, actually it's about what my 7200rpm local server drives get. I've been using my laptop to test and it's definitely faster than the local laptop drive. I was able to get it working without issues.

UPDATED.

badkarma: I see. From what you are telling me it seems like AoE/iSCSI is the better solution. Is it possible to use AoE with sata hd's?

You can use AoE with any HD device on the target. It's agnostic that way. You just point the AoE Target application to an image file on your filesystem.

I looked at the pxe method, and it's definitely loading Windows into RAM. They talk about Win98 images being less than 90MB to fit into a pxe client with 128MB of RAM. The other thing is pxe boot, all changes would be lost once the device is powered down unless changes are saved manually. AoE/iSCSI will write to the "local AoE/iSCSI disk".

Right now, the only problem is there is no AoE target for Windows and no free iSCSI bootable XP initiator. So in order to boot Windows you have to use a linux target. However, I'm talking to a guy that's working on writing an AoE target for windows. Will hopefully be beta testing it.

What do you mean by a linux target? Does that mean you're using it to boot up a Linux operating system? If so, then it really isn't necessary to have that. I'm netbooting Debian GNU/Linux using plain old PXE/tftp/nfs. The reason I'm not getting as good performance as you is that I'm using cheap old 100mbit ethernet instead of gigabit.

Still...that makes me wonder about virtualization. I've never used VMWare or anything like that, but perhaps you could netboot a very basic Linux OS to serve as a host OS for a virtual machine running Windows? As far as Windows is concerned, it's running off a local ntfs hard drive.

AoE/iSCSI uses the term initiator/target for client/server. So when I say AoE target, it means the server serving the Windows OS AoE disk must be linux as there is no Windows AoE target (yet). Linux has good initiator/target support so that's not the issue. And as you have mentioned PXE/tftp/nfs can work for linux as well.

I've actually tried VMWare server with a linux guest OS as a AoE target with Windows bootable initiator to get around my OS limitations. The performance over gigabit is very poor compared to native linux target. I haven't had much luck trying to optimize I/O performance within VMWare guest OS's. However, Vmware Server beta 2 seems to have better performance, but it no longer allows native disk access.

Could i have a discless client/initiator boot windows of a linux server/target?

Absolutely. Works well too in my testing. My problem is that I require native Windows access to DVB cards, so running linux native target is a roadblock. Obviously, in order to get the higher disk access speeds you will need gigabit with jumbo frames enabled. My tests were without jumbo frames (switch doesn't support it) when I was getting 50MBps.

Windows AoE Initiator
http://www.winaoe.org

There are a few linux AoE targets, I just used vbladed, it's generally packaged in the distros I've used ie. OpenSUSE, Ubuntu, etc

HOW-TO Boot Windows from a SAN (gPXE)
http://www.etherboot.org/wiki/sanboot/winxp

The chainloading of gPXE via PXE is missing from that HOW-TO. Basically you setup your DHCP server to tftp and load a gPXE ROM from your TFTP server. Then have another section in your DHCP server to look for the gPXE vendor-option and tell gPXE the AoE info so that when the gPXE ROM does a DHCP discover it will get AoE info and not the tftp command. Even better is flashing your NIC's ROM to have gPXE natively. =D Been messing around with Mobo bios's to see if I can load gPXE into my bios.

badkarma, how did you go flashing gPXE into your NIC or motherboard BIOS? I'm looking for a recommendation for PCI Express gigabit NICs with jumbo frame support and programmable rom for PXE. Any suggestions?