Hi Tracy, very thoughtful posting, I've arrived at similar conclusions and I am in a similar way from an early stage emotionally attached to a protocol of fine craftsmanship that has potential per se. In the beginning I've advocated the AoE protocol to be converted into an RFC, that would have helped in the first place and would have been an incentive for kernel maintainers to include it earlier and spread the word. Due but not only attributed to this fact AoE initiator support in the kernel materialized quite late. IMHO storage stuff belongs into the kernel, period. Besides, what benefit does an initiator have without a target implementation? The two user-space target implementations are dead for years and lagging behind the AoE specifications due to obvious reasons - conflict of interests. Right now we have a proprietary protocol and a single vendor that supports it with no incentive to have any alternative target implementations out there. Besides, the very important hypervisor vendors are not interested in anything beyond iSCSI and FCoE, so why should they bother to natively support AoE? Hence I consider it mission impossible to convince clients to bet their mission-critical storage future on an arguably fine nevertheless exotic single-vendor product and protocol only based on the price argument. I am happy that Coraid is doing fine as a niche player though and wish them all the best for the future. If you look at a success story how this could have worked out have a look at DRBD and Linbit. Regards, Gernot A m Freitag, den 19.10.2012, 11:47 -0700 schrieb Tracy Reed: > My primary pain points in decreasing order of severity: > > 1. For years I've been battling alignment issues in virtual machines with no > success. I can't take the performance hit or invest the time in twiddling > alignment anymore. TCP overhead is nothing compared to extra writes due to > disk alignment. iSCSI straight out of the box doesn't have this issue. > > 2. Zero distribution (CentOS/RedHat) integration. Having to maintain my own > init scripts and making the system unload the aoe module before the bonding > module on shutdown, start/stop the vblades, and various other bits of > trickery has become a real hassle. > > 3. Despite claims that AoE use is accelerating I've never run into another > person who uses it or even really knows what it is. I've never seen/heard > about it in any trade magazine aside from a Linux Journal article years ago. > Nobody stops in the AoE IRC channel except for disappointed "Age of Empires" > players. Traffic on this list is nearly dead. The blog you set up is quiet. > I've had an AoE google alert setup for years. There is nearly no chatter > about it. I have had an RSS search feed for AoE on ServerFault for quite > some time. It very rarely comes up. Currently, it hasn't been mentioned > since July: > > http://serverfault.com/questions/412957/how-to-access-aoe-block-device-from-redhat-hypervisor > > And look at the comments. Ouch. > > There is a lot of support and training material for iSCSI and it is part of > the RHCE exam. I have had to do all of my own in-house training for AoE and > it scares customers that it is such a niche technology with so few people on > earth knowing how to use it, despite its on-the-wire simplicity. It actually > hurts my business for people to know I use AoE. At one point I thought I > could sell it as a competitive advantage. Not only is it an operational > hassle but it has turned into a competitive disadvantage. > > 4. While I have always advocated AoE on the basis of its simplicity, the actual > number of steps needed to get iSCSI working on RedHat/CentOS are far fewer > than AoE which in practice makes AoE not simple. I have now > scripted/cookbooked the steps to getting iSCSI up and running and my junior > admin can do it. Mounting an iSCSI volume is a minor task on the RHCE exam > taken by relatively newbie Linux admins. The issues involved in getting AoE > working reliably and especially the troubleshooting are beyond many > administrators. I have puppet scripts to distribute kernel modules (have to > distribute it and compile it per kernel if you aren't running the exact same > version everywhere, and deal with a recompile when you yum update the > kernel), manage overly complicated init scripts, tweak sysctls, manage > vblade configs, etc. Using the builtin distro iSCSI none of this is > necessary. > > I could perhaps resolve all of this myself by getting involved in Fedora and > contributing patches to make AoE work as smoothly as iSCSI in RHEL. But so far > I've been completely technically incapable of solving the alignment issue and > as for the rest: Why should I invest so much time and effort duplicating the > excellent work of the iSCSI people? > > Anne Hathaway of Coraid actually contacted me last year about working for > Coraid. I was very tempted but simply unable to move away from San Diego. And > the first thing I would have wanted to do would be to make AoE work well in > RHEL as target and initiator which I really doubt Coraid would be willing to > pay me to do... > > I have often had the feeling that AoE's lack of distro integration has been > because Coraid would prefer we all spend tens of thousands on Coraid hardware > and if the free version worked well enough there would be less incentive for > people to do so. I understand that every business needs to make money. But I'm > not sure I can get behind that as a business model. Last I looked (it's been a > while) a Coraid box was all Supermicro hardware which I could assemble myself > from Newegg. I seem to recall it was Plan 9 for the OS, a strange choice. I > suspect it's something to do with licensing, which makes me somewhat > uncomfortable. Again, they need to make money but I'm not interested in helping > them to work against my own interest. > > I've got a business to run too, can't afford to purchase Coraid, and just > wanted a decent ethernet SAN which, given today's level of commodity > technology, is quite possible for a very reasonable price. I'm afraid I've hurt > myself spending so much time on making AoE work. I wish AoE well and maybe some > day I'll come back to it but for six years it has cost me way too much time > (and therefore money) and now I'm a bit embarrassed by the whole thing. > > On Fri, Oct 19, 2012 at 06:19:13AM PDT, Ed Cashin spake thusly: > > AoE use is accelerating rapidly, and iSCSI persists because even though it's not the best fit for same-LAN data storage, folks can still get it to work if they are willing and able to deal with the complexity, sacrifice the performance, and don't need the kind of scaling that virtualization and cloud deployments require. > > > > Although a lot of the expansion of AoE use lately has been by users of the Coraid HBA, there are still a lot of first-time AoE users using the coraid.com-distributed Linux initiator and also the kernel.org-distributed initiator. Recently there have been a lot of patches from Coraid to the Linux Kernel Mailing List for bringing the kernel.org-distributed driver up to date. For example, > > > > http://thread.gmane.org/gmane.linux.kernel/1377362 > > > > We also fixed a regression in the Linux kernel's network layer that affected AoE performance: > > > > http://thread.gmane.org/gmane.linux.network/243626 > > > > I don't have much time to participate on the aoetools-discuss mailing list right now, partly so that I can keep generating those patches, but I think pessimism is especially inappropriate today, when the use of AoE is accelerating along with the development of its associated technologies, including open source technologies. > > > > -- > > Ed Cashin > > ecashin@xxxxxxxxxx > > > > > ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct _______________________________________________ Aoetools-discuss mailing list Aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
