That's true. Do you think that the README for the aoe driver is a good place to mention that, or is there a better place? On Oct 19, 2013, at 5:29 PM, <devzero@xxxxxx> wrote: > trying to use ordinary filesystems on shared blockdevices really seems to be a very common caveat, i have often seeing people trying that and wondering why it doesn`t work. > > regards > roland > > >> Gesendet: Samstag, 19. Oktober 2013 um 18:22 Uhr >> Von: "Ed Cashin" <ecashin@xxxxxxxxxx> >> An: "12navidb2@xxxxxxxxx" <12navidb2@xxxxxxxxx> >> Cc: aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx >> Betreff: Re: AOE sync problem >> >> Hi. If you're trying to use a regular file system, you should be aware of the way the Linux kernel aggressively caches written data in RAM. It usually won't flush out dirty data (from writes) until it feels like it, letting the writes pile up in memory. >> >> That's good for the usual case, but it's confusing to people who are used to a single host to single disk relationship. Using an ethernet network instead of an ATA cable provides more flexibility and exposes the reality behind the illusion that transparent caching provides. >> >> It is only safe to access the AoE target from other hosts if you are aware of the implications of this caching. Specifically, only cluster file systems like Red Hat's GFS are designed for sharing block storage to provide a single shared file system. Other file systems like ext3 are "traditional" and rely on the assumption that they have exclusive access to the block device. If other hosts violate that assumption (e.g., by changing the filesystem metadata on the shared block device), then data corruption is likely to occur. >> >> Example: >> >> Host "A" reads filesystem metadata from the block device, caching it in its RAM. >> Host "B" does the same. >> Host "B" removes a file, changing the metadata in its RAM. >> Host "B" flushes the data from RAM to the shared block device. >> Host "A", unaware of host "B", attempts to read the file that B just deleted. >> Host "A" updates the atime in its cached metadata and flushes the updated metadata to the shared block device. >> >> You can see that the two hosts will create a jumbled mess of the file system if they don't have a way to coordinate their actions. Cluster file systems provide those coordination mechanisms. >> >> You can force the Linux kernel to flush writes out to the backing storage by using "sync" and other mechanisms, but performance will be affected, since a disk, whether local or remote, cannot be as fast as RAM. >> >> Jumbos are not required for many configurations but do increase throughput significantly. >> >> If I had to guess, the MAC address destinations are those of hosts on your network, virtual or otherwise, that you didn't know about or have forgotten about. The only address that is used without it needing to be on the network is the broadcast address, ff:ff:ff:ff:ff:ff. Of course, you didn't say what software you're using, so if you're using a program I don't know about, then all bets are off. :) >> >> On Oct 19, 2013, at 9:25 AM, 12navidb2@xxxxxxxxx wrote: >> >>> Hi, we set up AOE on our systems, then we mounted the exported partitions but unfortunately the changes on mounted partitions are not reflected on the main exported partition. Also other systems can not see the changes. Everything seems alright; except synchronization. >>> >>> Also i red in a blog post that we must use jumbo frames but i do not use jumbo frames. can this be the source of the problem?? >>> I checked the network traffic using "tcpdump ether proto 0x88a2" in the result of this command i saw bunch of frames with destionation addresses like bc:5f:f4:6b:af:58 that does not exist in my local network!!! A part of output looks like the following: >>> >>> 09:23:42.000905 30:85:a9:92:ba:ff (oui Unknown) > 1c:6f:65:7f:42:14 (oui Unknown), ethertype Unknown (0x88a2), length 1060: >>> 0x0000: 1000 0000 0000 5a27 a1ed 4000 0224 0000 ......Z'..@..$.. >>> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x00a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x00e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x00f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> 0x0100: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >>> >>> >>> As you can see most of the transferred frame is just zero!!! Any suggestions?? >> >> >> -- >> Ed Cashin >> ecashin@xxxxxxxxxx >> >> >> >> ------------------------------------------------------------------------------ >> October Webinars: Code for Performance >> Free Intel webinars can help you accelerate application performance. >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from >> the latest Intel processors and coprocessors. See abstracts and register > >> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk >> _______________________________________________ >> Aoetools-discuss mailing list >> Aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx >> https://lists.sourceforge.net/lists/listinfo/aoetools-discuss >> -- Ed Cashin ecashin@xxxxxxxxxx ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk _______________________________________________ Aoetools-discuss mailing list Aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
