Hi Brent, Can you put "option debug on" in afr subvolume and try the du/rsync operations and mail the log? We are not able to reproduce the problem here, nfs is working fine over afr. Thanks Krishna On 8/4/07, Krishna Srinivas <krishna@xxxxxxxxxxxxx> wrote: > rsync was failing for me without no_root_squash, so thought that > might have been the culprit. > > If i put no_root_squash, nfs over afr works fine for me. > > Yes you are right, for some reason readdir() is not functioning > properly I think because of which paths are getting corrupted. > > will get back to you. > > Thanks > Krishna > > On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote: > > All of my tests were done with no_root_squash already, and all tests were > > done as root. > > > > Without AFR, gluster and NFS reexports work fine with du and rsync. > > > > With AFR, gluster by itself is fine, but du and rsync from an NFS client > > do not work properly. rsync gives lots of I/O errors and occasional "file > > has vanished" messages for paths where the last element is junk. du gives > > incorrect sizes (smaller than it should) and occassionally gives "no such > > file or directory", also for paths where the last element is junk. See > > output below for examples from both of this junk. Perhaps if you could > > figure out how those paths are getting corrupted, the whole problem will > > be resolved... > > > > Thanks, > > > > Brent > > > > On Sat, 4 Aug 2007, Krishna Srinivas wrote: > > > > > Hi Brent, > > > > > > Can you add no_root_squash to exports file and reexport and mount > > > using nfs and try to rsync as root and see if it works? > > > > > > like: "/mnt/gluster *(rw,no_root_squash,sync,fsid=3)" > > > > > > Thanks > > > Krishna > > > > > > On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote: > > >> Woops, scratch that. I accidentally tested the 2nd GlusterFS directory, > > >> not the final NFS mount. Even with the GlusterFS reexport of the original > > >> GlusterFS, the issue is still present. > > >> > > >> Thanks and sorry for the confusion, > > >> > > >> Brent > > >> > > >> On Fri, 3 Aug 2007, Brent A Nelson wrote: > > >> > > >>> I do have a workaround which can hide this bug, thanks to the wonderful > > >>> flexibility of GlusterFS and the fact that it in itself is POSIX. If I mount > > >>> the GlusterFS as usual, but then use another glusterfs/glusterfsd pair to > > >>> export and mount it and NFS reexport THAT, the problem does not appear. > > >>> > > >>> Presumably, server-side AFR instead of client-side would also bypass the > > >>> issue (not tested)... > > >>> > > >>> Thanks, > > >>> > > >>> Brent > > >>> > > >>> On Fri, 3 Aug 2007, Brent A Nelson wrote: > > >>> > > >>>> I turned off self-heal on all the AFR volumes, remounted and reexported (I > > >>>> didn't delete the data; let me know if that is needed). > > >>>> > > >>>> du -sk /tmp/blah/* (via NFS) > > >>>> du: cannot access `/tmp/blah/usr0/include/c++/4.1.2/\a': No such file or > > >>>> directory > > >>>> 171832 /tmp/blah/usr0 > > >>>> 109476 /tmp/blah/usr0-copy > > >>>> du: cannot access `/tmp/blah/usr1/include/sys/\337O\004': No such file or > > >>>> directory > > >>>> du: cannot access > > >>>> `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/\v': No such > > >>>> file or directory > > >>>> du: cannot access > > >>>> `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/&\324\004': No > > >>>> such file or directory > > >>>> du: cannot access > > >>>> `/tmp/blah/usr1/src/linux-headers-2.6.20-16/drivers/\006': No such file or > > >>>> directory > > >>>> 117472 /tmp/blah/usr1 > > >>>> 58392 /tmp/blah/usr1-copy > > >>>> > > >>>> It appears that self-heal isn't the culprit. > > >>>> > > >>>> Thanks, > > >>>> > > >>>> Brent > > >>>> > > >>>> On Fri, 3 Aug 2007, Krishna Srinivas wrote: > > >>>> > > >>>>> Hi Brent, > > >>>>> > > >>>>> Can you turn self-heal off (option self-heal off) and see how it > > >>>>> behaves? > > >>>>> > > >>>>> Thanks > > >>>>> Krishna > > >>>>> > > >>>>> On 8/3/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote: > > >>>>>> A hopefully relevant strace snippet: > > >>>>>> > > >>>>>> open("share/perl/5.8.8/unicore/lib/jt", > > >>>>>> O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 > > >>>>>> fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 > > >>>>>> fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 > > >>>>>> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > > >>>>>> 0) = 0xb7c63000 > > >>>>>> getdents64(3, /* 6 entries */, 1048576) = 144 > > >>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/C.pl", {st_mode=S_IFREG|0644, > > >>>>>> st_size=220, ...}) = 0 > > >>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/U.pl", {st_mode=S_IFREG|0644, > > >>>>>> st_size=251, ...}) = 0 > > >>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/D.pl", {st_mode=S_IFREG|0644, > > >>>>>> st_size=438, ...}) = 0 > > >>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/R.pl", {st_mode=S_IFREG|0644, > > >>>>>> st_size=426, ...}) = 0 > > >>>>>> getdents64(3, /* 0 entries */, 1048576) = 0 > > >>>>>> munmap(0xb7c63000, 1052672) = 0 > > >>>>>> close(3) = 0 > > >>>>>> open("share/perl/5.8.8/unicore/lib/gc_sc", > > >>>>>> O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3 > > >>>>>> fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 > > >>>>>> fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 > > >>>>>> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > > >>>>>> 0) = 0xb7c63000 > > >>>>>> getdents64(3, 0xb7c63024, 1048576) = -1 EIO (Input/output error) > > >>>>>> write(2, "rsync: readdir(\"/tmp/blah/usr0/s"..., 91rsync: > > >>>>>> readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"): > > >>>>>> Input/output > > >>>>>> error (5)) = 91 > > >>>>>> write(2, "\n", 1 > > >>>>>> ) = 1 > > >>>>>> munmap(0xb7c63000, 1052672) = 0 > > >>>>>> close(3) = 0 > > >>>>>> > > >>>>>> Thanks, > > >>>>>> > > >>>>>> Brent > > >>>>>> > > >>>>>> On Thu, 2 Aug 2007, Brent A Nelson wrote: > > >>>>>> > > >>>>>>> NFS reexport of a unified GlusterFS seems to be working fine as of TLA > > >>>>>>> 409. > > >>>>>>> I can make identical copies of a /usr area local-to-glusterfs and > > >>>>>>> glusterfs-to-glusterfs, hardlinks and all. Awesome! > > >>>>>>> > > >>>>>>> However, this is not true when AFR is added to the mix (rsync > > >>>>>>> glusterfs-to-glusterfs via NFS reexport): > > >>>>>>> > > >>>>>>> rsync: readdir("/tmp/blah/usr0/lib/perl/5.8.8/auto/POSIX"): Input/output > > >>>>>>> error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8"): Input/output error > > >>>>>>> (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/i18n/locales"): Input/output error > > >>>>>>> (5) > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/share/locale-langpack/en_GB/LC_MESSAGES"): > > >>>>>>> Input/output error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/groff/1.18.1/font/devps"): > > >>>>>>> Input/output > > >>>>>>> error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/man/man7"): Input/output error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/X11/xkb/symbols"): Input/output > > >>>>>>> error > > >>>>>>> (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Africa"): > > >>>>>>> Input/output > > >>>>>>> error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Asia"): Input/output > > >>>>>>> error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/America"): > > >>>>>>> Input/output > > >>>>>>> error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/Asia"): Input/output error > > >>>>>>> (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/doc"): Input/output error (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/share/consolefonts"): Input/output error > > >>>>>>> (5) > > >>>>>>> rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5) > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc64"): > > >>>>>>> Input/output error (5) > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"): > > >>>>>>> Input/output error (5) > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-mips"): > > >>>>>>> Input/output error (5) > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-parisc"): > > >>>>>>> Input/output error (5) > > >>>>>>> file has vanished: > > >>>>>>> "/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc/\#012" > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"): > > >>>>>>> Input/output error (5) > > >>>>>>> rsync: > > >>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"): > > >>>>>>> Input/output error (5) > > >>>>>>> ... > > >>>>>>> > > >>>>>>> Any ideas? Meanwhile, I'll try to track it down in strace (the output > > >>>>>>> will be > > >>>>>>> huge, but maybe I'll get lucky)... > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> > > >>>>>>> Brent > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> Gluster-devel mailing list > > >>>>>> Gluster-devel@xxxxxxxxxx > > >>>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > >> > > >> _______________________________________________ > > >> Gluster-devel mailing list > > >> Gluster-devel@xxxxxxxxxx > > >> http://lists.nongnu.org/mailman/listinfo/gluster-devel > > >> > > > > > > > > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@xxxxxxxxxx > > http://lists.nongnu.org/mailman/listinfo/gluster-devel > > >