Re: Re: NFS reexport status

[Brent A Nelson <brent@xxxxxxxxxxxx>]

All of my tests were done with no_root_squash already, and all tests were 
done as root.

Without AFR, gluster and NFS reexports work fine with du and rsync.

With AFR, gluster by itself is fine, but du and rsync from an NFS client 
do not work properly. rsync gives lots of I/O errors and occasional "file 
has vanished" messages for paths where the last element is junk.  du gives 
incorrect sizes (smaller than it should) and occassionally gives "no such 
file or directory", also for paths where the last element is junk.  See 
output below for examples from both of this junk.  Perhaps if you could 
figure out how those paths are getting corrupted, the whole problem will 
be resolved...

Thanks,

Brent

On Sat, 4 Aug 2007, Krishna Srinivas wrote:

> Hi Brent,
>
> Can you add no_root_squash to exports file and reexport and mount
> using nfs and try to rsync as root and see if it works?
>
> like: "/mnt/gluster *(rw,no_root_squash,sync,fsid=3)"
>
> Thanks
> Krishna
>
> On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > Woops, scratch that.  I accidentally tested the 2nd GlusterFS directory,
> > not the final NFS mount.  Even with the GlusterFS reexport of the original
> > GlusterFS, the issue is still present.
> >
> > Thanks and sorry for the confusion,
> >
> > Brent
> >
> > On Fri, 3 Aug 2007, Brent A Nelson wrote:
> >
> > > I do have a workaround which can hide this bug, thanks to the wonderful
> > > flexibility of GlusterFS and the fact that it in itself is POSIX.  If I mount
> > > the GlusterFS as usual, but then use another glusterfs/glusterfsd pair to
> > > export and mount it and NFS reexport THAT, the problem does not appear.
> > >
> > > Presumably, server-side AFR instead of client-side would also bypass the
> > > issue (not tested)...
> > >
> > > Thanks,
> > >
> > > Brent
> > >
> > > On Fri, 3 Aug 2007, Brent A Nelson wrote:
> > >
> > > > I turned off self-heal on all the AFR volumes, remounted and reexported (I
> > > > didn't delete the data; let me know if that is needed).
> > > >
> > > > du -sk /tmp/blah/* (via NFS)
> > > > du: cannot access `/tmp/blah/usr0/include/c++/4.1.2/\a': No such file or
> > > > directory
> > > > 171832  /tmp/blah/usr0
> > > > 109476  /tmp/blah/usr0-copy
> > > > du: cannot access `/tmp/blah/usr1/include/sys/\337O\004': No such file or
> > > > directory
> > > > du: cannot access
> > > > `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/\v': No such
> > > > file or directory
> > > > du: cannot access
> > > > `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/&\324\004': No
> > > > such file or directory
> > > > du: cannot access
> > > > `/tmp/blah/usr1/src/linux-headers-2.6.20-16/drivers/\006': No such file or
> > > > directory
> > > > 117472  /tmp/blah/usr1
> > > > 58392   /tmp/blah/usr1-copy
> > > >
> > > > It appears that self-heal isn't the culprit.
> > > >
> > > > Thanks,
> > > >
> > > > Brent
> > > >
> > > > On Fri, 3 Aug 2007, Krishna Srinivas wrote:
> > > >
> > > > > Hi Brent,
> > > > >
> > > > > Can you turn self-heal off (option self-heal off) and see how it
> > > > > behaves?
> > > > >
> > > > > Thanks
> > > > > Krishna
> > > > >
> > > > > On 8/3/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > > > > > A hopefully relevant strace snippet:
> > > > > >
> > > > > > open("share/perl/5.8.8/unicore/lib/jt",
> > > > > > O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
> > > > > > fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> > > > > > fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
> > > > > > mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > > > > > 0) = 0xb7c63000
> > > > > > getdents64(3, /* 6 entries */, 1048576) = 144
> > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/C.pl", {st_mode=S_IFREG|0644,
> > > > > > st_size=220, ...}) = 0
> > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/U.pl", {st_mode=S_IFREG|0644,
> > > > > > st_size=251, ...}) = 0
> > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/D.pl", {st_mode=S_IFREG|0644,
> > > > > > st_size=438, ...}) = 0
> > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/R.pl", {st_mode=S_IFREG|0644,
> > > > > > st_size=426, ...}) = 0
> > > > > > getdents64(3, /* 0 entries */, 1048576) = 0
> > > > > > munmap(0xb7c63000, 1052672)             = 0
> > > > > > close(3)                                = 0
> > > > > > open("share/perl/5.8.8/unicore/lib/gc_sc",
> > > > > > O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
> > > > > > fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> > > > > > fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
> > > > > > mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > > > > > 0) = 0xb7c63000
> > > > > > getdents64(3, 0xb7c63024, 1048576)      = -1 EIO (Input/output error)
> > > > > > write(2, "rsync: readdir(\"/tmp/blah/usr0/s"..., 91rsync:
> > > > > > readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"):
> > > > > > Input/output
> > > > > > error (5)) = 91
> > > > > > write(2, "\n", 1
> > > > > > )                       = 1
> > > > > > munmap(0xb7c63000, 1052672)             = 0
> > > > > > close(3)                                = 0
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Brent
> > > > > >
> > > > > > On Thu, 2 Aug 2007, Brent A Nelson wrote:
> > > > > >
> > > > > > > NFS reexport of a unified GlusterFS seems to be working fine as of TLA
> > > > > > > 409.
> > > > > > > I can make identical copies of a /usr area local-to-glusterfs and
> > > > > > > glusterfs-to-glusterfs, hardlinks and all.  Awesome!
> > > > > > >
> > > > > > > However, this is not true when AFR is added to the mix (rsync
> > > > > > > glusterfs-to-glusterfs via NFS reexport):
> > > > > > >
> > > > > > > rsync: readdir("/tmp/blah/usr0/lib/perl/5.8.8/auto/POSIX"): Input/output
> > > > > > > error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8"): Input/output error
> > > > > > > (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/i18n/locales"): Input/output error
> > > > > > > (5)
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/share/locale-langpack/en_GB/LC_MESSAGES"):
> > > > > > > Input/output error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/groff/1.18.1/font/devps"):
> > > > > > > Input/output
> > > > > > > error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/man/man7"): Input/output error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/X11/xkb/symbols"): Input/output
> > > > > > > error
> > > > > > > (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Africa"):
> > > > > > > Input/output
> > > > > > > error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Asia"): Input/output
> > > > > > > error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/America"):
> > > > > > > Input/output
> > > > > > > error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/Asia"): Input/output error
> > > > > > > (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/doc"): Input/output error (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/share/consolefonts"): Input/output error
> > > > > > > (5)
> > > > > > > rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc64"):
> > > > > > > Input/output error (5)
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
> > > > > > > Input/output error (5)
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-mips"):
> > > > > > > Input/output error (5)
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-parisc"):
> > > > > > > Input/output error (5)
> > > > > > > file has vanished:
> > > > > > > "/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc/\#012"
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
> > > > > > > Input/output error (5)
> > > > > > > rsync:
> > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
> > > > > > > Input/output error (5)
> > > > > > > ...
> > > > > > >
> > > > > > > Any ideas? Meanwhile, I'll try to track it down in strace (the output
> > > > > > > will be
> > > > > > > huge, but maybe I'll get lucky)...
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Brent
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Gluster-devel mailing list
> > > > > > Gluster-devel@xxxxxxxxxx
> > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel@xxxxxxxxxx
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel