Re: Re: NFS reexport status

[Brent A Nelson <brent@xxxxxxxxxxxx>]

Hmm, I take part of my statement back.  rsync gives a few I/O errors in 
this simple scenario, BUT the copies seem to be good when checked later 
(this is not the case with more complex specs, I believe). Perhaps 
failures occur when writing to one subvolume but not the other, and then 
self-heal fixes it.  This would be consistent with the fact that the first 
du I run just after the rsync tends to be different from all subsequent 
ones.

Also odd: my rsyncs stopped complaining after a while. Unmounting the NFS 
and remounting brought the misbehavior back, though.

Thanks,

Brent

On Wed, 8 Aug 2007, Brent A Nelson wrote:

> On Wed, 8 Aug 2007, Krishna Srinivas wrote:
>
> > Hi Brent,
> >
> > Thanks. So if you use storage/posix under afr, you don't see
> > problem in nfs reexport.
>
> Correct, that worked fine.  Once I introduced protocol/client and 
> protocol/server, though, rsync -aH /usr/ /mount/nfs0/ gives I/O errors and an 
> inconsistent copy.
>
> > We are not able to reproduce this behaviour here.
>
> Did you try with the spec files I sent you (they only need two directories 
> available on a single machine), with an rsync of your /usr partition to the 
> NFS reexport (this can also be done via localhost, no additional machines 
> needed)? You are using the kernel NFS server, I assume, not one of the 
> user-mode NFS servers?
>
> > Can you give us access to your machines? is it possible?
>
> Yes, if the above doesn't do the trick, we can coordinate some way to get you 
> access.  Do you have an SSH public key I could add as an authorized key?
>
> Thanks,
>
> Brent
>
> > On 8/8/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > > Today, I tried switching to the Gluster-modified fuse-2.7.0, but I still
> > > encountered the same misbehavior with NFS reexport.  Heads-up: like
> > > someone else on the mailing list, I found that GlusterFS performance is
> > > MUCH slower with 2.7.0 than with my old 2.6.3, at least for simple "du"
> > > tests...
> > >
> > > Failing that, I thought I'd try to figure out the simplest specs to
> > > exhibit the issue; see attached.  I first tried glusterfs (no glusterfsd);
> > > it worked for a simple afr as well as unification of two afrs with no NFS
> > > reexport trouble.  As soon as I introduced a glusterfsd exporting to the
> > > glusterfs via protocol/client and protocol/server (via localhost),
> > > however, the rsync problems appeared.  I didn't see the issues with du in
> > > this simple setup, though (perhaps that problem will disappear when this
> > > problem is fixed, perhaps not).
> > >
> > > Thanks,
> > >
> > > Brent
> > >
> > > On Tue, 7 Aug 2007, Krishna Srinivas wrote:
> > >
> > > > Hi Brent,
> > > >
> > > > Those messages in log are harmless, I have removed them from the
> > > > source. Can you mail the spec files? I will see again if it can be
> > > > repro'd
> > > >
> > > > Thanks
> > > > Krishna
> > > >
> > > >
> > > > On 8/7/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > > > > I added debugging to all the AFR subvolumes.  On the du test, all it
> > > > > produced were lines like this over and over:
> > > > > 2007-08-06 17:23:41 C [dict.c:1094:data_to_ptr] libglusterfs/dict:
> > > > > @data=(nil)
> > > > >
> > > > > For the rsync (in addition to the @data=(nil) messages):
> > > > > rsync -a /tmp/blah/usr0/ /tmp/blah/nfs0/
> > > > > rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"):
> > > > > Input/output error (5)
> > > > > rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output error (5)
> > > > > rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output error (5)
> > > > > rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
> > > > > rsync:
> > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
> > > > > Input/output error (5)
> > > > > rsync:
> > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
> > > > > Input/output error (5)
> > > > > rsync:
> > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
> > > > > Input/output error (5)
> > > > > rsync: writefd_unbuffered failed to write 2672 bytes [sender]: Broken 
> > > > > pipe
> > > > > (32)
> > > > > rsync: close failed on "/tmp/blah/nfs0/games/.banner.vl3iqI": Operation
> > > > > not permitted (1)
> > > > > rsync: connection unexpectedly closed (98 bytes received so far) 
> > > > > [sender]
> > > > > rsync error: error in rsync protocol data stream (code 12) at io.c(454)
> > > > > [sender=2.6.9]
> > > > >
> > > > > The debug output is:
> > > > > 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] mirror3:
> > > > > (path=/nfs0/games/.banner.vl3iqI child=share3-0) op_ret=-1 op_errno=61
> > > > > 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] mirror3:
> > > > > (path=/nfs0/games/.banner.vl3iqI child=share3-1) op_ret=-1 op_errno=61
> > > > > 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] ns0:
> > > > > (path=/nfs0/games/.banner.vl3iqI child=ns0-0) op_ret=-1 op_errno=61
> > > > > 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] ns0:
> > > > > (path=/nfs0/games/.banner.vl3iqI child=ns0-1) op_ret=-1 op_errno=61
> > > > >
> > > > > This is new behavior; rsync didn't used to actually die, it just made
> > > > > incomplete copies.
> > > > >
> > > > >
> > > > > On Tue, 7 Aug 2007, Krishna Srinivas wrote:
> > > > >
> > > > > > Hi Brent,
> > > > > >
> > > > > > Can you put "option debug on" in afr subvolume and try the
> > > > > > du/rsync operations and mail the log?
> > > > > >
> > > > > > We are not able to reproduce the problem here, nfs is working
> > > > > > fine over afr.
> > > > > >
> > > > > > Thanks
> > > > > > Krishna
> > > > > >
> > > > > > On 8/4/07, Krishna Srinivas <krishna@xxxxxxxxxxxxx> wrote:
> > > > > > > rsync was failing for me without no_root_squash, so thought that
> > > > > > > might have been the culprit.
> > > > > > >
> > > > > > > If i put no_root_squash, nfs over afr works fine for me.
> > > > > > >
> > > > > > > Yes you are right, for some reason readdir() is not functioning
> > > > > > > properly I think because of which paths are getting corrupted.
> > > > > > >
> > > > > > > will get back to you.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Krishna
> > > > > > >
> > > > > > > On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > > > > > > > All of my tests were done with no_root_squash already, and all tests 
> > > > > > > > were
> > > > > > > > done as root.
> > > > > > > >
> > > > > > > > Without AFR, gluster and NFS reexports work fine with du and rsync.
> > > > > > > >
> > > > > > > > With AFR, gluster by itself is fine, but du and rsync from an NFS 
> > > > > > > > client
> > > > > > > > do not work properly. rsync gives lots of I/O errors and occasional 
> > > > > > > > "file
> > > > > > > > has vanished" messages for paths where the last element is junk.  du 
> > > > > > > > gives
> > > > > > > > incorrect sizes (smaller than it should) and occassionally gives "no 
> > > > > > > > such
> > > > > > > > file or directory", also for paths where the last element is junk. 
> > > > > > > > See
> > > > > > > > output below for examples from both of this junk.  Perhaps if you 
> > > > > > > > could
> > > > > > > > figure out how those paths are getting corrupted, the whole problem 
> > > > > > > > will
> > > > > > > > be resolved...
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Brent
> > > > > > > >
> > > > > > > > On Sat, 4 Aug 2007, Krishna Srinivas wrote:
> > > > > > > >
> > > > > > > > > Hi Brent,
> > > > > > > > >
> > > > > > > > > Can you add no_root_squash to exports file and reexport and mount
> > > > > > > > > using nfs and try to rsync as root and see if it works?
> > > > > > > > >
> > > > > > > > > like: "/mnt/gluster *(rw,no_root_squash,sync,fsid=3)"
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Krishna
> > > > > > > > >
> > > > > > > > > On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > > > > > > > > > Woops, scratch that.  I accidentally tested the 2nd GlusterFS 
> > > > > > > > > > directory,
> > > > > > > > > > not the final NFS mount.  Even with the GlusterFS reexport of the 
> > > > > > > > > > original
> > > > > > > > > > GlusterFS, the issue is still present.
> > > > > > > > > >
> > > > > > > > > > Thanks and sorry for the confusion,
> > > > > > > > > >
> > > > > > > > > > Brent
> > > > > > > > > >
> > > > > > > > > > On Fri, 3 Aug 2007, Brent A Nelson wrote:
> > > > > > > > > >
> > > > > > > > > > > I do have a workaround which can hide this bug, thanks to the 
> > > > > > > > > > > wonderful
> > > > > > > > > > > flexibility of GlusterFS and the fact that it in itself is POSIX. 
> > > > > > > > > > > If I mount
> > > > > > > > > > > the GlusterFS as usual, but then use another glusterfs/glusterfsd 
> > > > > > > > > > > pair to
> > > > > > > > > > > export and mount it and NFS reexport THAT, the problem does not 
> > > > > > > > > > > appear.
> > > > > > > > > > >
> > > > > > > > > > > Presumably, server-side AFR instead of client-side would also 
> > > > > > > > > > > bypass the
> > > > > > > > > > > issue (not tested)...
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Brent
> > > > > > > > > > >
> > > > > > > > > > > On Fri, 3 Aug 2007, Brent A Nelson wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I turned off self-heal on all the AFR volumes, remounted and 
> > > > > > > > > > > > reexported (I
> > > > > > > > > > > > didn't delete the data; let me know if that is needed).
> > > > > > > > > > > >
> > > > > > > > > > > > du -sk /tmp/blah/* (via NFS)
> > > > > > > > > > > > du: cannot access `/tmp/blah/usr0/include/c++/4.1.2/\a': No such 
> > > > > > > > > > > > file or
> > > > > > > > > > > > directory
> > > > > > > > > > > > 171832  /tmp/blah/usr0
> > > > > > > > > > > > 109476  /tmp/blah/usr0-copy
> > > > > > > > > > > > du: cannot access `/tmp/blah/usr1/include/sys/\337O\004': No such 
> > > > > > > > > > > > file or
> > > > > > > > > > > > directory
> > > > > > > > > > > > du: cannot access
> > > > > > > > > > > > `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/\v': 
> > > > > > > > > > > > No such
> > > > > > > > > > > > file or directory
> > > > > > > > > > > > du: cannot access
> > > > > > > > > > > > `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/&\324\004': 
> > > > > > > > > > > > No
> > > > > > > > > > > > such file or directory
> > > > > > > > > > > > du: cannot access
> > > > > > > > > > > > `/tmp/blah/usr1/src/linux-headers-2.6.20-16/drivers/\006': No 
> > > > > > > > > > > > such file or
> > > > > > > > > > > > directory
> > > > > > > > > > > > 117472  /tmp/blah/usr1
> > > > > > > > > > > > 58392   /tmp/blah/usr1-copy
> > > > > > > > > > > >
> > > > > > > > > > > > It appears that self-heal isn't the culprit.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Brent
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, 3 Aug 2007, Krishna Srinivas wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Brent,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Can you turn self-heal off (option self-heal off) and see how it
> > > > > > > > > > > > > behaves?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Krishna
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 8/3/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> > > > > > > > > > > > > > A hopefully relevant strace snippet:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > open("share/perl/5.8.8/unicore/lib/jt",
> > > > > > > > > > > > > > O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
> > > > > > > > > > > > > > fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> > > > > > > > > > > > > > fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
> > > > > > > > > > > > > > mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, 
> > > > > > > > > > > > > > MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > > > > > > > > > > > > > 0) = 0xb7c63000
> > > > > > > > > > > > > > getdents64(3, /* 6 entries */, 1048576) = 144
> > > > > > > > > > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/C.pl", 
> > > > > > > > > > > > > > {st_mode=S_IFREG|0644,
> > > > > > > > > > > > > > st_size=220, ...}) = 0
> > > > > > > > > > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/U.pl", 
> > > > > > > > > > > > > > {st_mode=S_IFREG|0644,
> > > > > > > > > > > > > > st_size=251, ...}) = 0
> > > > > > > > > > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/D.pl", 
> > > > > > > > > > > > > > {st_mode=S_IFREG|0644,
> > > > > > > > > > > > > > st_size=438, ...}) = 0
> > > > > > > > > > > > > > lstat64("share/perl/5.8.8/unicore/lib/jt/R.pl", 
> > > > > > > > > > > > > > {st_mode=S_IFREG|0644,
> > > > > > > > > > > > > > st_size=426, ...}) = 0
> > > > > > > > > > > > > > getdents64(3, /* 0 entries */, 1048576) = 0
> > > > > > > > > > > > > > munmap(0xb7c63000, 1052672)             = 0
> > > > > > > > > > > > > > close(3)                                = 0
> > > > > > > > > > > > > > open("share/perl/5.8.8/unicore/lib/gc_sc",
> > > > > > > > > > > > > > O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
> > > > > > > > > > > > > > fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> > > > > > > > > > > > > > fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
> > > > > > > > > > > > > > mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, 
> > > > > > > > > > > > > > MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > > > > > > > > > > > > > 0) = 0xb7c63000
> > > > > > > > > > > > > > getdents64(3, 0xb7c63024, 1048576)      = -1 EIO (Input/output 
> > > > > > > > > > > > > > error)
> > > > > > > > > > > > > > write(2, "rsync: readdir(\"/tmp/blah/usr0/s"..., 91rsync:
> > > > > > > > > > > > > > readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"):
> > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > error (5)) = 91
> > > > > > > > > > > > > > write(2, "\n", 1
> > > > > > > > > > > > > > )                       = 1
> > > > > > > > > > > > > > munmap(0xb7c63000, 1052672)             = 0
> > > > > > > > > > > > > > close(3)                                = 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Brent
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, 2 Aug 2007, Brent A Nelson wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > NFS reexport of a unified GlusterFS seems to be working fine 
> > > > > > > > > > > > > > > as of TLA
> > > > > > > > > > > > > > > 409.
> > > > > > > > > > > > > > > I can make identical copies of a /usr area local-to-glusterfs 
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > glusterfs-to-glusterfs, hardlinks and all.  Awesome!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > However, this is not true when AFR is added to the mix (rsync
> > > > > > > > > > > > > > > glusterfs-to-glusterfs via NFS reexport):
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/lib/perl/5.8.8/auto/POSIX"): 
> > > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8"): 
> > > > > > > > > > > > > > > Input/output error
> > > > > > > > > > > > > > > (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/i18n/locales"): 
> > > > > > > > > > > > > > > Input/output error
> > > > > > > > > > > > > > > (5)
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/share/locale-langpack/en_GB/LC_MESSAGES"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > rsync: 
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/share/groff/1.18.1/font/devps"):
> > > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output 
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output 
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/man/man7"): Input/output 
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/X11/xkb/symbols"): 
> > > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > > error
> > > > > > > > > > > > > > > (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Africa"):
> > > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Asia"): 
> > > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/America"):
> > > > > > > > > > > > > > > Input/output
> > > > > > > > > > > > > > > error (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/zoneinfo/Asia"): 
> > > > > > > > > > > > > > > Input/output error
> > > > > > > > > > > > > > > (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/doc"): Input/output error 
> > > > > > > > > > > > > > > (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/share/consolefonts"): 
> > > > > > > > > > > > > > > Input/output error
> > > > > > > > > > > > > > > (5)
> > > > > > > > > > > > > > > rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc64"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-mips"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-parisc"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > file has vanished:
> > > > > > > > > > > > > > > "/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc/\#012"
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > rsync:
> > > > > > > > > > > > > > > readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
> > > > > > > > > > > > > > > Input/output error (5)
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any ideas? Meanwhile, I'll try to track it down in strace (the 
> > > > > > > > > > > > > > > output
> > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > huge, but maybe I'll get lucky)...
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Brent
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > > > Gluster-devel mailing list
> > > > > > > > > > > > > > Gluster-devel@xxxxxxxxxx
> > > > > > > > > > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Gluster-devel mailing list
> > > > > > > > > > Gluster-devel@xxxxxxxxxx
> > > > > > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Gluster-devel mailing list
> > > > > > > > Gluster-devel@xxxxxxxxxx
> > > > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-devel