Re: Re: NFS reexport status

Brent A Nelson <brent@xxxxxxxxxxxx> · Fri, 3 Aug 2007 16:26:07 -0400 (EDT)

Woops, scratch that.  I accidentally tested the 2nd GlusterFS directory, 
not the final NFS mount.  Even with the GlusterFS reexport of the original 
GlusterFS, the issue is still present.

Thanks and sorry for the confusion,

Brent

On Fri, 3 Aug 2007, Brent A Nelson wrote:

I do have a workaround which can hide this bug, thanks to the wonderful 
flexibility of GlusterFS and the fact that it in itself is POSIX.  If I mount 
the GlusterFS as usual, but then use another glusterfs/glusterfsd pair to 
export and mount it and NFS reexport THAT, the problem does not appear.

Presumably, server-side AFR instead of client-side would also bypass the 
issue (not tested)...

Thanks,

Brent

On Fri, 3 Aug 2007, Brent A Nelson wrote:

I turned off self-heal on all the AFR volumes, remounted and reexported (I 
didn't delete the data; let me know if that is needed).

du -sk /tmp/blah/* (via NFS)
du: cannot access `/tmp/blah/usr0/include/c++/4.1.2/\a': No such file or 
directory
171832  /tmp/blah/usr0
109476  /tmp/blah/usr0-copy
du: cannot access `/tmp/blah/usr1/include/sys/\337O\004': No such file or 
directory
du: cannot access 
`/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/\v': No such 
file or directory
du: cannot access 
`/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/&\324\004': No 
such file or directory
du: cannot access 
`/tmp/blah/usr1/src/linux-headers-2.6.20-16/drivers/\006': No such file or 
directory
117472  /tmp/blah/usr1
58392   /tmp/blah/usr1-copy

It appears that self-heal isn't the culprit.

Thanks,

Brent

On Fri, 3 Aug 2007, Krishna Srinivas wrote:

Hi Brent,

Can you turn self-heal off (option self-heal off) and see how it
behaves?

Thanks
Krishna

On 8/3/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
A hopefully relevant strace snippet:

open("share/perl/5.8.8/unicore/lib/jt",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7c63000
getdents64(3, /* 6 entries */, 1048576) = 144
lstat64("share/perl/5.8.8/unicore/lib/jt/C.pl", {st_mode=S_IFREG|0644,
st_size=220, ...}) = 0
lstat64("share/perl/5.8.8/unicore/lib/jt/U.pl", {st_mode=S_IFREG|0644,
st_size=251, ...}) = 0
lstat64("share/perl/5.8.8/unicore/lib/jt/D.pl", {st_mode=S_IFREG|0644,
st_size=438, ...}) = 0
lstat64("share/perl/5.8.8/unicore/lib/jt/R.pl", {st_mode=S_IFREG|0644,
st_size=426, ...}) = 0
getdents64(3, /* 0 entries */, 1048576) = 0
munmap(0xb7c63000, 1052672)             = 0
close(3)                                = 0
open("share/perl/5.8.8/unicore/lib/gc_sc",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7c63000
getdents64(3, 0xb7c63024, 1048576)      = -1 EIO (Input/output error)
write(2, "rsync: readdir(\"/tmp/blah/usr0/s"..., 91rsync:
readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"): 
Input/output
error (5)) = 91
write(2, "\n", 1
)                       = 1
munmap(0xb7c63000, 1052672)             = 0
close(3)                                = 0

Thanks,

Brent

On Thu, 2 Aug 2007, Brent A Nelson wrote:

NFS reexport of a unified GlusterFS seems to be working fine as of TLA 
409.
I can make identical copies of a /usr area local-to-glusterfs and
glusterfs-to-glusterfs, hardlinks and all.  Awesome!

However, this is not true when AFR is added to the mix (rsync
glusterfs-to-glusterfs via NFS reexport):

rsync: readdir("/tmp/blah/usr0/lib/perl/5.8.8/auto/POSIX"): Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8"): Input/output error 
(5)
rsync: readdir("/tmp/blah/usr0/share/i18n/locales"): Input/output error 
(5)
rsync: 
readdir("/tmp/blah/usr0/share/locale-langpack/en_GB/LC_MESSAGES"):
Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/groff/1.18.1/font/devps"): 
Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/man/man7"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/X11/xkb/symbols"): Input/output 
error
(5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Africa"): 
Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Asia"): Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/America"): 
Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/Asia"): Input/output error 
(5)
rsync: readdir("/tmp/blah/usr0/share/doc"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/consolefonts"): Input/output error 
(5)
rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc64"):
Input/output error (5)
rsync: 
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-mips"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-parisc"):
Input/output error (5)
file has vanished:
"/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc/\#012"
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
Input/output error (5)
...

Any ideas? Meanwhile, I'll try to track it down in strace (the output 
will be
huge, but maybe I'll get lucky)...

Thanks,

Brent

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel