Re: Re: NFS reexport status

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



All of my tests were done with no_root_squash already, and all tests were done as root.

Without AFR, gluster and NFS reexports work fine with du and rsync.

With AFR, gluster by itself is fine, but du and rsync from an NFS client do not work properly. rsync gives lots of I/O errors and occasional "file has vanished" messages for paths where the last element is junk. du gives incorrect sizes (smaller than it should) and occassionally gives "no such file or directory", also for paths where the last element is junk. See output below for examples from both of this junk. Perhaps if you could figure out how those paths are getting corrupted, the whole problem will be resolved...

Thanks,

Brent

On Sat, 4 Aug 2007, Krishna Srinivas wrote:

Hi Brent,

Can you add no_root_squash to exports file and reexport and mount
using nfs and try to rsync as root and see if it works?

like: "/mnt/gluster *(rw,no_root_squash,sync,fsid=3)"

Thanks
Krishna

On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
Woops, scratch that.  I accidentally tested the 2nd GlusterFS directory,
not the final NFS mount.  Even with the GlusterFS reexport of the original
GlusterFS, the issue is still present.

Thanks and sorry for the confusion,

Brent

On Fri, 3 Aug 2007, Brent A Nelson wrote:

I do have a workaround which can hide this bug, thanks to the wonderful
flexibility of GlusterFS and the fact that it in itself is POSIX.  If I mount
the GlusterFS as usual, but then use another glusterfs/glusterfsd pair to
export and mount it and NFS reexport THAT, the problem does not appear.

Presumably, server-side AFR instead of client-side would also bypass the
issue (not tested)...

Thanks,

Brent

On Fri, 3 Aug 2007, Brent A Nelson wrote:

I turned off self-heal on all the AFR volumes, remounted and reexported (I
didn't delete the data; let me know if that is needed).

du -sk /tmp/blah/* (via NFS)
du: cannot access `/tmp/blah/usr0/include/c++/4.1.2/\a': No such file or
directory
171832  /tmp/blah/usr0
109476  /tmp/blah/usr0-copy
du: cannot access `/tmp/blah/usr1/include/sys/\337O\004': No such file or
directory
du: cannot access
`/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/\v': No such
file or directory
du: cannot access
`/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/&\324\004': No
such file or directory
du: cannot access
`/tmp/blah/usr1/src/linux-headers-2.6.20-16/drivers/\006': No such file or
directory
117472  /tmp/blah/usr1
58392   /tmp/blah/usr1-copy

It appears that self-heal isn't the culprit.

Thanks,

Brent

On Fri, 3 Aug 2007, Krishna Srinivas wrote:

Hi Brent,

Can you turn self-heal off (option self-heal off) and see how it
behaves?

Thanks
Krishna

On 8/3/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
A hopefully relevant strace snippet:

open("share/perl/5.8.8/unicore/lib/jt",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7c63000
getdents64(3, /* 6 entries */, 1048576) = 144
lstat64("share/perl/5.8.8/unicore/lib/jt/C.pl", {st_mode=S_IFREG|0644,
st_size=220, ...}) = 0
lstat64("share/perl/5.8.8/unicore/lib/jt/U.pl", {st_mode=S_IFREG|0644,
st_size=251, ...}) = 0
lstat64("share/perl/5.8.8/unicore/lib/jt/D.pl", {st_mode=S_IFREG|0644,
st_size=438, ...}) = 0
lstat64("share/perl/5.8.8/unicore/lib/jt/R.pl", {st_mode=S_IFREG|0644,
st_size=426, ...}) = 0
getdents64(3, /* 0 entries */, 1048576) = 0
munmap(0xb7c63000, 1052672)             = 0
close(3)                                = 0
open("share/perl/5.8.8/unicore/lib/gc_sc",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7c63000
getdents64(3, 0xb7c63024, 1048576)      = -1 EIO (Input/output error)
write(2, "rsync: readdir(\"/tmp/blah/usr0/s"..., 91rsync:
readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"):
Input/output
error (5)) = 91
write(2, "\n", 1
)                       = 1
munmap(0xb7c63000, 1052672)             = 0
close(3)                                = 0

Thanks,

Brent

On Thu, 2 Aug 2007, Brent A Nelson wrote:

NFS reexport of a unified GlusterFS seems to be working fine as of TLA
409.
I can make identical copies of a /usr area local-to-glusterfs and
glusterfs-to-glusterfs, hardlinks and all.  Awesome!

However, this is not true when AFR is added to the mix (rsync
glusterfs-to-glusterfs via NFS reexport):

rsync: readdir("/tmp/blah/usr0/lib/perl/5.8.8/auto/POSIX"): Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8"): Input/output error
(5)
rsync: readdir("/tmp/blah/usr0/share/i18n/locales"): Input/output error
(5)
rsync:
readdir("/tmp/blah/usr0/share/locale-langpack/en_GB/LC_MESSAGES"):
Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/groff/1.18.1/font/devps"):
Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/man/man7"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/X11/xkb/symbols"): Input/output
error
(5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Africa"):
Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Asia"): Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/America"):
Input/output
error (5)
rsync: readdir("/tmp/blah/usr0/share/zoneinfo/Asia"): Input/output error
(5)
rsync: readdir("/tmp/blah/usr0/share/doc"): Input/output error (5)
rsync: readdir("/tmp/blah/usr0/share/consolefonts"): Input/output error
(5)
rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc64"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-mips"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-parisc"):
Input/output error (5)
file has vanished:
"/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc/\#012"
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
Input/output error (5)
rsync:
readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
Input/output error (5)
...

Any ideas? Meanwhile, I'll try to track it down in strace (the output
will be
huge, but maybe I'll get lucky)...

Thanks,

Brent



_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel






_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel






[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux