Hi Brent,
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsQ1/cLv2ycs16AmTRv530jMIpfwkjCzKlCeAtGW6fKEa2FQSxXFjIJe1r3UN5K0b6dSTZ3qsUCFd/UYcskirwyrqFa56dRXnPjEl1bQFmE7pB24ObFt6EfcmeL7JcaHzRgBP41vjygEN3i5zT5BTLfA9P8FzkfJUNkoAPsKx9e1I9TF1TGAOJW9FjvSqO0TSrJgwv4/cGzsgF/kDDl7iGUDuGmw2U8QlFc9oZun8tgWEb36bRuJ8/pEY/ej3d6s0/YQREyH5zo37VYfG1GO5o5ilHU3L0/V2EMeMSSqyHikL0gbKJAm5we4pcIZlCmdB2d9BHgLBImwI094zE4clyw==
krishna@krishna
Thats my rsa public key.
I tried with kernel based nfs server with more complicated sepcs with out
"success"
Thanks
Krishna
On 8/8/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> Hmm, I take part of my statement back. rsync gives a few I/O errors in
> this simple scenario, BUT the copies seem to be good when checked later
> (this is not the case with more complex specs, I believe). Perhaps
> failures occur when writing to one subvolume but not the other, and then
> self-heal fixes it. This would be consistent with the fact that the first
> du I run just after the rsync tends to be different from all subsequent
> ones.
>
> Also odd: my rsyncs stopped complaining after a while. Unmounting the NFS
> and remounting brought the misbehavior back, though.
>
> Thanks,
>
> Brent
>
> On Wed, 8 Aug 2007, Brent A Nelson wrote:
>
> > On Wed, 8 Aug 2007, Krishna Srinivas wrote:
> >
> >> Hi Brent,
> >>
> >> Thanks. So if you use storage/posix under afr, you don't see
> >> problem in nfs reexport.
> >
> > Correct, that worked fine. Once I introduced protocol/client and
> > protocol/server, though, rsync -aH /usr/ /mount/nfs0/ gives I/O errors and an
> > inconsistent copy.
> >
> >> We are not able to reproduce this behaviour here.
> >
> > Did you try with the spec files I sent you (they only need two directories
> > available on a single machine), with an rsync of your /usr partition to the
> > NFS reexport (this can also be done via localhost, no additional machines
> > needed)? You are using the kernel NFS server, I assume, not one of the
> > user-mode NFS servers?
> >
> >> Can you give us access to your machines? is it possible?
> >>
> >
> > Yes, if the above doesn't do the trick, we can coordinate some way to get you
> > access. Do you have an SSH public key I could add as an authorized key?
> >
> > Thanks,
> >
> > Brent
> >
> >> On 8/8/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> >>> Today, I tried switching to the Gluster-modified fuse-2.7.0, but I still
> >>> encountered the same misbehavior with NFS reexport. Heads-up: like
> >>> someone else on the mailing list, I found that GlusterFS performance is
> >>> MUCH slower with 2.7.0 than with my old 2.6.3, at least for simple "du"
> >>> tests...
> >>>
> >>> Failing that, I thought I'd try to figure out the simplest specs to
> >>> exhibit the issue; see attached. I first tried glusterfs (no glusterfsd);
> >>> it worked for a simple afr as well as unification of two afrs with no NFS
> >>> reexport trouble. As soon as I introduced a glusterfsd exporting to the
> >>> glusterfs via protocol/client and protocol/server (via localhost),
> >>> however, the rsync problems appeared. I didn't see the issues with du in
> >>> this simple setup, though (perhaps that problem will disappear when this
> >>> problem is fixed, perhaps not).
> >>>
> >>> Thanks,
> >>>
> >>> Brent
> >>>
> >>> On Tue, 7 Aug 2007, Krishna Srinivas wrote:
> >>>
> >>>> Hi Brent,
> >>>>
> >>>> Those messages in log are harmless, I have removed them from the
> >>>> source. Can you mail the spec files? I will see again if it can be
> >>>> repro'd
> >>>>
> >>>> Thanks
> >>>> Krishna
> >>>>
> >>>>
> >>>> On 8/7/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> >>>>> I added debugging to all the AFR subvolumes. On the du test, all it
> >>>>> produced were lines like this over and over:
> >>>>> 2007-08-06 17:23:41 C [dict.c:1094:data_to_ptr] libglusterfs/dict:
> >>>>> @data=(nil)
> >>>>>
> >>>>> For the rsync (in addition to the @data=(nil) messages):
> >>>>> rsync -a /tmp/blah/usr0/ /tmp/blah/nfs0/
> >>>>> rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"):
> >>>>> Input/output error (5)
> >>>>> rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output error (5)
> >>>>> rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output error (5)
> >>>>> rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
> >>>>> rsync:
> >>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
> >>>>> Input/output error (5)
> >>>>> rsync:
> >>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
> >>>>> Input/output error (5)
> >>>>> rsync:
> >>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
> >>>>> Input/output error (5)
> >>>>> rsync: writefd_unbuffered failed to write 2672 bytes [sender]: Broken
> >>>>> pipe
> >>>>> (32)
> >>>>> rsync: close failed on "/tmp/blah/nfs0/games/.banner.vl3iqI": Operation
> >>>>> not permitted (1)
> >>>>> rsync: connection unexpectedly closed (98 bytes received so far)
> >>>>> [sender]
> >>>>> rsync error: error in rsync protocol data stream (code 12) at io.c(454)
> >>>>> [sender=2.6.9]
> >>>>>
> >>>>> The debug output is:
> >>>>> 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] mirror3:
> >>>>> (path=/nfs0/games/.banner.vl3iqI child=share3-0) op_ret=-1 op_errno=61
> >>>>> 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] mirror3:
> >>>>> (path=/nfs0/games/.banner.vl3iqI child=share3-1) op_ret=-1 op_errno=61
> >>>>> 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] ns0:
> >>>>> (path=/nfs0/games/.banner.vl3iqI child=ns0-0) op_ret=-1 op_errno=61
> >>>>> 2007-08-06 17:33:58 E [afr.c:1389:afr_selfheal_getxattr_cbk] ns0:
> >>>>> (path=/nfs0/games/.banner.vl3iqI child=ns0-1) op_ret=-1 op_errno=61
> >>>>>
> >>>>> This is new behavior; rsync didn't used to actually die, it just made
> >>>>> incomplete copies.
> >>>>>
> >>>>>
> >>>>> On Tue, 7 Aug 2007, Krishna Srinivas wrote:
> >>>>>
> >>>>>> Hi Brent,
> >>>>>>
> >>>>>> Can you put "option debug on" in afr subvolume and try the
> >>>>>> du/rsync operations and mail the log?
> >>>>>>
> >>>>>> We are not able to reproduce the problem here, nfs is working
> >>>>>> fine over afr.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Krishna
> >>>>>>
> >>>>>> On 8/4/07, Krishna Srinivas <krishna@xxxxxxxxxxxxx> wrote:
> >>>>>>> rsync was failing for me without no_root_squash, so thought that
> >>>>>>> might have been the culprit.
> >>>>>>>
> >>>>>>> If i put no_root_squash, nfs over afr works fine for me.
> >>>>>>>
> >>>>>>> Yes you are right, for some reason readdir() is not functioning
> >>>>>>> properly I think because of which paths are getting corrupted.
> >>>>>>>
> >>>>>>> will get back to you.
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Krishna
> >>>>>>>
> >>>>>>> On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> >>>>>>>> All of my tests were done with no_root_squash already, and all tests
> >>>>>>>> were
> >>>>>>>> done as root.
> >>>>>>>>
> >>>>>>>> Without AFR, gluster and NFS reexports work fine with du and rsync.
> >>>>>>>>
> >>>>>>>> With AFR, gluster by itself is fine, but du and rsync from an NFS
> >>>>>>>> client
> >>>>>>>> do not work properly. rsync gives lots of I/O errors and occasional
> >>>>>>>> "file
> >>>>>>>> has vanished" messages for paths where the last element is junk. du
> >>>>>>>> gives
> >>>>>>>> incorrect sizes (smaller than it should) and occassionally gives "no
> >>>>>>>> such
> >>>>>>>> file or directory", also for paths where the last element is junk.
> >>>>>>>> See
> >>>>>>>> output below for examples from both of this junk. Perhaps if you
> >>>>>>>> could
> >>>>>>>> figure out how those paths are getting corrupted, the whole problem
> >>>>>>>> will
> >>>>>>>> be resolved...
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Brent
> >>>>>>>>
> >>>>>>>> On Sat, 4 Aug 2007, Krishna Srinivas wrote:
> >>>>>>>>
> >>>>>>>>> Hi Brent,
> >>>>>>>>>
> >>>>>>>>> Can you add no_root_squash to exports file and reexport and mount
> >>>>>>>>> using nfs and try to rsync as root and see if it works?
> >>>>>>>>>
> >>>>>>>>> like: "/mnt/gluster *(rw,no_root_squash,sync,fsid=3)"
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>> Krishna
> >>>>>>>>>
> >>>>>>>>> On 8/4/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> >>>>>>>>>> Woops, scratch that. I accidentally tested the 2nd GlusterFS
> >>>>>>>>>> directory,
> >>>>>>>>>> not the final NFS mount. Even with the GlusterFS reexport of the
> >>>>>>>>>> original
> >>>>>>>>>> GlusterFS, the issue is still present.
> >>>>>>>>>>
> >>>>>>>>>> Thanks and sorry for the confusion,
> >>>>>>>>>>
> >>>>>>>>>> Brent
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 3 Aug 2007, Brent A Nelson wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I do have a workaround which can hide this bug, thanks to the
> >>>>>>>>>>> wonderful
> >>>>>>>>>>> flexibility of GlusterFS and the fact that it in itself is POSIX.
> >>>>>>>>>>> If I mount
> >>>>>>>>>>> the GlusterFS as usual, but then use another glusterfs/glusterfsd
> >>>>>>>>>>> pair to
> >>>>>>>>>>> export and mount it and NFS reexport THAT, the problem does not
> >>>>>>>>>>> appear.
> >>>>>>>>>>>
> >>>>>>>>>>> Presumably, server-side AFR instead of client-side would also
> >>>>>>>>>>> bypass the
> >>>>>>>>>>> issue (not tested)...
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Brent
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, 3 Aug 2007, Brent A Nelson wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I turned off self-heal on all the AFR volumes, remounted and
> >>>>>>>>>>>> reexported (I
> >>>>>>>>>>>> didn't delete the data; let me know if that is needed).
> >>>>>>>>>>>>
> >>>>>>>>>>>> du -sk /tmp/blah/* (via NFS)
> >>>>>>>>>>>> du: cannot access `/tmp/blah/usr0/include/c++/4.1.2/\a': No such
> >>>>>>>>>>>> file or
> >>>>>>>>>>>> directory
> >>>>>>>>>>>> 171832 /tmp/blah/usr0
> >>>>>>>>>>>> 109476 /tmp/blah/usr0-copy
> >>>>>>>>>>>> du: cannot access `/tmp/blah/usr1/include/sys/\337O\004': No such
> >>>>>>>>>>>> file or
> >>>>>>>>>>>> directory
> >>>>>>>>>>>> du: cannot access
> >>>>>>>>>>>> `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/\v':
> >>>>>>>>>>>> No such
> >>>>>>>>>>>> file or directory
> >>>>>>>>>>>> du: cannot access
> >>>>>>>>>>>> `/tmp/blah/usr1/src/linux-headers-2.6.20-16/include/asm-ia64/&\324\004':
> >>>>>>>>>>>> No
> >>>>>>>>>>>> such file or directory
> >>>>>>>>>>>> du: cannot access
> >>>>>>>>>>>> `/tmp/blah/usr1/src/linux-headers-2.6.20-16/drivers/\006': No
> >>>>>>>>>>>> such file or
> >>>>>>>>>>>> directory
> >>>>>>>>>>>> 117472 /tmp/blah/usr1
> >>>>>>>>>>>> 58392 /tmp/blah/usr1-copy
> >>>>>>>>>>>>
> >>>>>>>>>>>> It appears that self-heal isn't the culprit.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Brent
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, 3 Aug 2007, Krishna Srinivas wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Brent,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Can you turn self-heal off (option self-heal off) and see how it
> >>>>>>>>>>>>> behaves?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>> Krishna
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 8/3/07, Brent A Nelson <brent@xxxxxxxxxxxx> wrote:
> >>>>>>>>>>>>>> A hopefully relevant strace snippet:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> open("share/perl/5.8.8/unicore/lib/jt",
> >>>>>>>>>>>>>> O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
> >>>>>>>>>>>>>> fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> >>>>>>>>>>>>>> fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
> >>>>>>>>>>>>>> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE,
> >>>>>>>>>>>>>> MAP_PRIVATE|MAP_ANONYMOUS, -1,
> >>>>>>>>>>>>>> 0) = 0xb7c63000
> >>>>>>>>>>>>>> getdents64(3, /* 6 entries */, 1048576) = 144
> >>>>>>>>>>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/C.pl",
> >>>>>>>>>>>>>> {st_mode=S_IFREG|0644,
> >>>>>>>>>>>>>> st_size=220, ...}) = 0
> >>>>>>>>>>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/U.pl",
> >>>>>>>>>>>>>> {st_mode=S_IFREG|0644,
> >>>>>>>>>>>>>> st_size=251, ...}) = 0
> >>>>>>>>>>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/D.pl",
> >>>>>>>>>>>>>> {st_mode=S_IFREG|0644,
> >>>>>>>>>>>>>> st_size=438, ...}) = 0
> >>>>>>>>>>>>>> lstat64("share/perl/5.8.8/unicore/lib/jt/R.pl",
> >>>>>>>>>>>>>> {st_mode=S_IFREG|0644,
> >>>>>>>>>>>>>> st_size=426, ...}) = 0
> >>>>>>>>>>>>>> getdents64(3, /* 0 entries */, 1048576) = 0
> >>>>>>>>>>>>>> munmap(0xb7c63000, 1052672) = 0
> >>>>>>>>>>>>>> close(3) = 0
> >>>>>>>>>>>>>> open("share/perl/5.8.8/unicore/lib/gc_sc",
> >>>>>>>>>>>>>> O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
> >>>>>>>>>>>>>> fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> >>>>>>>>>>>>>> fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
> >>>>>>>>>>>>>> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE,
> >>>>>>>>>>>>>> MAP_PRIVATE|MAP_ANONYMOUS, -1,
> >>>>>>>>>>>>>> 0) = 0xb7c63000
> >>>>>>>>>>>>>> getdents64(3, 0xb7c63024, 1048576) = -1 EIO (Input/output
> >>>>>>>>>>>>>> error)
> >>>>>>>>>>>>>> write(2, "rsync: readdir(\"/tmp/blah/usr0/s"..., 91rsync:
> >>>>>>>>>>>>>> readdir("/tmp/blah/usr0/share/perl/5.8.8/unicore/lib/gc_sc"):
> >>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>> error (5)) = 91
> >>>>>>>>>>>>>> write(2, "\n", 1
> >>>>>>>>>>>>>> ) = 1
> >>>>>>>>>>>>>> munmap(0xb7c63000, 1052672) = 0
> >>>>>>>>>>>>>> close(3) = 0
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Brent
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, 2 Aug 2007, Brent A Nelson wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> NFS reexport of a unified GlusterFS seems to be working fine
> >>>>>>>>>>>>>>> as of TLA
> >>>>>>>>>>>>>>> 409.
> >>>>>>>>>>>>>>> I can make identical copies of a /usr area local-to-glusterfs
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> glusterfs-to-glusterfs, hardlinks and all. Awesome!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> However, this is not true when AFR is added to the mix (rsync
> >>>>>>>>>>>>>>> glusterfs-to-glusterfs via NFS reexport):
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/lib/perl/5.8.8/auto/POSIX"):
> >>>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/perl/5.8.8"):
> >>>>>>>>>>>>>>> Input/output error
> >>>>>>>>>>>>>>> (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/i18n/locales"):
> >>>>>>>>>>>>>>> Input/output error
> >>>>>>>>>>>>>>> (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/share/locale-langpack/en_GB/LC_MESSAGES"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/share/groff/1.18.1/font/devps"):
> >>>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/man/man1"): Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/man/man8"): Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/man/man7"): Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/X11/xkb/symbols"):
> >>>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>> (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Africa"):
> >>>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/Asia"):
> >>>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/right/America"):
> >>>>>>>>>>>>>>> Input/output
> >>>>>>>>>>>>>>> error (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/zoneinfo/Asia"):
> >>>>>>>>>>>>>>> Input/output error
> >>>>>>>>>>>>>>> (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/doc"): Input/output error
> >>>>>>>>>>>>>>> (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/share/consolefonts"):
> >>>>>>>>>>>>>>> Input/output error
> >>>>>>>>>>>>>>> (5)
> >>>>>>>>>>>>>>> rsync: readdir("/tmp/blah/usr0/bin"): Input/output error (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc64"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/linux"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-mips"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-parisc"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> file has vanished:
> >>>>>>>>>>>>>>> "/tmp/blah/usr0/src/linux-headers-2.6.20-16/include/asm-sparc/\#012"
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/config"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> rsync:
> >>>>>>>>>>>>>>> readdir("/tmp/blah/usr0/src/linux-headers-2.6.20-16-server/include/linux"):
> >>>>>>>>>>>>>>> Input/output error (5)
> >>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Any ideas? Meanwhile, I'll try to track it down in strace (the
> >>>>>>>>>>>>>>> output
> >>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>> huge, but maybe I'll get lucky)...
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Brent
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> Gluster-devel mailing list
> >>>>>>>>>>>>>> Gluster-devel@xxxxxxxxxx
> >>>>>>>>>>>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> Gluster-devel mailing list
> >>>>>>>>>> Gluster-devel@xxxxxxxxxx
> >>>>>>>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Gluster-devel mailing list
> >>>>>>>> Gluster-devel@xxxxxxxxxx
> >>>>>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
