Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.
Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
mounts as a glusterfs-client and exports as a nfs-server a ditributed
glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
14.04.
[root@cuentas11 ~]# cat /proc/mounts | grep home
recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
0 0
[root@cuentas11 ~]#
As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client
mount point: /home):
nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
over TCP --> glusterfs server (v3.7.4)
The errors at the nfs-client side are the same:
cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
-r--r--r-- 1 root root 7 nov 18 2016 444.txt
-r--r--r-- 1 root root 7 nov 18 2016 kk.txt
root@l056:/home/cdc#
root@l056:/home/cdc#
root@l056:/home/cdc# exit
cdc@l056:~$
See at the end the logs with and without the debug option activated at
the glusterfs client.
Thanks again, and sorry for the long message.
Omar
* Logs on the glusterfs client side with NO DEBUG:
[2016-11-18 15:45:43.681215] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt
(cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
[2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
[2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
* Logs on the glusterfs client side with DEBUG activated:
[2016-11-18 16:06:17.040092] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
[2016-11-18 16:06:17.047671] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.051871] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret 0 [Invalid argument]
[2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
[2016-11-18 16:06:17.065066] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.065897] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.066016] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.066669] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.066898] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.067019] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.067066] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:06:17.067151] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:06:17.067179] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.068781] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.068805] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069345] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069507] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069585] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069983] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:06:17.070038] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:06:17.070098] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.083168] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.083282] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084043] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.084048] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084092] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.084114] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084167] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.084091] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084377] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.084406] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:06:17.084511] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:06:17.084534] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.087173] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.087795] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.088014] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.088639] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.088804] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.088855] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.088819] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.089069] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.089380] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.089421] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.089443] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:06:17.089463] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:06:17.089575] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.093216] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt
(09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
[2016-11-18 16:06:17.095341] D [MSGID: 0]
[client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied
[Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0]
[dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
home-lab-3-client-3 returned -1 [Permission denied]
[2016-11-18 16:06:17.095480] D [MSGID: 0]
[dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
[Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095963] D [MSGID: 0]
[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0]
[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0]
[io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
[Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
[2016-11-18 16:06:17.097104] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)
** TEST DONE as user cdc --> ERROR
[2016-11-18 16:06:46.737686] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.246:49172: ping timer event already removed
[2016-11-18 16:06:49.545698] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.177:49157: ping timer event already removed
[2016-11-18 16:06:49.546474] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.189:49157: ping timer event already removed
[2016-11-18 16:06:49.547093] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.248:49157: ping timer event already removed
** user CDC becomed root at this point
[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
[2016-11-18 16:07:11.907884] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
[2016-11-18 16:07:11.909254] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:07:11.911018] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
[2016-11-18 16:07:11.913065] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.913757] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.913788] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914379] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914448] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914617] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914698] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:07:11.914743] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:07:11.914764] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916134] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.916158] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916780] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916994] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916906] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.917117] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:07:11.917368] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:07:11.917437] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
[2016-11-18 16:07:11.926690] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.927376] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.927401] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.927987] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.928043] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.928055] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.928058] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.928170] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.928174] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.928277] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.928360] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:07:11.928381] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:07:11.928403] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:07:11.929862] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931124] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.931148] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931927] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.932002] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931994] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931991] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.932078] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.932116] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.932315] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.932363] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:07:11.932383] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:07:11.932407] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:07:11.932545] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
** TEST DONE as root --> NO PROBLEM
[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)
** USER ROOT EXITED at this point
El 18/11/16 a las 15:16, Miklos Szeredi escribió:
On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente <omar@xxxxxxxxxx> wrote:
In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon
Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse included
(not as a module) so if it would be necessary to change the fuse mount
options some help would be needed.
Can you reproduce this on latest (e.g 4.8) kernel?
Also fuse debug logs (with whaterver kernel) could be useful; not sure
how to enable it for glusterfs, usual convention for fuse filesystems
is "-d".
Thanks,
Miklos
--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: omar@xxxxxxxxxx Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
