I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
On 10-12-13 09:30, Johan Huysmans wrote:
Hi All,
When reading some files we get this error:
md5sum: /path/to/file.xml: Structure needs cleaning
in /var/log/glusterfs/mnt-sharedfs.log we see these errors:
[2013-12-10 08:07:32.256910] W
[client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0:
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
[client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1:
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
This is my gluster configuration:
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
Options Reconfigured:
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
And this is how the applications work:
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem, and
can read that file without problems.
On the other node we have problems (error messages above) reading that
file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and mount
of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
[2013-12-10 08:37:58.532425] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.532493] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.532513] D [afr-self-heal-common.c:887:afr_mark_sources] 0-testvolume-replicate-0: Number of sources: 0
[2013-12-10 08:37:58.532530] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-testvolume-replicate-0: returning read_child: 1
[2013-12-10 08:37:58.532546] D [afr-common.c:1380:afr_lookup_select_read_child] 0-testvolume-replicate-0: Source selected as 1 for /
[2013-12-10 08:37:58.532564] D [afr-common.c:1117:afr_lookup_build_response_params] 0-testvolume-replicate-0: Building lookup response from 1
[2013-12-10 08:37:58.533041] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path: failed to get the gfid from dict
[2013-12-10 08:37:58.540362] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.540395] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.540412] D [afr-self-heal-common.c:887:afr_mark_sources] 0-testvolume-replicate-0: Number of sources: 0
[2013-12-10 08:37:58.540428] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-testvolume-replicate-0: returning read_child: 0
[2013-12-10 08:37:58.540443] D [afr-common.c:1380:afr_lookup_select_read_child] 0-testvolume-replicate-0: Source selected as 0 for /path
[2013-12-10 08:37:58.540460] D [afr-common.c:1117:afr_lookup_build_response_params] 0-testvolume-replicate-0: Building lookup response from 0
[2013-12-10 08:37:58.540804] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to: failed to get the gfid from dict
[2013-12-10 08:37:58.541377] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.541408] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.541425] D [afr-self-heal-common.c:887:afr_mark_sources] 0-testvolume-replicate-0: Number of sources: 0
[2013-12-10 08:37:58.541440] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-testvolume-replicate-0: returning read_child: 1
[2013-12-10 08:37:58.541455] D [afr-common.c:1380:afr_lookup_select_read_child] 0-testvolume-replicate-0: Source selected as 1 for /path/to
[2013-12-10 08:37:58.541473] D [afr-common.c:1117:afr_lookup_build_response_params] 0-testvolume-replicate-0: Building lookup response from 1
[2013-12-10 08:37:58.541794] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some: failed to get the gfid from dict
[2013-12-10 08:37:58.542360] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.542393] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.542410] D [afr-self-heal-common.c:887:afr_mark_sources] 0-testvolume-replicate-0: Number of sources: 0
[2013-12-10 08:37:58.542426] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-testvolume-replicate-0: returning read_child: 0
[2013-12-10 08:37:58.542441] D [afr-common.c:1380:afr_lookup_select_read_child] 0-testvolume-replicate-0: Source selected as 0 for /path/to/some
[2013-12-10 08:37:58.542458] D [afr-common.c:1117:afr_lookup_build_response_params] 0-testvolume-replicate-0: Building lookup response from 0
[2013-12-10 08:37:58.542837] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some/dir: failed to get the gfid from dict
[2013-12-10 08:37:58.543328] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.543358] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-testvolume-replicate-0: pending_matrix: [ 0 0 ]
[2013-12-10 08:37:58.543396] D [afr-self-heal-common.c:887:afr_mark_sources] 0-testvolume-replicate-0: Number of sources: 0
[2013-12-10 08:37:58.543413] D [afr-self-heal-data.c:825:afr_lookup_select_read_child_by_txn_type] 0-testvolume-replicate-0: returning read_child: 1
[2013-12-10 08:37:58.543433] D [afr-common.c:1380:afr_lookup_select_read_child] 0-testvolume-replicate-0: Source selected as 1 for /path/to/some/dir
[2013-12-10 08:37:58.543451] D [afr-common.c:1117:afr_lookup_build_response_params] 0-testvolume-replicate-0: Building lookup response from 1
[2013-12-10 08:37:58.543701] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.544148] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory
[2013-12-10 08:37:58.544573] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory
[2013-12-10 08:37:58.544677] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.545082] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.545443] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.545592] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some/dir/file.xml: failed to get the gfid from dict
[2013-12-10 08:37:58.545960] D [dht-common.c:267:dht_discover_cbk] 0-testvolume-dht: lookup of /path/to/some/dir/file.xml on testvolume-replicate-0 returned error (No such file or directory)
[2013-12-10 08:37:58.546018] D [dht-layout.c:626:dht_layout_normalize] 0-testvolume-dht: directory /path/to/some/dir/file.xml looked up first time
[2013-12-10 08:37:58.546121] D [dht-layout.c:646:dht_layout_normalize] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x51e) [0xec172e] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_cbk+0x282) [0x3a30f2] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_complete+0x364) [0x39c3a4]))) 0-testvolume-dht: path=/path/to/some/dir/file.xml err=No such file or directory on subvol=testvolume-replicate-0
[2013-12-10 08:37:58.546141] D [dht-common.c:195:dht_discover_complete] 0-testvolume-dht: normalizing failed on /path/to/some/dir/file.xml (overlaps/holes present: yes, ENOENT errors: 0)
[2013-12-10 08:37:58.546458] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.546854] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory
[2013-12-10 08:37:58.583538] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory
[2013-12-10 08:37:58.583676] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.584127] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.584745] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.584870] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some/dir/file.xml: failed to get the gfid from dict
[2013-12-10 08:37:58.585273] D [dht-common.c:267:dht_discover_cbk] 0-testvolume-dht: lookup of /path/to/some/dir/file.xml on testvolume-replicate-0 returned error (No such file or directory)
[2013-12-10 08:37:58.585306] D [dht-layout.c:626:dht_layout_normalize] 0-testvolume-dht: directory /path/to/some/dir/file.xml looked up first time
[2013-12-10 08:37:58.585362] D [dht-layout.c:646:dht_layout_normalize] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x51e) [0xec172e] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_cbk+0x282) [0x3a30f2] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_complete+0x364) [0x39c3a4]))) 0-testvolume-dht: path=/path/to/some/dir/file.xml err=No such file or directory on subvol=testvolume-replicate-0
[2013-12-10 08:37:58.585372] D [dht-common.c:195:dht_discover_complete] 0-testvolume-dht: normalizing failed on /path/to/some/dir/file.xml (overlaps/holes present: yes, ENOENT errors: 0)
[2013-12-10 08:37:58.585692] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.586058] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory
[2013-12-10 08:37:58.586441] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory
[2013-12-10 08:37:58.586499] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.586707] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.587095] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.587211] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some/dir/file.xml: failed to get the gfid from dict
[2013-12-10 08:37:58.587676] D [dht-common.c:267:dht_discover_cbk] 0-testvolume-dht: lookup of /path/to/some/dir/file.xml on testvolume-replicate-0 returned error (No such file or directory)
[2013-12-10 08:37:58.587718] D [dht-layout.c:626:dht_layout_normalize] 0-testvolume-dht: directory /path/to/some/dir/file.xml looked up first time
[2013-12-10 08:37:58.587766] D [dht-layout.c:646:dht_layout_normalize] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x51e) [0xec172e] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_cbk+0x282) [0x3a30f2] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_complete+0x364) [0x39c3a4]))) 0-testvolume-dht: path=/path/to/some/dir/file.xml err=No such file or directory on subvol=testvolume-replicate-0
[2013-12-10 08:37:58.587785] D [dht-common.c:195:dht_discover_complete] 0-testvolume-dht: normalizing failed on /path/to/some/dir/file.xml (overlaps/holes present: yes, ENOENT errors: 0)
[2013-12-10 08:37:58.588079] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.588516] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory
[2013-12-10 08:37:58.588846] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory
[2013-12-10 08:37:58.588969] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.589397] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.589769] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.589879] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some/dir/file.xml: failed to get the gfid from dict
[2013-12-10 08:37:58.590284] D [dht-common.c:267:dht_discover_cbk] 0-testvolume-dht: lookup of /path/to/some/dir/file.xml on testvolume-replicate-0 returned error (No such file or directory)
[2013-12-10 08:37:58.590331] D [dht-layout.c:626:dht_layout_normalize] 0-testvolume-dht: directory /path/to/some/dir/file.xml looked up first time
[2013-12-10 08:37:58.590382] D [dht-layout.c:646:dht_layout_normalize] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x51e) [0xec172e] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_cbk+0x282) [0x3a30f2] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_complete+0x364) [0x39c3a4]))) 0-testvolume-dht: path=/path/to/some/dir/file.xml err=No such file or directory on subvol=testvolume-replicate-0
[2013-12-10 08:37:58.590393] D [dht-common.c:195:dht_discover_complete] 0-testvolume-dht: normalizing failed on /path/to/some/dir/file.xml (overlaps/holes present: yes, ENOENT errors: 0)
[2013-12-10 08:37:58.590641] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.591094] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory
[2013-12-10 08:37:58.591434] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory
[2013-12-10 08:37:58.591548] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.591846] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.592155] D [client-rpc-fops.c:1112:client3_3_getxattr_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory. Path: /path/to/some/dir/file.xml (b5a719ba-9dc4-4f00-be24-28c03876b9dc). Key: trusted.glusterfs.dht.linkto
[2013-12-10 08:37:58.592288] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-testvolume-replicate-0: /path/to/some/dir/file.xml: failed to get the gfid from dict
[2013-12-10 08:37:58.592677] D [dht-common.c:267:dht_discover_cbk] 0-testvolume-dht: lookup of /path/to/some/dir/file.xml on testvolume-replicate-0 returned error (No such file or directory)
[2013-12-10 08:37:58.592721] D [dht-layout.c:626:dht_layout_normalize] 0-testvolume-dht: directory /path/to/some/dir/file.xml looked up first time
[2013-12-10 08:37:58.592769] D [dht-layout.c:646:dht_layout_normalize] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x51e) [0xec172e] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_cbk+0x282) [0x3a30f2] (-->/usr/lib/glusterfs/3.4.1/xlator/cluster/distribute.so(dht_discover_complete+0x364) [0x39c3a4]))) 0-testvolume-dht: path=/path/to/some/dir/file.xml err=No such file or directory on subvol=testvolume-replicate-0
[2013-12-10 08:37:58.592779] D [dht-common.c:195:dht_discover_complete] 0-testvolume-dht: normalizing failed on /path/to/some/dir/file.xml (overlaps/holes present: yes, ENOENT errors: 0)
[2013-12-10 08:37:58.593128] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
[2013-12-10 08:37:58.593461] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-0: remote operation failed: No such file or directory
[2013-12-10 08:37:58.593760] W [client-rpc-fops.c:526:client3_3_stat_cbk] 0-testvolume-client-1: remote operation failed: No such file or directory
[2013-12-10 08:37:58.593839] D [afr-common.c:745:afr_get_call_child] 0-testvolume-replicate-0: Returning 0, call_child: 0, last_index: -1
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
