Hi everybody,
we are experiencing random problems accessing some files on a volume via NFS. Any help is highly
appreciated.
The setup:
CentOS release 6.5
glusterfs-libs-3.4.7-1.el6.x86_64
glusterfs-fuse-3.4.7-1.el6.x86_64
glusterfs-server-3.4.7-1.el6.x86_64
glusterfs-3.4.7-1.el6.x86_64
glusterfs-cli-3.4.7-1.el6.x86_64
Volume Name: repsilo
Type: Distributed-Replicate
Volume ID: 94d8f999-bf1e-4f99-a96f-5bb6564222d9
Status: Started
Number of Bricks: 10 x 2 = 20
Transport-type: tcp
Bricks:
Brick1: gls07.srv:/mnt/glusterRawL
Brick2: gls08.srv:/mnt/glusterRawL
Brick3: gls07.srv:/mnt/glusterRawR
Brick4: gls08.srv:/mnt/glusterRawR
Brick5: gls14.srv:/mnt/glusterRawL
Brick6: gls15.srv:/mnt/glusterRawL
Brick7: gls14.srv:/mnt/glusterRawR
Brick8: gls15.srv:/mnt/glusterRawR
Brick9: gls12.srv:/mnt/glusterRawL
Brick10: gls13.srv:/mnt/glusterRawL
Brick11: gls12.srv:/mnt/glusterRawR
Brick12: gls13.srv:/mnt/glusterRawR
Brick13: gls01.srv:/mnt/glusterRawL
Brick14: gls02.srv:/mnt/glusterRawL
Brick15: gls01.srv:/mnt/glusterRawR
Brick16: gls02.srv:/mnt/glusterRawR
Brick17: gls03.srv:/mnt/glusterRawL
Brick18: gls04.srv:/mnt/glusterRawL
Brick19: gls03.srv:/mnt/glusterRawR
Brick20: gls04.srv:/mnt/glusterRawR
While the files are readable through the native client, trying to read them via NFS fails on some
servers. This is an example - relevant content of nfs.log on the gls04 server:
[2015-09-02 08:54:51.916981] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-18:
remote operation
failed: Permission denied. Path:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackReq
uest.1.h5 (e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.917051] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-19:
remote operation
failed: Permission denied. Path:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5
(e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.917109] E [dht-helper.c:1259:dht_inode_ctx_get]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d)
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e)
[0x7fbfd837be6e]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34)
[0x7fbfd837de84]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.917140] E [dht-helper.c:1278:dht_inode_ctx_set]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d)
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e)
[0x7fbfd837be6e]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52)
[0x7fbfd837dea2]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.918177] W [nfs3.c:726:nfs3svc_getattr_stat_cbk] 0-nfs: a4fcea5f:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1
(Invalid argument)
[2015-09-02 08:54:51.918225] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a4fcea5f,
GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2015-09-02 08:54:51.920121] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-19:
remote operation failed: Permission denied. Path:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5
(e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.920208] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-18:
remote operation failed: Permission denied. Path:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5
(e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12)
[2015-09-02 08:54:51.920267] E [dht-helper.c:1259:dht_inode_ctx_get]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d)
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e)
[0x7fbfd837be6e]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34)
[0x7fbfd837de84]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.920298] E [dht-helper.c:1278:dht_inode_ctx_set]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d)
[0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e)
[0x7fbfd837be6e]
(-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52)
[0x7fbfd837dea2]))) 0-repsilo-dht: invalid argument: inode
[2015-09-02 08:54:51.920679] W [nfs3.c:1512:nfs3svc_access_cbk] 0-nfs: a6fcea5f:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1
(Invalid argument)
[2015-09-02 08:54:51.920703] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a6fcea5f,
ACCESS: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2015-09-02 08:54:51.921374] W [nfs3.c:1512:nfs3svc_access_cbk] 0-nfs: a8fcea5f:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1
(Invalid argument)
[2015-09-02 08:54:51.921391] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a8fcea5f,
ACCESS: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
[2015-09-02 08:55:07.041430] W [nfs3.c:726:nfs3svc_getattr_stat_cbk] 0-nfs: a9fcea5f:
/CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1
(Invalid argument)
[2015-09-02 08:55:07.041493] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a9fcea5f,
GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument)
gluster volume heal repsilo info heal-failed
and
gluster volume heal repsilo info split-brain
both give Number of entries: 0 on all bricks
Thanks.
Regards,
Jan
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
