On Wed, May 4, 2016 at 9:12 PM, Egidijus Ligeika <egidijus.ligeika@xxxxxxxx> wrote:
Hello,I have glusterfs client and server v3.7.11.Th servers are running inside docker on debian image, on a centos host.I am using the fuse client mounting to a list of DNS a records.I use XFS as the underlying storage, inode size 512 and xfs is formatted with 512 inode size.I can see the client and the cluster is clean and happy, heals work. when writing tens of gigabytes to the cluster via the fuse client I see errors like this ON THE FUSE CLIENT /var/log/glusterfs/mountname:[2016-05-04 14:43:17.779936] W [rpc-clnt.c:1606:rpc_clnt_submit] 0-thevolume-client-1: failed to submit rpc-request (XID: 0xe75f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (thevolume-client-1) [2016-05-04 14:43:17.779958] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-thevolume-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2016-05-04 14:43:17.780382] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-thevolume-client-2: remote operation failed [Transport endpoint is not connected] [2016-05-04 14:43:17.780525] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f03bf304ae2] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f03bf0cf90e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f03bf0cfa1e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f03bf0d140a] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f03bf0d1c38] ))))) 0-thevolume-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-05-04 14:42:47.766130 (xid=0xe75df) [2016-05-04 14:43:17.780541] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-thevolume-client-1: socket disconnected [2016-05-04 14:43:17.780645] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f03bf304ae2] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f03bf0cf90e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f03bf0cfa1e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f03bf0d140a] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f03bf0d1c38] ))))) 0-thevolume-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-05-04 14:42:55.772720 (xid=0xe75e0) [2016-05-04 14:43:17.780675] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-thevolume-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2016-05-04 14:43:17.780880] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f03bf304ae2] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f03bf0cf90e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f03bf0cfa1e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f03bf0d140a] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f03bf0d1c38] ))))) 0-thevolume-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-05-04 14:43:12.445399 (xid=0xe75e1) [2016-05-04 14:43:17.780898] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-thevolume-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2016-05-04 14:43:17.780963] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-thevolume-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2016-05-04 14:43:17.781039] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-thevolume-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2016-05-04 14:43:17.781088] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover_cbk+0x3fc) [0x7f03b7965edc] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.781843] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.781937] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.784540] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_lookup+0x199) [0x7f03b795e1c9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.784576] W [fuse-bridge.c:467:fuse_entry_cbk] 0-glusterfs-fuse: 630384: LOOKUP() /magento/catalog/product/e/m/emmett_3seater_sierra_blue_lb5.jpg => -1 (Transport endpoint is not connected) [2016-05-04 14:43:17.789047] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_lookup+0x199) [0x7f03b795e1c9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.789080] W [fuse-bridge.c:467:fuse_entry_cbk] 0-glusterfs-fuse: 630386: LOOKUP() /magento/catalog/product/e/m/emmett_3seater_sierra_blue_lb5_1.jpg => -1 (Transport endpoint is not connected) [2016-05-04 14:43:17.792013] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_lookup+0x199) [0x7f03b795e1c9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.792047] W [fuse-bridge.c:467:fuse_entry_cbk] 0-glusterfs-fuse: 630388: LOOKUP() /magento/catalog/product/e/m/emmett_3seater_sierra_blue_lb6.jpg => -1 (Transport endpoint is not connected) [2016-05-04 14:43:17.792272] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-thevolume-client-1: remote operation failed [Transport endpoint is not connected] [2016-05-04 14:43:17.794634] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_lookup+0x199) [0x7f03b795e1c9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.794664] W [fuse-bridge.c:467:fuse_entry_cbk] 0-glusterfs-fuse: 630390: LOOKUP() /magento/catalog/product/e/m/emmett_3seater_sierra_blue_lb6_1.jpg => -1 (Transport endpoint is not connected) [2016-05-04 14:43:17.796674] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.796770] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.796923] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.799405] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.799580] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_lookup+0x199) [0x7f03b795e1c9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.799604] W [fuse-bridge.c:467:fuse_entry_cbk] 0-glusterfs-fuse: 630396: LOOKUP() /magento/catalog/product/e/m/emmett_3seater_sierra_blue_lb6_2.jpg => -1 (Transport endpoint is not connected) [2016-05-04 14:43:17.801365] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.802111] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:17.818490] E [dht-helper.c:1597:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.7.11/xlator/cluster/replicate.so(afr_discover+0x14a) [0x7f03b795db3a] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x359) [0x7f03b76dc0f9] -->/usr/lib64/glusterfs/3.7.11/xlator/cluster/distribute.so(dht_inode_ctx_time_update+0x210) [0x7f03b76b8b20] ) 0-thevolume-dht: invalid argument: inode [Invalid argument] [2016-05-04 14:43:19.770865] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-thevolume-client-2: remote operation failed [Transport endpoint is not connected] [2016-05-04 14:43:19.770930] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-thevolume-client-2: remote operation failed [Transport endpoint is not connected] [2016-05-04 14:43:19.771811] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-thevolume-client-1: remote operation failed [Invalid argument] [2016-05-04 14:43:19.771852] E [MSGID: 108010] [afr-lk-common.c:665:afr_unlock_inodelk_cbk] 0-thevolume-replicate-0: path=(null) gfid=00000000-0000-0000-0000-000000000000: unlock failed on subvolume thevolume-client-1 with lock owner d03a00a8037f0000 [Invalid argument] [2016-05-04 14:43:19.771878] W [fuse-bridge.c:1287:fuse_err_cbk] 0-glusterfs-fuse: 630361: FLUSH() ERR => -1 (Transport endpoint is not connected) [2016-05-04 14:43:19.771937] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-thevolume-client-1: remote operation failed [Invalid argument] [2016-05-04 14:43:19.771959] E [MSGID: 108010] [afr-lk-common.c:665:afr_unlock_inodelk_cbk] 0-thevolume-replicate-0: path=(null) gfid=00000000-0000-0000-0000-000000000000: unlock failed on subvolume thevolume-client-1 with lock owner b43800a8037f0000 [Invalid argument] [2016-05-04 14:43:19.771979] W [fuse-bridge.c:1287:fuse_err_cbk] 0-glusterfs-fuse: 630375: FLUSH() ERR => -1 (Transport endpoint is not connected) The message "W [MSGID: 114031] [client-rpc-fops.c:1917:client3_3_fxattrop_cbk] 0-thevolume-client-2: remote operation failed" repeated 19 times between [2016-05-04 14:43:15.770856] and [2016-05-04 14:43:15.773223]Error: Parse error on line 1: [2016-05-04 14:43:17.7799 -----^ Expecting 'EOF', '}', ',', ']'The mounted volume becomes read only on the client only. The "gluster" volume is still writeable on the gluster servers.
gluster volume status does not report anything funky.If I kill the glusterfs process on the client (the box that has gluster volume mounted via fuse), and then MOUNT again, I can carry on copying and writing, until I get those errors again.I have successfully copied the same amount of data directly to the XFS volume on the glusterfs server nodes, I believe my XFS works.All the machines are on AWS, and none of the resources are exhausted IO/RAM/CPU/NETWORK, not on client, not on gluster cluster.Please help!
Posting the `gluster vol info` output for the volume will help everyone get a better picture about your volume (sanitize it to remove any sensitive information first).
Have you enabled any options on the volume, particularly any quorum options? Client-quorum makes a client read-only when it cannot connect to a quorum of servers (normally 50% of replica count +1 ).
You mount log shows that you've had connections issue (a lot of 'Transport endpoint is not connected' messages).
Can you verify that the network between the client and server isn't having problems when you see these errors?
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
