question about Transport endpoint is not connected

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I found a log as below in the client log.
...
[2016-10-31 04:05:05.762520] W [socket.c:588:__socket_rwv] 0-c_glusterfs-client-10: readv on 10.32.0.48:49309 failed (No data available)
[2016-10-31 04:05:05.762621] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-c_glusterfs-client-10: disconnected from c_glusterfs-client-10. Client process will keep trying to connect to glusterd until brick's port is available
...

It seems that the disconnection happen.
I did not find the code which trying to reconnect the server.
Can anyone tell me where is the point in that the client keep trying to connect to server when disconnection happened.

Thanks,
Xin 


Hi all,
I have a question to consult you.

Gluterfs version is 3.7.6.
I create a replicate volume using two node.

When I access the mount point, it show that "Transport endpoint is not connected" in client log.
From log it seems that the glusterfs and glusterfsd disconnect.But I don't why disconnection happen on local sockect.


Below is the log from client and service.

mnt-c.log(client log) 

[2016-10-31 04:04:08.023782] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --acl --volfile-server=10.32.1.144 --volfile-id=c_glusterfs /mnt/c)
[2016-10-31 04:04:08.042622] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-10-31 04:04:08.052641] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-md-cache: adding option 'cache-posix-acl' for volume 'c_glusterfs-md-cache' with value 'true'
[2016-10-31 04:04:08.058854] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-10-31 04:04:08.060066] I [MSGID: 114020] [client.c:2118:notify] 0-c_glusterfs-client-9: parent translators are ready, attempting connect on transport
[2016-10-31 04:04:08.061169] I [MSGID: 114020] [client.c:2118:notify] 0-c_glusterfs-client-10: parent translators are ready, attempting connect on transport
[2016-10-31 04:04:08.061694] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume c_glusterfs-client-9
  2:     type protocol/client
  3:     option ping-timeout 4
  4:     option remote-host 10.32.1.144
  5:     option remote-subvolume /opt/lvmdir/c2/brick
  6:     option transport-type socket
  7:     option username 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
  8:     option password ade893cc-19dd-4d67-ae8e-cb929ef4e770
  9:     option send-gids true
 10: end-volume
 11:  
 12: volume c_glusterfs-client-10
 13:     type protocol/client
 14:     option ping-timeout 4
 15:     option remote-host 10.32.0.48
 16:     option remote-subvolume /opt/lvmdir/c2/brick
 17:     option transport-type socket
 18:     option username 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
 19:     option password ade893cc-19dd-4d67-ae8e-cb929ef4e770
 20:     option send-gids true
 21: end-volume
 22:  
 23: volume c_glusterfs-replicate-0
 24:     type cluster/replicate
 25:     subvolumes c_glusterfs-client-9 c_glusterfs-client-10
 26: end-volume
 27:  
 28: volume c_glusterfs-dht
 29:     type cluster/distribute
 30:     subvolumes c_glusterfs-replicate-0
 31: end-volume
 32:  
 33: volume c_glusterfs-write-behind
 34:     type performance/write-behind
 35:     subvolumes c_glusterfs-dht
 36: end-volume
 37:  
 38: volume c_glusterfs-read-ahead
 39:     type performance/read-ahead
 40:     subvolumes c_glusterfs-write-behind
 41: end-volume
 42:  
 43: volume c_glusterfs-readdir-ahead
 44:     type performance/readdir-ahead
 45:     subvolumes c_glusterfs-read-ahead
 46: end-volume
 47:  
 48: volume c_glusterfs-io-cache
 49:     type performance/io-cache
 50:     subvolumes c_glusterfs-readdir-ahead
 51: end-volume
 52:  
 53: volume c_glusterfs-quick-read
 54:     type performance/quick-read
 55:     subvolumes c_glusterfs-io-cache
 56: end-volume
 57:  
 58: volume c_glusterfs-open-behind
 59:     type performance/open-behind
 60:     subvolumes c_glusterfs-quick-read
 61: end-volume
 62:  
 63: volume c_glusterfs-md-cache
 64:     type performance/md-cache
 65:     option cache-posix-acl true
 66:     subvolumes c_glusterfs-open-behind
 67: end-volume
 68:  
 69: volume c_glusterfs
 70:     type debug/io-stats
 71:     option latency-measurement off
 72:     option count-fop-hits off
 73:     subvolumes c_glusterfs-md-cache
 74: end-volume
 75:  
 76: volume posix-acl-autoload
 77:     type system/posix-acl
 78:     subvolumes c_glusterfs
 79: end-volume
 80:  
 81: volume meta-autoload
 82:     type meta
 83:     subvolumes posix-acl-autoload
 84: end-volume
 85:  
+------------------------------------------------------------------------------+
[2016-10-31 04:04:08.062941] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:04:08.063367] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-10: changing port to 49309 (from 0)
[2016-10-31 04:04:08.063543] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:04:08.063576] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:04:08.063716] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:04:08.063785] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1
[2016-10-31 04:04:08.064659] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:04:08.065344] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-10: Connected to c_glusterfs-client-10, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:04:08.065374] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-10: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:04:08.074629] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-10-31 04:04:08.074884] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-10: Server lk version = 1
[2016-10-31 04:04:08.075006] I [fuse-bridge.c:4030:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-10-31 04:04:08.077023] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-c_glusterfs-replicate-0: selecting local read_child c_glusterfs-client-9

[2016-10-31 04:04:15.908933] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] 
                                                      -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] 
                                                      -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] 
                                                      ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]


[2016-10-31 04:04:15.909054] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.913787] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.913885] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.923030] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.923135] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.944227] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.944343] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.949661] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.949766] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.954882] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.954982] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.960768] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
...
[2016-10-31 04:04:16.107262] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 199e1862-7e74-4b13-8c42-c4cea7e9f2fc
[2016-10-31 04:04:16.111886] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 199e1862-7e74-4b13-8c42-c4cea7e9f2fc. source=0 sinks=1 
[2016-10-31 04:04:16.114518] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.114620] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.122321] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 014e2eca-bb61-4939-9a4a-9a1af4351178
[2016-10-31 04:04:16.126960] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 014e2eca-bb61-4939-9a4a-9a1af4351178. source=0 sinks=1 
[2016-10-31 04:04:16.128101] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.128202] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.134165] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 60b5380b-0cea-423d-a1af-d3064968cc54
[2016-10-31 04:04:16.139036] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 60b5380b-0cea-423d-a1af-d3064968cc54. source=0 sinks=1 
[2016-10-31 04:04:16.141317] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.141432] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.147456] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on accef130-4cf5-4171-be08-2089e7ba1ca5
[2016-10-31 04:04:16.152572] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on accef130-4cf5-4171-be08-2089e7ba1ca5. source=0 sinks=1 
[2016-10-31 04:04:16.153633] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.153744] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.159954] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 9c666f31-f3d7-40c6-b17b-0f41768ddea4
[2016-10-31 04:04:16.164773] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 9c666f31-f3d7-40c6-b17b-0f41768ddea4. source=0 sinks=1 
[2016-10-31 04:04:16.167219] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.167323] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
...


[2016-10-31 04:06:03.627345] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] 
                                                                     ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) 
                                                                    op(FINODELK(30)) called at 2016-10-31 04:06:03.626033 (xid=0x7f5e)





[2016-10-31 04:06:03.627395] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]



[2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 0)



[2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (c_glusterfs-client-9)




[2016-10-31 04:06:03.628466] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]



[2016-10-31 04:06:03.628475] I [MSGID: 108019] [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: unable to lock on even one child
[2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed.


[2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not connected)




[2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] 
                                                                      ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) 
                                                                     op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a)

[2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)

[2016-10-31 04:06:03.629210] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]


[2016-10-31 04:06:03.629266] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 255)


[2016-10-31 04:06:03.629277] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (gfid = b0e5a94e-a432-4dce-b86f-a551555780a2). Holes=1 overlaps=0


[2016-10-31 04:06:03.629293] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f62 Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (c_glusterfs-client-9)



[2016-10-31 04:06:03.629333] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: b0e5a94e-a432-4dce-b86f-a551555780a2: failed to resolve (Transport endpoint is not connected)


[2016-10-31 04:06:03.629363] W [fuse-bridge.c:3385:fuse_getxattr_resume] 0-glusterfs-fuse: 20784: GETXATTR b0e5a94e-a432-4dce-b86f-a551555780a2/70366685109140 (system.posix_acl_access) resolution failed



[2016-10-31 04:06:03.629729] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 
                                                                     0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) 
                                                                     op(LOOKUP(27)) called at 2016-10-31 04:06:03.624568 (xid=0x7f5b)





[2016-10-31 04:06:03.629779] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /pmd/2 (fa866805-ab22-4afe-8acb-528d15ce4117) [Transport endpoint is not connected]


[2016-10-31 04:06:03.629784] E [MSGID: 114031] [client-rpc-fops.c:2883:client3_3_opendir_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]



[2016-10-31 04:06:03.629836] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2 (gfid = fa866805-ab22-4afe-8acb-528d15ce4117). Holes=1 overlaps=0



[2016-10-31 04:06:03.629857] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20794: OPENDIR() /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews => -1 (Transport endpoint is not connected)



[2016-10-31 04:06:03.629896] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: fa866805-ab22-4afe-8acb-528d15ce4117: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.629990] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20785: GETATTR 70366685017568 (fa866805-ab22-4afe-8acb-528d15ce4117) resolution failed
[2016-10-31 04:06:03.630164] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/protocol/client.so(client_local_wipe-0x42d64) [0x3fff86c9bc5c] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk
[2016-10-31 04:06:03.630367] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.625675 (xid=0x7f5c)
[2016-10-31 04:06:03.630368] W [MSGID: 114031] [client-rpc-fops.c:1569:client3_3_fstat_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.630497] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) [Transport endpoint is not connected]
[2016-10-31 04:06:03.630871] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:06:03.631488] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:06:03.631523] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:06:03.631555] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-c_glusterfs-client-9: 6 fds open - Delaying child_up until they are re-opened
[2016-10-31 04:06:03.631475] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/ethmac.cfg (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]




[2016-10-31 04:06:03.632256] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 1)
[2016-10-31 04:06:03.632728] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:257add5a-9b87-4014-8a3a-09dd0a699eec> succeeded (fd = 2)
[2016-10-31 04:06:03.632790] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:6fd47fe9-a566-47c7-8df7-e7a5e379101f> succeeded (fd = 3)
[2016-10-31 04:06:03.632847] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:fa866805-ab22-4afe-8acb-528d15ce4117> succeeded (fd = 4)
[2016-10-31 04:06:03.632905] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:139f5034-5b42-4fb8-9372-6dd7cac1e674> succeeded (fd = 5)





[2016-10-31 04:06:03.632935] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-c_glusterfs-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-10-31 04:06:03.633028] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:06:03.633386] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1



[2016-10-31 04:06:03.634579] W [fuse-bridge.c:758:fuse_attr_cbk] 0-glusterfs-fuse: 20795: FSTAT() /configuration/oamrdncy.cfg => -1 (Transport endpoint is not connected)



[2016-10-31 04:06:03.634889] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(XATTROP(33)) called at 2016-10-31 04:06:03.625716 (xid=0x7f5d)



[2016-10-31 04:06:03.634950] W [MSGID: 114031] [client-rpc-fops.c:1845:client3_3_xattrop_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f)



[2016-10-31 04:06:03.635134] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-c_glusterfs-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid 200b0bb2-d903-4924-b025-262d805c0f2f. (Possible split-brain)


[2016-10-31 04:06:03.635364] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-c_glusterfs-client-9: disconnected from c_glusterfs-client-9. Client process will keep trying to connect to glusterd until brick's port is available



[2016-10-31 04:06:03.635420] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-c_glusterfs-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.





[2016-10-31 04:06:03.635762] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.635812] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/15 (gfid = 52a8bce8-00ac-48a7-8abf-9a3fdf9a498b). Holes=1 overlaps=0
[2016-10-31 04:06:03.635858] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: 52a8bce8-00ac-48a7-8abf-9a3fdf9a498b: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.635885] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20802: GETATTR 70366685021780 (52a8bce8-00ac-48a7-8abf-9a3fdf9a498b) resolution failed
[2016-10-31 04:06:03.638853] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.639020] E [MSGID: 108008] [afr-read-txn.c:89:afr_read_txn_refresh_done] 0-c_glusterfs-replicate-0: Failing FSTAT on gfid 200b0bb2-d903-4924-b025-262d805c0f2f: split-brain observed. [Input/output error]
[2016-10-31 04:06:03.640564] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641160] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20808: LOOKUP() /license => -1 (Transport endpoint is not connected)
The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up" repeated 6 times between [2016-10-31 04:06:03.640564] and [2016-10-31 04:06:03.641444]
[2016-10-31 04:06:03.641484] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Invalid argument]
[2016-10-31 04:06:03.641562] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641573] E [MSGID: 108010] [afr-lk-common.c:665:afr_unlock_inodelk_cbk] 0-c_glusterfs-replicate-0: path=/configuration/oamrdncy.cfg gfid=200b0bb2-d903-4924-b025-262d805c0f2f: unlock failed on subvolume c_glusterfs-client-9 with lock owner 6cd0c189ff3f0000 [Invalid argument]
[2016-10-31 04:06:03.641655] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up





[2016-10-31 04:06:03.641716] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20809: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.642387] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20813: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.647166] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20815: LOOKUP() /license => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.647647] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20816: LOOKUP() /tmp => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.648023] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20817: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653160] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20819: LOOKUP() /tmp => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653635] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20820: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653992] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20821: LOOKUP() /security => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.654333] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20822: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.660036] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20824: LOOKUP() /security => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.660675] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20825: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.661018] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20826: LOOKUP() /up => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.665422] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20828: LOOKUP() /up => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.665886] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20829: LOOKUP() /usr => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.666242] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20830: LOOKUP() /xb => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.666790] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20832: LOOKUP() /xb => -1 (Transport endpoint is not connected)




The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up" repeated 61 times between [2016-10-31 04:06:03.641655] and [2016-10-31 04:06:03.670371]
[2016-10-31 04:06:03.670392] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2/000301 (gfid = 7af60568-eace-40f1-a94f-5ba92a3afadf). Holes=1 overlaps=0
[2016-10-31 04:06:03.670499] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: 7af60568-eace-40f1-a94f-5ba92a3afadf: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.670529] W [fuse-bridge.c:2517:fuse_opendir_resume] 0-glusterfs-fuse: 20834: OPENDIR (7af60568-eace-40f1-a94f-5ba92a3afadf) resolution failed
[2016-10-31 04:06:03.675300] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.675553] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20835: LOOKUP() /usr => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.675990] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20836: LOOKUP() /xb => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.676348] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20837: LOOKUP() /node_id => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.680652] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20839: LOOKUP() /node_id => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.681150] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20842: OPENDIR() /configuration => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.681510] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/cluster/distribute.so(dht_local_wipe-0x70e58) [0x3fff86b88f58] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk

opt-lvmdir-c2-brick.log(server log)

[2016-10-31 04:03:41.554325] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.6 (args: /usr/sbin/glusterfsd -s 10.32.1.144 --volfile-id c_glusterfs.10.32.1.144.opt-lvmdir-c2-brick -p /system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid -S /var/run/gluster/697c0e4a16ebc734cd06fd9150723005.socket --brick-name /opt/lvmdir/c2/brick -l /var/log/glusterfs/bricks/opt-lvmdir-c2-brick.log --xlator-option *-posix.glusterd-uuid=5818a372-d517-41af-bd2f-0fce648773df --brick-port 49391 --xlator-option c_glusterfs-server.listen-port=49391)
[2016-10-31 04:03:41.571442] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-10-31 04:03:41.588390] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-server: adding option 'listen-port' for volume 'c_glusterfs-server' with value '49391'
[2016-10-31 04:03:41.588440] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-posix: adding option 'glusterd-uuid' for volume 'c_glusterfs-posix' with value '5818a372-d517-41af-bd2f-0fce648773df'
[2016-10-31 04:03:41.588815] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-10-31 04:03:41.588870] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/opt/lvmdir/c2/brick: skip format check for non-addr auth option auth.login./opt/lvmdir/c2/brick.allow
[2016-10-31 04:03:41.588907] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/opt/lvmdir/c2/brick: skip format check for non-addr auth option auth.login.5c5afe17-02ce-4b9b-aef4-e66efa1c6132.password
[2016-10-31 04:03:41.590577] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2016-10-31 04:03:41.590773] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-c_glusterfs-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2016-10-31 04:03:41.593929] I [trash.c:2366:init] 0-c_glusterfs-trash: no option specified for 'eliminate', using NULL
[2016-10-31 04:03:41.595494] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2016-10-31 04:03:41.595528] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-unix' is not recognized
[2016-10-31 04:03:41.595556] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-null' is not recognized
[2016-10-31 04:03:41.595619] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-quota: option 'timeout' is not recognized
[2016-10-31 04:03:41.595647] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-marker: option 'quota-version' is not recognized
[2016-10-31 04:03:41.595677] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-trash: option 'brick-path' is not recognized
[2016-10-31 04:03:41.605103] W [MSGID: 113036] [posix.c:2018:posix_rename] 0-c_glusterfs-posix: found directory at /opt/lvmdir/c2/brick/.trashcan/ while expecting ENOENT [File exists]
Final graph:
+------------------------------------------------------------------------------+
  1: volume c_glusterfs-posix
  2:     type storage/posix
  3:     option glusterd-uuid 5818a372-d517-41af-bd2f-0fce648773df
  4:     option directory /opt/lvmdir/c2/brick
  5:     option volume-id 0a50e4e5-ecfc-432c-88d9-2083455e4baf
  6: end-volume
  7:  
  8: volume c_glusterfs-trash
  9:     type features/trash
 10:     option trash-dir .trashcan
 11:     option brick-path /opt/lvmdir/c2/brick
 12:     option trash-internal-op off
 13:     subvolumes c_glusterfs-posix
 14: end-volume
 15:  
 16: volume c_glusterfs-changelog
 17:     type features/changelog
 18:     option changelog-brick /opt/lvmdir/c2/brick
 19:     option changelog-dir /opt/lvmdir/c2/brick/.glusterfs/changelogs
 20:     option changelog-barrier-timeout 120
 21:     subvolumes c_glusterfs-trash
 22: end-volume
 23:  
 24: volume c_glusterfs-bitrot-stub
 25:     type features/bitrot-stub
 26:     option export /opt/lvmdir/c2/brick
 27:     subvolumes c_glusterfs-changelog
 28: end-volume
 29:  
 30: volume c_glusterfs-access-control
 31:     type features/access-control
 32:     subvolumes c_glusterfs-bitrot-stub
 33: end-volume
 34:  
 35: volume c_glusterfs-locks
 36:     type features/locks
 37:     subvolumes c_glusterfs-access-control
 38: end-volume
 39:  
 40: volume c_glusterfs-upcall
 41:     type features/upcall
 42:     option cache-invalidation off
 43:     subvolumes c_glusterfs-locks
 44: end-volume
 45:  
 46: volume c_glusterfs-io-threads
 47:     type performance/io-threads
 48:     subvolumes c_glusterfs-upcall
 49: end-volume
 50:  
 51: volume c_glusterfs-marker
 52:     type features/marker
 53:     option volume-uuid 0a50e4e5-ecfc-432c-88d9-2083455e4baf
 54:     option timestamp-file /system/glusterd/vols/c_glusterfs/marker.tstamp
 55:     option quota-version 0
 56:     option xtime off
 57:     option gsync-force-xtime off
 58:     option quota off
 59:     option inode-quota off
 60:     subvolumes c_glusterfs-io-threads
 61: end-volume
 62:  
 63: volume c_glusterfs-barrier
 64:     type features/barrier
 65:     option barrier disable
 66:     option barrier-timeout 120
 67:     subvolumes c_glusterfs-marker
 68: end-volume
 69:  
 70: volume c_glusterfs-index
 71:     type features/index
 72:     option index-base /opt/lvmdir/c2/brick/.glusterfs/indices
 73:     subvolumes c_glusterfs-barrier
 74: end-volume
 75:  
 76: volume c_glusterfs-quota
 77:     type features/quota
 78:     option volume-uuid c_glusterfs
 79:     option server-quota off
 80:     option timeout 0
 81:     option deem-statfs off
 82:     subvolumes c_glusterfs-index
 83: end-volume
 84:  
 85: volume c_glusterfs-worm
 86:     type features/worm
 87:     option worm off
 88:     subvolumes c_glusterfs-quota
 89: end-volume
 90:  
 91: volume c_glusterfs-read-only
 92:     type features/read-only
 93:     option read-only off
 94:     subvolumes c_glusterfs-worm
 95: end-volume
 96:  
 97: volume /opt/lvmdir/c2/brick
 98:     type debug/io-stats
 99:     option latency-measurement off
100:     option count-fop-hits off
101:     subvolumes c_glusterfs-read-only
102: end-volume
103:  
104: volume c_glusterfs-server
105:     type protocol/server
106:     option transport.socket.listen-port 49391
107:     option rpc-auth.auth-glusterfs on
108:     option rpc-auth.auth-unix on
109:     option rpc-auth.auth-null on
110:     option rpc-auth-allow-insecure on
111:     option transport-type tcp
112:     option auth.login./opt/lvmdir/c2/brick.allow 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
113:     option auth.login.5c5afe17-02ce-4b9b-aef4-e66efa1c6132.password ade893cc-19dd-4d67-ae8e-cb929ef4e770
114:     option auth.addr./opt/lvmdir/c2/brick.allow *
115:     option transport.tcp-user-timeout 4
116:     subvolumes /opt/lvmdir/c2/brick
117: end-volume
118:  
+------------------------------------------------------------------------------+
[2016-10-31 04:03:43.191850] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:03:43.191927] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:03:45.556522] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:03:45.556581] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-1599-2016/10/31-04:03:41:530232-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:07.831958] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:07.832023] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:04:07.974625] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:07.974688] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:08.063317] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:08.063359] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:10.032591] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:04:10.032672] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:05:04.530186] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:04.530267] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 60a17763-95fd-4162-8402-9651dfbe9ec4 held by {client=0x3fff7c002990, pid=0 lk-owner=a87b0a7cff3f0000}
[2016-10-31 04:05:04.530306] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:05:04.530375] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760618] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760693] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:57.341892] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:05:57.341928] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1604-2016/10/31-04:05:53:637848-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.291380] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.291441] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.403916] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.403963] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3292-2016/10/31-04:05:57:685417-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:03.631296] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:03.631332] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:06:03.634762] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.634945] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 200b0bb2-d903-4924-b025-262d805c0f2f held by {client=0x3fff7c026520, pid=6232 lk-owner=6cd0c189ff3f0000}
[2016-10-31 04:06:03.634987] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /pmd/2
[2016-10-31 04:06:03.635035] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /configuration/oamrdncy.cfg
[2016-10-31 04:06:03.635349] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /
[2016-10-31 04:06:03.635414] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:06:03.635468] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001
[2016-10-31 04:06:03.635554] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl
[2016-10-31 04:06:03.635889] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001/cello
[2016-10-31 04:06:03.640329] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.641143] E [inodelk.c:309:__inode_unlock_lock] 0-c_glusterfs-locks:  Matching lock not found for unlock 0-9223372036854775807, by 6cd0c189ff3f0000 on 0x3fff840fd9a0
[2016-10-31 04:06:03.641200] E [MSGID: 115053] [server-rpc-fops.c:275:server_inodelk_cbk] 0-c_glusterfs-server: 32623: INODELK /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) ==> (Invalid argument) [Invalid argument]
[2016-10-31 04:06:06.270314] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0
[2016-10-31 04:06:06.270449] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0



Thanks,
Xin


 



 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux