Hi all,
I have a question to consult you.
Gluterfs version is 3.7.6.
I create a replicate volume using two node.
When I access the mount point, it show that "Transport endpoint is not connected" in client log.
From log it seems that the glusterfs and glusterfsd disconnect.But I don't why disconnection happen on local sockect.
Below is the log from client and service.
mnt-c.log(client log)
[2016-10-31 04:04:08.023782] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --acl --volfile-server=10.32.1.144 --volfile-id=c_glusterfs /mnt/c)
[2016-10-31 04:04:08.042622] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-10-31 04:04:08.052641] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-md-cache: adding option 'cache-posix-acl' for volume 'c_glusterfs-md-cache' with value 'true'
[2016-10-31 04:04:08.058854] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-10-31 04:04:08.060066] I [MSGID: 114020] [client.c:2118:notify] 0-c_glusterfs-client-9: parent translators are ready, attempting connect on transport
[2016-10-31 04:04:08.061169] I [MSGID: 114020] [client.c:2118:notify] 0-c_glusterfs-client-10: parent translators are ready, attempting connect on transport
[2016-10-31 04:04:08.061694] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)
Final graph:
+------------------------------------------------------------------------------+
1: volume c_glusterfs-client-9
2: type protocol/client
3: option ping-timeout 4
4: option remote-host 10.32.1.144
5: option remote-subvolume /opt/lvmdir/c2/brick
6: option transport-type socket
7: option username 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
8: option password ade893cc-19dd-4d67-ae8e-cb929ef4e770
9: option send-gids true
10: end-volume
11:
12: volume c_glusterfs-client-10
13: type protocol/client
14: option ping-timeout 4
15: option remote-host 10.32.0.48
16: option remote-subvolume /opt/lvmdir/c2/brick
17: option transport-type socket
18: option username 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
19: option password ade893cc-19dd-4d67-ae8e-cb929ef4e770
20: option send-gids true
21: end-volume
22:
23: volume c_glusterfs-replicate-0
24: type cluster/replicate
25: subvolumes c_glusterfs-client-9 c_glusterfs-client-10
26: end-volume
27:
28: volume c_glusterfs-dht
29: type cluster/distribute
30: subvolumes c_glusterfs-replicate-0
31: end-volume
32:
33: volume c_glusterfs-write-behind
34: type performance/write-behind
35: subvolumes c_glusterfs-dht
36: end-volume
37:
38: volume c_glusterfs-read-ahead
39: type performance/read-ahead
40: subvolumes c_glusterfs-write-behind
41: end-volume
42:
43: volume c_glusterfs-readdir-ahead
44: type performance/readdir-ahead
45: subvolumes c_glusterfs-read-ahead
46: end-volume
47:
48: volume c_glusterfs-io-cache
49: type performance/io-cache
50: subvolumes c_glusterfs-readdir-ahead
51: end-volume
52:
53: volume c_glusterfs-quick-read
54: type performance/quick-read
55: subvolumes c_glusterfs-io-cache
56: end-volume
57:
58: volume c_glusterfs-open-behind
59: type performance/open-behind
60: subvolumes c_glusterfs-quick-read
61: end-volume
62:
63: volume c_glusterfs-md-cache
64: type performance/md-cache
65: option cache-posix-acl true
66: subvolumes c_glusterfs-open-behind
67: end-volume
68:
69: volume c_glusterfs
70: type debug/io-stats
71: option latency-measurement off
72: option count-fop-hits off
73: subvolumes c_glusterfs-md-cache
74: end-volume
75:
76: volume posix-acl-autoload
77: type system/posix-acl
78: subvolumes c_glusterfs
79: end-volume
80:
81: volume meta-autoload
82: type meta
83: subvolumes posix-acl-autoload
84: end-volume
85:
+------------------------------------------------------------------------------+
[2016-10-31 04:04:08.062941] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:04:08.063367] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-10: changing port to 49309 (from 0)
[2016-10-31 04:04:08.063543] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:04:08.063576] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:04:08.063716] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:04:08.063785] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1
[2016-10-31 04:04:08.064659] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:04:08.065344] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-10: Connected to c_glusterfs-client-10, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:04:08.065374] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-10: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:04:08.074629] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-10-31 04:04:08.074884] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-10: Server lk version = 1
[2016-10-31 04:04:08.075006] I [fuse-bridge.c:4030:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-10-31 04:04:08.077023] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-c_glusterfs-replicate-0: selecting local read_child c_glusterfs-client-9
[2016-10-31 04:04:15.908933] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574]
-->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8]
-->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c]
) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.909054] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.913787] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.913885] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.923030] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.923135] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.944227] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.944343] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.949661] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.949766] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.954882] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.954982] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.960768] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
...
[2016-10-31 04:04:16.107262] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 199e1862-7e74-4b13-8c42-c4cea7e9f2fc
[2016-10-31 04:04:16.111886] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 199e1862-7e74-4b13-8c42-c4cea7e9f2fc. source=0 sinks=1
[2016-10-31 04:04:16.114518] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.114620] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.122321] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 014e2eca-bb61-4939-9a4a-9a1af4351178
[2016-10-31 04:04:16.126960] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 014e2eca-bb61-4939-9a4a-9a1af4351178. source=0 sinks=1
[2016-10-31 04:04:16.128101] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.128202] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.134165] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 60b5380b-0cea-423d-a1af-d3064968cc54
[2016-10-31 04:04:16.139036] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 60b5380b-0cea-423d-a1af-d3064968cc54. source=0 sinks=1
[2016-10-31 04:04:16.141317] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.141432] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.147456] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on accef130-4cf5-4171-be08-2089e7ba1ca5
[2016-10-31 04:04:16.152572] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on accef130-4cf5-4171-be08-2089e7ba1ca5. source=0 sinks=1
[2016-10-31 04:04:16.153633] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.153744] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.159954] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 9c666f31-f3d7-40c6-b17b-0f41768ddea4
[2016-10-31 04:04:16.164773] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 9c666f31-f3d7-40c6-b17b-0f41768ddea4. source=0 sinks=1
[2016-10-31 04:04:16.167219] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.167323] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
...
[2016-10-31 04:06:03.627345] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808]
))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2016-10-31 04:06:03.626033 (xid=0x7f5e)
[2016-10-31 04:06:03.627395] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 0)
[2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (c_glusterfs-client-9)
[2016-10-31 04:06:03.628466] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.628475] I [MSGID: 108019] [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: unable to lock on even one child
[2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed.
[2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808]
))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a)
[2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)
[2016-10-31 04:06:03.629210] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]
[2016-10-31 04:06:03.629266] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 255)
[2016-10-31 04:06:03.629277] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (gfid = b0e5a94e-a432-4dce-b86f-a551555780a2). Holes=1 overlaps=0
[2016-10-31 04:06:03.629293] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f62 Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (c_glusterfs-client-9)
[2016-10-31 04:06:03.629333] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: b0e5a94e-a432-4dce-b86f-a551555780a2: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.629363] W [fuse-bridge.c:3385:fuse_getxattr_resume] 0-glusterfs-fuse: 20784: GETXATTR b0e5a94e-a432-4dce-b86f-a551555780a2/70366685109140 (system.posix_acl_access) resolution failed
[2016-10-31 04:06:03.629729] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] )))))
0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2016-10-31 04:06:03.624568 (xid=0x7f5b)
[2016-10-31 04:06:03.629779] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /pmd/2 (fa866805-ab22-4afe-8acb-528d15ce4117) [Transport endpoint is not connected]
[2016-10-31 04:06:03.629784] E [MSGID: 114031] [client-rpc-fops.c:2883:client3_3_opendir_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]
[2016-10-31 04:06:03.629836] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2 (gfid = fa866805-ab22-4afe-8acb-528d15ce4117). Holes=1 overlaps=0
[2016-10-31 04:06:03.629857] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20794: OPENDIR() /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.629896] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: fa866805-ab22-4afe-8acb-528d15ce4117: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.629990] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20785: GETATTR 70366685017568 (fa866805-ab22-4afe-8acb-528d15ce4117) resolution failed
[2016-10-31 04:06:03.630164] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/protocol/client.so(client_local_wipe-0x42d64) [0x3fff86c9bc5c] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk
[2016-10-31 04:06:03.630367] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.625675 (xid=0x7f5c)
[2016-10-31 04:06:03.630368] W [MSGID: 114031] [client-rpc-fops.c:1569:client3_3_fstat_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.630497] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) [Transport endpoint is not connected]
[2016-10-31 04:06:03.630871] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:06:03.631488] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:06:03.631523] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:06:03.631555] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-c_glusterfs-client-9: 6 fds open - Delaying child_up until they are re-opened
[2016-10-31 04:06:03.631475] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/ethmac.cfg (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2016-10-31 04:06:03.632256] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 1)
[2016-10-31 04:06:03.632728] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:257add5a-9b87-4014-8a3a-09dd0a699eec> succeeded (fd = 2)
[2016-10-31 04:06:03.632790] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:6fd47fe9-a566-47c7-8df7-e7a5e379101f> succeeded (fd = 3)
[2016-10-31 04:06:03.632847] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:fa866805-ab22-4afe-8acb-528d15ce4117> succeeded (fd = 4)
[2016-10-31 04:06:03.632905] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:139f5034-5b42-4fb8-9372-6dd7cac1e674> succeeded (fd = 5)
[2016-10-31 04:06:03.632935] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-c_glusterfs-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-10-31 04:06:03.633028] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:06:03.633386] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1
[2016-10-31 04:06:03.634579] W [fuse-bridge.c:758:fuse_attr_cbk] 0-glusterfs-fuse: 20795: FSTAT() /configuration/oamrdncy.cfg => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.634889] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(XATTROP(33)) called at 2016-10-31 04:06:03.625716 (xid=0x7f5d)
[2016-10-31 04:06:03.634950] W [MSGID: 114031] [client-rpc-fops.c:1845:client3_3_xattrop_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f)
[2016-10-31 04:06:03.635134] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-c_glusterfs-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid 200b0bb2-d903-4924-b025-262d805c0f2f. (Possible split-brain)
[2016-10-31 04:06:03.635364] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-c_glusterfs-client-9: disconnected from c_glusterfs-client-9. Client process will keep trying to connect to glusterd until brick's port is available
[2016-10-31 04:06:03.635420] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-c_glusterfs-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2016-10-31 04:06:03.635762] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.635812] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/15 (gfid = 52a8bce8-00ac-48a7-8abf-9a3fdf9a498b). Holes=1 overlaps=0
[2016-10-31 04:06:03.635858] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: 52a8bce8-00ac-48a7-8abf-9a3fdf9a498b: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.635885] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20802: GETATTR 70366685021780 (52a8bce8-00ac-48a7-8abf-9a3fdf9a498b) resolution failed
[2016-10-31 04:06:03.638853] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.639020] E [MSGID: 108008] [afr-read-txn.c:89:afr_read_txn_refresh_done] 0-c_glusterfs-replicate-0: Failing FSTAT on gfid 200b0bb2-d903-4924-b025-262d805c0f2f: split-brain observed. [Input/output error]
[2016-10-31 04:06:03.640564] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641160] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20808: LOOKUP() /license => -1 (Transport endpoint is not connected)
The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up" repeated 6 times between [2016-10-31 04:06:03.640564] and [2016-10-31 04:06:03.641444]
[2016-10-31 04:06:03.641484] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Invalid argument]
[2016-10-31 04:06:03.641562] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641573] E [MSGID: 108010] [afr-lk-common.c:665:afr_unlock_inodelk_cbk] 0-c_glusterfs-replicate-0: path=/configuration/oamrdncy.cfg gfid=200b0bb2-d903-4924-b025-262d805c0f2f: unlock failed on subvolume c_glusterfs-client-9 with lock owner 6cd0c189ff3f0000 [Invalid argument]
[2016-10-31 04:06:03.641655] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641716] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20809: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.642387] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20813: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.647166] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20815: LOOKUP() /license => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.647647] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20816: LOOKUP() /tmp => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.648023] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20817: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653160] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20819: LOOKUP() /tmp => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653635] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20820: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653992] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20821: LOOKUP() /security => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.654333] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20822: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.660036] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20824: LOOKUP() /security => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.660675] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20825: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.661018] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20826: LOOKUP() /up => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.665422] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20828: LOOKUP() /up => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.665886] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20829: LOOKUP() /usr => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.666242] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20830: LOOKUP() /xb => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.666790] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20832: LOOKUP() /xb => -1 (Transport endpoint is not connected)
The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up" repeated 61 times between [2016-10-31 04:06:03.641655] and [2016-10-31 04:06:03.670371]
[2016-10-31 04:06:03.670392] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2/000301 (gfid = 7af60568-eace-40f1-a94f-5ba92a3afadf). Holes=1 overlaps=0
[2016-10-31 04:06:03.670499] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: 7af60568-eace-40f1-a94f-5ba92a3afadf: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.670529] W [fuse-bridge.c:2517:fuse_opendir_resume] 0-glusterfs-fuse: 20834: OPENDIR (7af60568-eace-40f1-a94f-5ba92a3afadf) resolution failed
[2016-10-31 04:06:03.675300] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.675553] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20835: LOOKUP() /usr => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.675990] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20836: LOOKUP() /xb => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.676348] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20837: LOOKUP() /node_id => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.680652] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20839: LOOKUP() /node_id => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.681150] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20842: OPENDIR() /configuration => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.681510] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/cluster/distribute.so(dht_local_wipe-0x70e58) [0x3fff86b88f58] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk
opt-lvmdir-c2-brick.log(server log)
[2016-10-31 04:03:41.554325] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.6 (args: /usr/sbin/glusterfsd -s 10.32.1.144 --volfile-id c_glusterfs.10.32.1.144.opt-lvmdir-c2-brick -p /system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid -S /var/run/gluster/697c0e4a16ebc734cd06fd9150723005.socket --brick-name /opt/lvmdir/c2/brick -l /var/log/glusterfs/bricks/opt-lvmdir-c2-brick.log --xlator-option *-posix.glusterd-uuid=5818a372-d517-41af-bd2f-0fce648773df --brick-port 49391 --xlator-option c_glusterfs-server.listen-port=49391)
[2016-10-31 04:03:41.571442] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-10-31 04:03:41.588390] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-server: adding option 'listen-port' for volume 'c_glusterfs-server' with value '49391'
[2016-10-31 04:03:41.588440] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-posix: adding option 'glusterd-uuid' for volume 'c_glusterfs-posix' with value '5818a372-d517-41af-bd2f-0fce648773df'
[2016-10-31 04:03:41.588815] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-10-31 04:03:41.588870] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/opt/lvmdir/c2/brick: skip format check for non-addr auth option auth.login./opt/lvmdir/c2/brick.allow
[2016-10-31 04:03:41.588907] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/opt/lvmdir/c2/brick: skip format check for non-addr auth option auth.login.5c5afe17-02ce-4b9b-aef4-e66efa1c6132.password
[2016-10-31 04:03:41.590577] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2016-10-31 04:03:41.590773] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-c_glusterfs-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2016-10-31 04:03:41.593929] I [trash.c:2366:init] 0-c_glusterfs-trash: no option specified for 'eliminate', using NULL
[2016-10-31 04:03:41.595494] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2016-10-31 04:03:41.595528] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-unix' is not recognized
[2016-10-31 04:03:41.595556] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-null' is not recognized
[2016-10-31 04:03:41.595619] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-quota: option 'timeout' is not recognized
[2016-10-31 04:03:41.595647] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-marker: option 'quota-version' is not recognized
[2016-10-31 04:03:41.595677] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-trash: option 'brick-path' is not recognized
[2016-10-31 04:03:41.605103] W [MSGID: 113036] [posix.c:2018:posix_rename] 0-c_glusterfs-posix: found directory at /opt/lvmdir/c2/brick/.trashcan/ while expecting ENOENT [File exists]
Final graph:
+------------------------------------------------------------------------------+
1: volume c_glusterfs-posix
2: type storage/posix
3: option glusterd-uuid 5818a372-d517-41af-bd2f-0fce648773df
4: option directory /opt/lvmdir/c2/brick
5: option volume-id 0a50e4e5-ecfc-432c-88d9-2083455e4baf
6: end-volume
7:
8: volume c_glusterfs-trash
9: type features/trash
10: option trash-dir .trashcan
11: option brick-path /opt/lvmdir/c2/brick
12: option trash-internal-op off
13: subvolumes c_glusterfs-posix
14: end-volume
15:
16: volume c_glusterfs-changelog
17: type features/changelog
18: option changelog-brick /opt/lvmdir/c2/brick
19: option changelog-dir /opt/lvmdir/c2/brick/.glusterfs/changelogs
20: option changelog-barrier-timeout 120
21: subvolumes c_glusterfs-trash
22: end-volume
23:
24: volume c_glusterfs-bitrot-stub
25: type features/bitrot-stub
26: option export /opt/lvmdir/c2/brick
27: subvolumes c_glusterfs-changelog
28: end-volume
29:
30: volume c_glusterfs-access-control
31: type features/access-control
32: subvolumes c_glusterfs-bitrot-stub
33: end-volume
34:
35: volume c_glusterfs-locks
36: type features/locks
37: subvolumes c_glusterfs-access-control
38: end-volume
39:
40: volume c_glusterfs-upcall
41: type features/upcall
42: option cache-invalidation off
43: subvolumes c_glusterfs-locks
44: end-volume
45:
46: volume c_glusterfs-io-threads
47: type performance/io-threads
48: subvolumes c_glusterfs-upcall
49: end-volume
50:
51: volume c_glusterfs-marker
52: type features/marker
53: option volume-uuid 0a50e4e5-ecfc-432c-88d9-2083455e4baf
54: option timestamp-file /system/glusterd/vols/c_glusterfs/marker.tstamp
55: option quota-version 0
56: option xtime off
57: option gsync-force-xtime off
58: option quota off
59: option inode-quota off
60: subvolumes c_glusterfs-io-threads
61: end-volume
62:
63: volume c_glusterfs-barrier
64: type features/barrier
65: option barrier disable
66: option barrier-timeout 120
67: subvolumes c_glusterfs-marker
68: end-volume
69:
70: volume c_glusterfs-index
71: type features/index
72: option index-base /opt/lvmdir/c2/brick/.glusterfs/indices
73: subvolumes c_glusterfs-barrier
74: end-volume
75:
76: volume c_glusterfs-quota
77: type features/quota
78: option volume-uuid c_glusterfs
79: option server-quota off
80: option timeout 0
81: option deem-statfs off
82: subvolumes c_glusterfs-index
83: end-volume
84:
85: volume c_glusterfs-worm
86: type features/worm
87: option worm off
88: subvolumes c_glusterfs-quota
89: end-volume
90:
91: volume c_glusterfs-read-only
92: type features/read-only
93: option read-only off
94: subvolumes c_glusterfs-worm
95: end-volume
96:
97: volume /opt/lvmdir/c2/brick
98: type debug/io-stats
99: option latency-measurement off
100: option count-fop-hits off
101: subvolumes c_glusterfs-read-only
102: end-volume
103:
104: volume c_glusterfs-server
105: type protocol/server
106: option transport.socket.listen-port 49391
107: option rpc-auth.auth-glusterfs on
108: option rpc-auth.auth-unix on
109: option rpc-auth.auth-null on
110: option rpc-auth-allow-insecure on
111: option transport-type tcp
112: option auth.login./opt/lvmdir/c2/brick.allow 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
113: option auth.login.5c5afe17-02ce-4b9b-aef4-e66efa1c6132.password ade893cc-19dd-4d67-ae8e-cb929ef4e770
114: option auth.addr./opt/lvmdir/c2/brick.allow *
115: option transport.tcp-user-timeout 4
116: subvolumes /opt/lvmdir/c2/brick
117: end-volume
118:
+------------------------------------------------------------------------------+
[2016-10-31 04:03:43.191850] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:03:43.191927] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:03:45.556522] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:03:45.556581] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-1599-2016/10/31-04:03:41:530232-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:07.831958] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:07.832023] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:04:07.974625] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:07.974688] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:08.063317] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:08.063359] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:10.032591] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:04:10.032672] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:05:04.530186] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:04.530267] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 60a17763-95fd-4162-8402-9651dfbe9ec4 held by {client=0x3fff7c002990, pid=0 lk-owner=a87b0a7cff3f0000}
[2016-10-31 04:05:04.530306] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:05:04.530375] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760618] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760693] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:57.341892] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:05:57.341928] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1604-2016/10/31-04:05:53:637848-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.291380] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.291441] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.403916] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.403963] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3292-2016/10/31-04:05:57:685417-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:03.631296] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:03.631332] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:06:03.634762] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.634945] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 200b0bb2-d903-4924-b025-262d805c0f2f held by {client=0x3fff7c026520, pid=6232 lk-owner=6cd0c189ff3f0000}
[2016-10-31 04:06:03.634987] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /pmd/2
[2016-10-31 04:06:03.635035] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /configuration/oamrdncy.cfg
[2016-10-31 04:06:03.635349] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /
[2016-10-31 04:06:03.635414] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:06:03.635468] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001
[2016-10-31 04:06:03.635554] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl
[2016-10-31 04:06:03.635889] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001/cello
[2016-10-31 04:06:03.640329] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.641143] E [inodelk.c:309:__inode_unlock_lock] 0-c_glusterfs-locks: Matching lock not found for unlock 0-9223372036854775807, by 6cd0c189ff3f0000 on 0x3fff840fd9a0
[2016-10-31 04:06:03.641200] E [MSGID: 115053] [server-rpc-fops.c:275:server_inodelk_cbk] 0-c_glusterfs-server: 32623: INODELK /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) ==> (Invalid argument) [Invalid argument]
[2016-10-31 04:06:06.270314] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0
[2016-10-31 04:06:06.270449] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0
Thanks,
Xin
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
