I don't think it is similar to Xavi fixed in https://review.gluster.org/#/c/glusterfs/+/24099/.
Is it possible to share the output "thread apply all bt full" after attaching the core with gdb?
Is it possible to share the output "thread apply all bt full" after attaching the core with gdb?
Regards,
Mohit Agrawal
On Sat, Feb 15, 2020 at 7:25 AM Amar Tumballi <amar@xxxxxxxxx> wrote:
Is this crash seen already ? Does https://review.gluster.org/#/c/glusterfs/+/24099/ fix this?On Sat, Feb 15, 2020 at 4:32 AM Joe Julian <joe@xxxxxxxxxxxxxxxx> wrote:These crashes have been happening almost daily. Any thoughts on how to
stabilize this?
[2020-02-14 19:02:13.932178] I [MSGID: 100030] [glusterfsd.c:2865:main]
0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 7.0
(args: /usr/bin/glusterfs --process-name fuse --volfile-server=gluster
--volfile-id=kube /tmp/hostpath_pv)
[2020-02-14 19:02:13.939208] I [glusterfsd.c:2593:daemonize]
0-glusterfs: Pid of current running process is 1828083
[2020-02-14 19:02:13.970829] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 0
[2020-02-14 19:02:13.970870] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2020-02-14 19:02:14.026097] W [MSGID: 108003]
[afr.c:98:fix_quorum_options] 0-kube-replicate-0: quorum-type none
overriding quorum-count 1
[2020-02-14 19:02:14.027753] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2020-02-14 19:02:14.027871] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 4
[2020-02-14 19:02:14.027939] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 5
[2020-02-14 19:02:14.027984] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 3
[2020-02-14 19:02:14.028010] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 6
[2020-02-14 19:02:14.028410] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 7
[2020-02-14 19:02:14.028826] I [MSGID: 114020] [client.c:2434:notify]
0-kube-client-0: parent translators are ready, attempting connect on
transport
[2020-02-14 19:02:14.031361] I [MSGID: 114020] [client.c:2434:notify]
0-kube-client-1: parent translators are ready, attempting connect on
transport
[2020-02-14 19:02:14.031840] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:02:14.031880] I [socket.c:863:__socket_shutdown]
0-kube-client-0: intentional socket shutdown(12)
Final graph:
+------------------------------------------------------------------------------+
1: volume kube-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host strabo
5: option remote-subvolume /data/gluster/kube
6: option transport-type socket
7: option transport.address-family inet
8: option username 51b95740-bff7-4744-bbfc-dbae17670997
9: option password 021da395-9404-43ac-b358-2f5dfc5acc71
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT on
12: option event-threads 8
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option send-gids true
18: end-volume
19:
20: volume kube-client-1
21: type protocol/client
22: option ping-timeout 42
23: option remote-host nightshade
24: option remote-subvolume /data/gluster/kube
25: option transport-type socket
26: option transport.address-family inet
27: option username 51b95740-bff7-4744-bbfc-dbae17670997
28: option password 021da395-9404-43ac-b358-2f5dfc5acc71
29: option transport.socket.ssl-enabled off
30: option filter-O_DIRECT on
31: option event-threads 8
32: option transport.tcp-user-timeout 0
33: option transport.socket.keepalive-time 20
34: option transport.socket.keepalive-interval 2
35: option transport.socket.keepalive-count 9
36: option send-gids true
37: end-volume
38:
39: volume kube-replicate-0
40: type cluster/replicate
41: option afr-pending-xattr kube-client-0,kube-client-1
42: option quorum-count 1
43: option use-compound-fops off
44: subvolumes kube-client-0 kube-client-1
45: end-volume
46:
47: volume kube-dht
48: type cluster/distribute
49: option readdir-optimize on
50: option lock-migration off
51: option force-migration off
52: subvolumes kube-replicate-0
53: end-volume
54:
55: volume kube-io-cache
56: type performance/io-cache
57: option cache-timeout 30
58: subvolumes kube-dht
59: end-volume
60:
61: volume kube-open-behind
62: type performance/open-behind
63: subvolumes kube-io-cache
64: end-volume
65:
66: volume kube-md-cache
67: type performance/md-cache
68: option cache-invalidation on
69: subvolumes kube-open-behind
70: end-volume
71:
72: volume kube-io-threads
73: type performance/io-threads
74: subvolumes kube-md-cache
75: end-volume
76:
77: volume kube
78: type debug/io-stats
79: option log-level INFO
80: option threads 16
81: option latency-measurement off
82: option count-fop-hits off
83: option global-threading off
84: subvolumes kube-io-threads
85: end-volume
86:
87: volume meta-autoload
88: type meta
89: subvolumes kube
90: end-volume
91:
+------------------------------------------------------------------------------+
[2020-02-14 19:02:14.052685] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-1: changing port to 49159 (from 0)
[2020-02-14 19:02:14.052792] I [socket.c:863:__socket_shutdown]
0-kube-client-1: intentional socket shutdown(13)
[2020-02-14 19:02:14.056240] I [MSGID: 114057]
[client-handshake.c:1373:select_server_supported_programs]
0-kube-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version
(400)
[2020-02-14 19:02:14.076058] I [MSGID: 114046]
[client-handshake.c:1104:client_setvolume_cbk] 0-kube-client-1:
Connected to kube-client-1, attached to remote volume '/data/gluster/kube'.
[2020-02-14 19:02:14.076113] I [MSGID: 108005]
[afr-common.c:5277:__afr_handle_child_up_event] 0-kube-replicate-0:
Subvolume 'kube-client-1' came back up; going online.
[2020-02-14 19:02:24.031553] I [fuse-bridge.c:5162:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
kernel 7.31
[2020-02-14 19:02:24.031605] I [fuse-bridge.c:5777:fuse_graph_sync]
0-fuse: switched to graph 0
[2020-02-14 19:04:27.205776] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:06:40.325688] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:08:53.444994] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:11:06.565527] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(1) op(LOOKUP)
frame : type(1) op(FLUSH)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(FLUSH)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2020-02-14 19:13:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 7.0
/usr/lib/libglusterfs.so.0(+0x25e12)[0x7fc026b55e12]
/usr/lib/libglusterfs.so.0(gf_print_trace+0x354)[0x7fc026b60b74]
/usr/lib/libc.so.6(+0x3bfb0)[0x7fc0268e3fb0]
/usr/lib/libglusterfs.so.0(mem_get+0xf0)[0x7fc026b81850]
/usr/lib/libglusterfs.so.0(+0x1a4cd)[0x7fc026b4a4cd]
/usr/lib/libglusterfs.so.0(dict_setn+0x1f1)[0x7fc026b4a941]
/usr/lib/libglusterfs.so.0(dict_set_dynptr+0x32)[0x7fc026b4e342]
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x3b318)[0x7fc0210cb318]
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x56d03)[0x7fc0210e6d03]
/usr/lib/libgfrpc.so.0(+0xe957)[0x7fc026afe957]
/usr/lib/libgfrpc.so.0(+0xecc9)[0x7fc026afecc9]
/usr/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fc026afb816]
/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0x4629)[0x7fc022294629]
/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0xc45c)[0x7fc02229c45c]
/usr/lib/libglusterfs.so.0(+0x8cedc)[0x7fc026bbcedc]
/usr/lib/libpthread.so.0(+0x94cf)[0x7fc026a794cf]
/usr/lib/libc.so.6(clone+0x43)[0x7fc0269a72d3]
---------
________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
--
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
