Re: crashing a lot

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I don't think it is similar to Xavi fixed in https://review.gluster.org/#/c/glusterfs/+/24099/.
Is it possible to share the output "thread apply all bt full" after attaching the core with gdb?

Regards,
Mohit Agrawal

On Sat, Feb 15, 2020 at 7:25 AM Amar Tumballi <amar@xxxxxxxxx> wrote:
Is this crash seen already ?  Does https://review.gluster.org/#/c/glusterfs/+/24099/ fix this?

On Sat, Feb 15, 2020 at 4:32 AM Joe Julian <joe@xxxxxxxxxxxxxxxx> wrote:
These crashes have been happening almost daily. Any thoughts on how to
stabilize this?


[2020-02-14 19:02:13.932178] I [MSGID: 100030] [glusterfsd.c:2865:main]
0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 7.0
(args: /usr/bin/glusterfs --process-name fuse --volfile-server=gluster
--volfile-id=kube /tmp/hostpath_pv)
[2020-02-14 19:02:13.939208] I [glusterfsd.c:2593:daemonize]
0-glusterfs: Pid of current running process is 1828083
[2020-02-14 19:02:13.970829] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 0
[2020-02-14 19:02:13.970870] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2020-02-14 19:02:14.026097] W [MSGID: 108003]
[afr.c:98:fix_quorum_options] 0-kube-replicate-0: quorum-type none
overriding quorum-count 1
[2020-02-14 19:02:14.027753] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2020-02-14 19:02:14.027871] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 4
[2020-02-14 19:02:14.027939] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 5
[2020-02-14 19:02:14.027984] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 3
[2020-02-14 19:02:14.028010] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 6
[2020-02-14 19:02:14.028410] I [MSGID: 101190]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 7
[2020-02-14 19:02:14.028826] I [MSGID: 114020] [client.c:2434:notify]
0-kube-client-0: parent translators are ready, attempting connect on
transport
[2020-02-14 19:02:14.031361] I [MSGID: 114020] [client.c:2434:notify]
0-kube-client-1: parent translators are ready, attempting connect on
transport
[2020-02-14 19:02:14.031840] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:02:14.031880] I [socket.c:863:__socket_shutdown]
0-kube-client-0: intentional socket shutdown(12)
Final graph:
+------------------------------------------------------------------------------+
   1: volume kube-client-0
   2:     type protocol/client
   3:     option ping-timeout 42
   4:     option remote-host strabo
   5:     option remote-subvolume /data/gluster/kube
   6:     option transport-type socket
   7:     option transport.address-family inet
   8:     option username 51b95740-bff7-4744-bbfc-dbae17670997
   9:     option password 021da395-9404-43ac-b358-2f5dfc5acc71
  10:     option transport.socket.ssl-enabled off
  11:     option filter-O_DIRECT on
  12:     option event-threads 8
  13:     option transport.tcp-user-timeout 0
  14:     option transport.socket.keepalive-time 20
  15:     option transport.socket.keepalive-interval 2
  16:     option transport.socket.keepalive-count 9
  17:     option send-gids true
  18: end-volume
  19:
  20: volume kube-client-1
  21:     type protocol/client
  22:     option ping-timeout 42
  23:     option remote-host nightshade
  24:     option remote-subvolume /data/gluster/kube
  25:     option transport-type socket
  26:     option transport.address-family inet
  27:     option username 51b95740-bff7-4744-bbfc-dbae17670997
  28:     option password 021da395-9404-43ac-b358-2f5dfc5acc71
  29:     option transport.socket.ssl-enabled off
  30:     option filter-O_DIRECT on
  31:     option event-threads 8
  32:     option transport.tcp-user-timeout 0
  33:     option transport.socket.keepalive-time 20
  34:     option transport.socket.keepalive-interval 2
  35:     option transport.socket.keepalive-count 9
  36:     option send-gids true
  37: end-volume
  38:
  39: volume kube-replicate-0
  40:     type cluster/replicate
  41:     option afr-pending-xattr kube-client-0,kube-client-1
  42:     option quorum-count 1
  43:     option use-compound-fops off
  44:     subvolumes kube-client-0 kube-client-1
  45: end-volume
  46:
  47: volume kube-dht
  48:     type cluster/distribute
  49:     option readdir-optimize on
  50:     option lock-migration off
  51:     option force-migration off
  52:     subvolumes kube-replicate-0
  53: end-volume
  54:
  55: volume kube-io-cache
  56:     type performance/io-cache
  57:     option cache-timeout 30
  58:     subvolumes kube-dht
  59: end-volume
  60:
  61: volume kube-open-behind
  62:     type performance/open-behind
  63:     subvolumes kube-io-cache
  64: end-volume
  65:
  66: volume kube-md-cache
  67:     type performance/md-cache
  68:     option cache-invalidation on
  69:     subvolumes kube-open-behind
  70: end-volume
  71:
  72: volume kube-io-threads
  73:     type performance/io-threads
  74:     subvolumes kube-md-cache
  75: end-volume
  76:
  77: volume kube
  78:     type debug/io-stats
  79:     option log-level INFO
  80:     option threads 16
  81:     option latency-measurement off
  82:     option count-fop-hits off
  83:     option global-threading off
  84:     subvolumes kube-io-threads
  85: end-volume
  86:
  87: volume meta-autoload
  88:     type meta
  89:     subvolumes kube
  90: end-volume
  91:
+------------------------------------------------------------------------------+
[2020-02-14 19:02:14.052685] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-1: changing port to 49159 (from 0)
[2020-02-14 19:02:14.052792] I [socket.c:863:__socket_shutdown]
0-kube-client-1: intentional socket shutdown(13)
[2020-02-14 19:02:14.056240] I [MSGID: 114057]
[client-handshake.c:1373:select_server_supported_programs]
0-kube-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version
(400)
[2020-02-14 19:02:14.076058] I [MSGID: 114046]
[client-handshake.c:1104:client_setvolume_cbk] 0-kube-client-1:
Connected to kube-client-1, attached to remote volume '/data/gluster/kube'.
[2020-02-14 19:02:14.076113] I [MSGID: 108005]
[afr-common.c:5277:__afr_handle_child_up_event] 0-kube-replicate-0:
Subvolume 'kube-client-1' came back up; going online.
[2020-02-14 19:02:24.031553] I [fuse-bridge.c:5162:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
kernel 7.31
[2020-02-14 19:02:24.031605] I [fuse-bridge.c:5777:fuse_graph_sync]
0-fuse: switched to graph 0
[2020-02-14 19:04:27.205776] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:06:40.325688] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:08:53.444994] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
[2020-02-14 19:11:06.565527] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
0-kube-client-0: changing port to 49159 (from 0)
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(1) op(LOOKUP)
frame : type(1) op(FLUSH)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(FLUSH)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2020-02-14 19:13:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 7.0
/usr/lib/libglusterfs.so.0(+0x25e12)[0x7fc026b55e12]
/usr/lib/libglusterfs.so.0(gf_print_trace+0x354)[0x7fc026b60b74]
/usr/lib/libc.so.6(+0x3bfb0)[0x7fc0268e3fb0]
/usr/lib/libglusterfs.so.0(mem_get+0xf0)[0x7fc026b81850]
/usr/lib/libglusterfs.so.0(+0x1a4cd)[0x7fc026b4a4cd]
/usr/lib/libglusterfs.so.0(dict_setn+0x1f1)[0x7fc026b4a941]
/usr/lib/libglusterfs.so.0(dict_set_dynptr+0x32)[0x7fc026b4e342]
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x3b318)[0x7fc0210cb318]
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x56d03)[0x7fc0210e6d03]
/usr/lib/libgfrpc.so.0(+0xe957)[0x7fc026afe957]
/usr/lib/libgfrpc.so.0(+0xecc9)[0x7fc026afecc9]
/usr/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fc026afb816]
/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0x4629)[0x7fc022294629]
/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0xc45c)[0x7fc02229c45c]
/usr/lib/libglusterfs.so.0(+0x8cedc)[0x7fc026bbcedc]
/usr/lib/libpthread.so.0(+0x94cf)[0x7fc026a794cf]
/usr/lib/libc.so.6(clone+0x43)[0x7fc0269a72d3]
---------

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
--
Container Storage made easy!

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux