Oops. I replied to Mohammed and not the whole list. Here's the backtrace
and the full backtrace too:
root@nfs3:/home/ernied# gdb /usr/sbin/glusterd /core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/glusterd...(no debugging symbols
found)...done.
warning: core file may not match specified executable file.
[New LWP 1519]
[New LWP 1520]
[New LWP 1516]
[New LWP 1780]
[New LWP 1518]
[New LWP 1517]
[New LWP 1781]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 pthread_spin_lock () at
../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
24 ../nptl/sysdeps/x86_64/pthread_spin_lock.S: No such file or
directory.
(gdb) bt
#0 pthread_spin_lock () at
../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
#1 0x00007fb81dee520d in __gf_free () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2 0x00007fb81deaa625 in data_destroy () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#3 0x00007fb81dead1cd in dict_get_str () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#4 0x00007fb8193f52f9 in glusterd_xfer_cli_probe_resp ()
from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
#5 0x00007fb8193f6017 in __glusterd_handle_cli_probe ()
from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
#6 0x00007fb8193ee9a0 in glusterd_big_locked_handler ()
from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
#7 0x00007fb81def38d2 in synctask_wrap () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#8 0x00007fb81d2c38b0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()
(gdb) bt full
#0 pthread_spin_lock () at
../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
No locals.
#1 0x00007fb81dee520d in __gf_free () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
No symbol table info available.
#2 0x00007fb81deaa625 in data_destroy () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
No symbol table info available.
#3 0x00007fb81dead1cd in dict_get_str () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
No symbol table info available.
#4 0x00007fb8193f52f9 in glusterd_xfer_cli_probe_resp ()
from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
No symbol table info available.
#5 0x00007fb8193f6017 in __glusterd_handle_cli_probe ()
from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
No symbol table info available.
#6 0x00007fb8193ee9a0 in glusterd_big_locked_handler ()
from
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
No symbol table info available.
#7 0x00007fb81def38d2 in synctask_wrap () from
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0
No symbol table info available.
#8 0x00007fb81d2c38b0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
On 2016-03-30 23:15, Mohammed Rafi K C wrote:
Hi Ernie,
Can you please paste the back trace from the core file.
Regards
Rafi KC
On 03/31/2016 02:31 AM, Ernie Dunbar wrote:
Hi everyone.
I'm trying to add a new Gluster node to our cluster, and when trying
to probing the first node in the cluster, the new node crashes with
the following report (logs start when the daemon starts):
---------
[2016-03-30 20:32:05.191659] I [MSGID: 100030]
[glusterfsd.c:2332:main] 0-/usr/sbin/glusterd: Started running
/usr/sbin/glusterd version 3.7.9 (args: /usr/sbin/glusterd -p
/var/run/glusterd.pid)
[2016-03-30 20:32:05.195695] I [MSGID: 106478] [glusterd.c:1337:init]
0-management: Maximum allowed open file descriptors set to 65536
[2016-03-30 20:32:05.195752] I [MSGID: 106479] [glusterd.c:1386:init]
0-management: Using /var/lib/glusterd as working directory
[2016-03-30 20:32:05.200609] W [MSGID: 103071]
[rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
channel creation failed [No such device]
[2016-03-30 20:32:05.200648] W [MSGID: 103055] [rdma.c:4901:init]
0-rdma.management: Failed to initialize IB Device
[2016-03-30 20:32:05.200662] W
[rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma'
initialization failed
[2016-03-30 20:32:05.200723] W [rpcsvc.c:1597:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2016-03-30 20:32:05.200743] E [MSGID: 106243] [glusterd.c:1610:init]
0-management: creation of 1 listeners failed, continuing with
succeeded transport
[2016-03-30 20:32:07.135310] I [MSGID: 106513]
[glusterd-store.c:2062:glusterd_restore_op_version] 0-glusterd:
retrieved op-version: 30501
[2016-03-30 20:32:07.135775] I [MSGID: 106498]
[glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo]
0-management: connect returned 0
[2016-03-30 20:32:07.135876] I
[rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting
frame-timeout to 600
[2016-03-30 20:32:07.136651] W [socket.c:870:__socket_keepalive]
0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid
argument
[2016-03-30 20:32:07.136673] E [socket.c:2966:socket_connect]
0-management: Failed to set keep-alive: Invalid argument
[2016-03-30 20:32:07.136908] I [MSGID: 106194]
[glusterd-store.c:3523:glusterd_store_retrieve_missed_snaps_list]
0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.socket.listen-backlog 128
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2016-03-30 20:32:07.138287] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 1
[2016-03-30 20:32:07.138980] I [MSGID: 106544]
[glusterd.c:159:glusterd_uuid_init] 0-management: retrieved UUID:
ae191e96-9cd6-4e2b-acae-18f2cc45e6ed
[2016-03-30 20:32:07.139422] I [MSGID: 106163]
[glusterd-handshake.c:1194:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 30501
[2016-03-30 20:32:14.394056] I [MSGID: 106487]
[glusterd-handler.c:1239:__glusterd_handle_cli_probe] 0-glusterd:
Received CLI probe req nfs1 24007
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2016-03-30 20:32:14
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.9
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f0401a78562]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f0401a9464d]
/lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f0400e76d40]
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_spin_lock+0x0)[0x7f04012120f0]
---------
Both nodes are running GlusterFS 3.7.9 on Ubuntu Trusty Tahr (14.04
LTS). Node 1 is running Linux kernel 3.13.0-55-generic #94-Ubuntu SMP,
and node 3 is running Linux kernel 3.13.0-77-generic #121-Ubuntu SMP.
To me, this seems to be the only difference between the systems,
although the new node has the very latest version of the Gluster
packages from the launchpad.net PPA. I would imagine that Node 1 has
the same update, but it's hard to tell.
Any help would be much appreciated.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
