Re: [Gluster-users] Not able to start glusterd

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Abhishek,

We need below information on investigate this issue.
1. gluster --version
2. Please run glusterd in gdb, so that we can capture the backtrace. I see some rpc errors in log, but backtrace will be more helpful.
    To run glusterd in gdb, you need start glusterd in gdb (i.e. gdb glusterd, and then give the command "run -N"). when you see a segmentation           fault, please capture the backtrace and paste it here.

On Wed, Mar 6, 2019 at 10:07 AM ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi Team,

I am facing the issue where at the time of starting the glusterd segmentation fault is reported.

Below are the logs

root@128:/usr/sbin# ./glusterd  --debug
[1970-01-01 15:19:43.940386] I [MSGID: 100030] [glusterfsd.c:2691:main] 0-./glusterd: Started running ./glusterd version 5.0 (args: ./glusterd --debug)
[1970-01-01 15:19:43.940855] D [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[1970-01-01 15:19:43.941736] D [MSGID: 0] [glusterfsd.c:747:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
[1970-01-01 15:19:43.945796] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so: undefined symbol: xlator_api. Fall back to old symbols
[1970-01-01 15:19:43.946279] I [MSGID: 106478] [glusterd.c:1435:init] 0-management: Maximum allowed open file descriptors set to 65536
[1970-01-01 15:19:43.946419] I [MSGID: 106479] [glusterd.c:1491:init] 0-management: Using /var/lib/glusterd as working directory
[1970-01-01 15:19:43.946515] I [MSGID: 106479] [glusterd.c:1497:init] 0-management: Using /var/run/gluster as pid file working directory
[1970-01-01 15:19:43.946968] D [MSGID: 0] [glusterd.c:458:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 10
[1970-01-01 15:19:43.947139] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC service inited.
[1970-01-01 15:19:43.947241] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[1970-01-01 15:19:43.947379] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.0/rpc-transport/socket.so
[1970-01-01 15:19:43.955198] D [socket.c:4464:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0
[1970-01-01 15:19:43.955316] D [socket.c:4482:socket_init] 0-socket.management: Reconfigued transport.keepalivecnt=9
[1970-01-01 15:19:43.955415] D [socket.c:4167:ssl_setup_connection_params] 0-socket.management: SSL support on the I/O path is NOT enabled
[1970-01-01 15:19:43.955504] D [socket.c:4170:ssl_setup_connection_params] 0-socket.management: SSL support for glusterd is NOT enabled
[1970-01-01 15:19:43.955612] D [name.c:572:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet6
[1970-01-01 15:19:43.955928] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so
[1970-01-01 15:19:43.956079] E [rpc-transport.c:273:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[1970-01-01 15:19:43.956177] W [rpc-transport.c:277:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[1970-01-01 15:19:43.956270] W [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[1970-01-01 15:19:43.956362] E [MSGID: 106244] [glusterd.c:1798:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[1970-01-01 15:19:43.956459] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0
[1970-01-01 15:19:43.956561] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0
[1970-01-01 15:19:43.956666] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
[1970-01-01 15:19:43.956758] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0
[1970-01-01 15:19:43.956853] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0
[1970-01-01 15:19:43.956946] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
[1970-01-01 15:19:43.957062] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
[1970-01-01 15:19:43.957205] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC service inited.
[1970-01-01 15:19:43.957303] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[1970-01-01 15:19:43.957408] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.0/rpc-transport/socket.so
[1970-01-01 15:19:43.957563] D [socket.c:4424:socket_init] 0-socket.management: disabling nodelay
[1970-01-01 15:19:43.957650] D [socket.c:4464:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0
[1970-01-01 15:19:43.957738] D [socket.c:4482:socket_init] 0-socket.management: Reconfigued transport.keepalivecnt=9
[1970-01-01 15:19:43.957830] D [socket.c:4167:ssl_setup_connection_params] 0-socket.management: SSL support on the I/O path is NOT enabled
[1970-01-01 15:19:43.957922] D [socket.c:4170:ssl_setup_connection_params] 0-socket.management: SSL support for glusterd is NOT enabled
[1970-01-01 15:19:43.958186] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
[1970-01-01 15:19:43.958280] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0
[1970-01-01 15:19:43.958461] D [MSGID: 0] [glusterd-utils.c:7878:glusterd_sm_tr_log_init] 0-glusterd: returning 0
[1970-01-01 15:19:43.958557] D [MSGID: 0] [glusterd.c:1875:init] 0-management: cannot get run-with-valgrind value
[1970-01-01 15:19:43.960895] E [MSGID: 101032] [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[1970-01-01 15:19:43.961016] D [MSGID: 0] [store.c:452:gf_store_handle_retrieve] 0-: Returning -1
[1970-01-01 15:19:43.961108] D [MSGID: 0] [glusterd-store.c:2169:glusterd_retrieve_op_version] 0-management: Unable to get store handle!
[1970-01-01 15:19:43.961216] E [MSGID: 101032] [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[1970-01-01 15:19:43.961325] D [MSGID: 0] [store.c:452:gf_store_handle_retrieve] 0-: Returning -1
[1970-01-01 15:19:43.961428] D [MSGID: 0] [glusterd-store.c:2345:glusterd_retrieve_uuid] 0-management: Unable to get storehandle!
[1970-01-01 15:19:43.961523] D [MSGID: 0] [glusterd-store.c:2366:glusterd_retrieve_uuid] 0-management: Returning -1
[1970-01-01 15:19:43.961617] I [MSGID: 106514] [glusterd-store.c:2304:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 50000
[1970-01-01 15:19:43.962658] D [MSGID: 0] [store.c:432:gf_store_handle_new] 0-: Returning 0
[1970-01-01 15:19:43.962750] D [MSGID: 0] [store.c:452:gf_store_handle_retrieve] 0-: Returning 0
[1970-01-01 15:19:43.963047] D [MSGID: 0] [store.c:515:gf_store_iter_new] 0-: Returning with 0
[1970-01-01 15:19:43.963194] D [MSGID: 0] [store.c:632:gf_store_iter_get_next] 0-: Returning with 0
[1970-01-01 15:19:43.963318] D [MSGID: 0] [store.c:632:gf_store_iter_get_next] 0-: Returning with -1
[1970-01-01 15:19:43.963455] D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-: Returning 0
[1970-01-01 15:19:43.963757] D [MSGID: 0] [glusterd-store.c:3546:glusterd_store_retrieve_volumes] 0-management: Returning with 0
[1970-01-01 15:19:43.964159] D [MSGID: 0] [glusterd-store.c:4662:glusterd_store_retrieve_peers] 0-management: Returning with 0
[1970-01-01 15:19:43.964471] I [MSGID: 106194] [glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
[1970-01-01 15:19:43.964580] D [MSGID: 0] [glusterd-store.c:4104:glusterd_store_retrieve_snaps] 0-management: Returning with 0
[1970-01-01 15:19:43.964680] D [MSGID: 0] [glusterd-store.c:4894:glusterd_restore] 0-management: Returning 0
[1970-01-01 15:19:43.965060] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-management: option event-threads using set value 1
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.listen-backlog 10
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:  
+------------------------------------------------------------------------------+
[1970-01-01 15:19:43.966808] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[1970-01-01 15:19:44.840454] E [rpcsvc.c:513:rpcsvc_request_create] 0-rpc-service: RPC version not supported (XID: 0x0, Ver: 0, Program: 0, ProgVers: 0, Proc: 2) from trans (socket.management)
[1970-01-01 15:19:44.840884] D [rpcsvc.c:1416:rpcsvc_error_reply] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4] (--> /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4] (--> /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] (--> /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] (--> /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] ))))) 0-: sending a RPC error reply
[1970-01-01 15:19:44.841055] D [logging.c:1805:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages
[1970-01-01 15:19:44.841156] D [logging.c:1808:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages
pending frames:
signal received: 11
time of crash: 
1970-01-01 15:19:44
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 5.0
/usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4]
/usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0]
./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec]
linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478]
/lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc]
/lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c]
/lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4]
/lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0]
/lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120]
/usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c]
/usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c]
/usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614]
/usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00]
/usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224]
/usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c]
/usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c]
/usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc]
/lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30]
/lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c]
---------
Segmentation fault (core dumped)

Could you  please help me, what actually the problem?


--




Regards
Abhishek Paliwal
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
Thanks,
Sanju
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel
[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux