Re: Not able to start glusterd

RAFI KC <rkavunga@xxxxxxxxxx> · Wed, 6 Mar 2019 14:27:44 +0530



    Hi Abhishek,
    Good to know that you have resolved your problem. Do you think
      any more information is required to add in the upgrade doc for a
      smooth upgrade flow. It would be great to see a PR to the repo
      https://github.com/gluster/glusterdocs/tree/master/docs/Upgrade-Guide
      for updating the doc with the information.
    

    Regards
    Rafi KC

    
    On 3/6/19 2:03 PM, ABHISHEK PALIWAL
      wrote:

    
      Hi Sanju,
        

        Thanks for the response.
        

        I have resolved the issue, actually I have updated from
          3.7.6 to 5.0, in new version RPC is coming from libtirpb , but
          I forgot to enable "--with-libtirpc" in configuration.
        

        After enabling able to start glusterd.
        

        Regards,
        Abhishek
      
      
        On Wed, Mar 6, 2019 at 12:58
          PM Sanju Rakonde <srakonde@xxxxxxxxxx> wrote:

        
          Abhishek,
            

            We need below information on investigate this issue.
            1. gluster --version
            2. Please run glusterd in gdb, so that we can capture
              the backtrace. I see some rpc errors in log, but backtrace
              will be more helpful.
                To run glusterd in gdb, you need start glusterd in
              gdb (i.e. gdb glusterd, and then give the command "run
              -N"). when you see a segmentation           fault, please
              capture the backtrace and paste it here.
          
          
            On Wed, Mar 6, 2019 at
              10:07 AM ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx>
              wrote:

            
                Hi Team,
                  

                  I am facing the issue where at the time of
                    starting the glusterd segmentation fault is
                    reported.
                  

                  Below are the logs
                  

                    root@128:/usr/sbin# ./glusterd  --debug
                    [1970-01-01 15:19:43.940386] I [MSGID: 100030]
                      [glusterfsd.c:2691:main] 0-./glusterd: Started
                      running ./glusterd version 5.0 (args: ./glusterd
                      --debug)
                    [1970-01-01 15:19:43.940855] D
                      [logging.c:1833:__gf_log_inject_timer_event]
                      0-logging-infra: Starting timer now. Timeout =
                      120, current buf size = 5
                    [1970-01-01 15:19:43.941736] D [MSGID: 0]
                      [glusterfsd.c:747:get_volfp] 0-glusterfsd: loading
                      volume file /etc/glusterfs/glusterd.vol
                    [1970-01-01 15:19:43.945796] D [MSGID: 101097]
                      [xlator.c:341:xlator_dynload_newway] 0-xlator:
                      dlsym(xlator_api) on
                      /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so:
                      undefined symbol: xlator_api. Fall back to old
                      symbols
                    [1970-01-01 15:19:43.946279] I [MSGID: 106478]
                      [glusterd.c:1435:init] 0-management: Maximum
                      allowed open file descriptors set to 65536
                    [1970-01-01 15:19:43.946419] I [MSGID: 106479]
                      [glusterd.c:1491:init] 0-management: Using
                      /var/lib/glusterd as working directory
                    [1970-01-01 15:19:43.946515] I [MSGID: 106479]
                      [glusterd.c:1497:init] 0-management: Using
                      /var/run/gluster as pid file working directory
                    [1970-01-01 15:19:43.946968] D [MSGID: 0]
                      [glusterd.c:458:glusterd_rpcsvc_options_build]
                      0-glusterd: listen-backlog value: 10
                    [1970-01-01 15:19:43.947139] D
                      [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC
                      service inited.
                    [1970-01-01 15:19:43.947241] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GF-DUMP,
                      Num: 123451501, Ver: 1, Port: 0
                    [1970-01-01 15:19:43.947379] D
                      [rpc-transport.c:269:rpc_transport_load]
                      0-rpc-transport: attempt to load file
                      /usr/lib64/glusterfs/5.0/rpc-transport/socket.so
                    [1970-01-01 15:19:43.955198] D
                      [socket.c:4464:socket_init] 0-socket.management:
                      Configued transport.tcp-user-timeout=0
                    [1970-01-01 15:19:43.955316] D
                      [socket.c:4482:socket_init] 0-socket.management:
                      Reconfigued transport.keepalivecnt=9
                    [1970-01-01 15:19:43.955415] D
                      [socket.c:4167:ssl_setup_connection_params]
                      0-socket.management: SSL support on the I/O path
                      is NOT enabled
                    [1970-01-01 15:19:43.955504] D
                      [socket.c:4170:ssl_setup_connection_params]
                      0-socket.management: SSL support for glusterd is
                      NOT enabled
                    [1970-01-01 15:19:43.955612] D
                      [name.c:572:server_fill_address_family]
                      0-socket.management: option address-family not
                      specified, defaulting to inet6
                    [1970-01-01 15:19:43.955928] D
                      [rpc-transport.c:269:rpc_transport_load]
                      0-rpc-transport: attempt to load file
                      /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so
                    [1970-01-01 15:19:43.956079] E
                      [rpc-transport.c:273:rpc_transport_load]
                      0-rpc-transport:
                      /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so:
                      cannot open shared object file: No such file or
                      directory
                    [1970-01-01 15:19:43.956177] W
                      [rpc-transport.c:277:rpc_transport_load]
                      0-rpc-transport: volume 'rdma.management':
                      transport-type 'rdma' is not valid or not found on
                      this machine
                    [1970-01-01 15:19:43.956270] W
                      [rpcsvc.c:1789:rpcsvc_create_listener]
                      0-rpc-service: cannot create listener, initing the
                      transport failed
                    [1970-01-01 15:19:43.956362] E [MSGID: 106244]
                      [glusterd.c:1798:init] 0-management: creation of 1
                      listeners failed, continuing with succeeded
                      transport
                    [1970-01-01 15:19:43.956459] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GlusterD
                      svc peer, Num: 1238437, Ver: 2, Port: 0
                    [1970-01-01 15:19:43.956561] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GlusterD
                      svc cli read-only, Num: 1238463, Ver: 2, Port: 0
                    [1970-01-01 15:19:43.956666] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GlusterD
                      svc mgmt, Num: 1238433, Ver: 2, Port: 0
                    [1970-01-01 15:19:43.956758] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GlusterD
                      svc mgmt v3, Num: 1238433, Ver: 3, Port: 0
                    [1970-01-01 15:19:43.956853] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: Gluster
                      Portmap, Num: 34123456, Ver: 1, Port: 0
                    [1970-01-01 15:19:43.956946] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: Gluster
                      Handshake, Num: 14398633, Ver: 2, Port: 0
                    [1970-01-01 15:19:43.957062] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: Gluster
                      MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
                    [1970-01-01 15:19:43.957205] D
                      [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC
                      service inited.
                    [1970-01-01 15:19:43.957303] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GF-DUMP,
                      Num: 123451501, Ver: 1, Port: 0
                    [1970-01-01 15:19:43.957408] D
                      [rpc-transport.c:269:rpc_transport_load]
                      0-rpc-transport: attempt to load file
                      /usr/lib64/glusterfs/5.0/rpc-transport/socket.so
                    [1970-01-01 15:19:43.957563] D
                      [socket.c:4424:socket_init] 0-socket.management:
                      disabling nodelay
                    [1970-01-01 15:19:43.957650] D
                      [socket.c:4464:socket_init] 0-socket.management:
                      Configued transport.tcp-user-timeout=0
                    [1970-01-01 15:19:43.957738] D
                      [socket.c:4482:socket_init] 0-socket.management:
                      Reconfigued transport.keepalivecnt=9
                    [1970-01-01 15:19:43.957830] D
                      [socket.c:4167:ssl_setup_connection_params]
                      0-socket.management: SSL support on the I/O path
                      is NOT enabled
                    [1970-01-01 15:19:43.957922] D
                      [socket.c:4170:ssl_setup_connection_params]
                      0-socket.management: SSL support for glusterd is
                      NOT enabled
                    [1970-01-01 15:19:43.958186] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: GlusterD
                      svc cli, Num: 1238463, Ver: 2, Port: 0
                    [1970-01-01 15:19:43.958280] D
                      [rpcsvc.c:2146:rpcsvc_program_register]
                      0-rpc-service: New program registered: Gluster
                      Handshake (CLI Getspec), Num: 14398633, Ver: 2,
                      Port: 0
                    [1970-01-01 15:19:43.958461] D [MSGID: 0]
                      [glusterd-utils.c:7878:glusterd_sm_tr_log_init]
                      0-glusterd: returning 0
                    [1970-01-01 15:19:43.958557] D [MSGID: 0]
                      [glusterd.c:1875:init] 0-management: cannot get
                      run-with-valgrind value
                    [1970-01-01 15:19:43.960895] E [MSGID: 101032]
                      [store.c:447:gf_store_handle_retrieve] 0-: Path
                      corresponding to /var/lib/glusterd/glusterd.info. [No
                      such file or directory]
                    [1970-01-01 15:19:43.961016] D [MSGID: 0]
                      [store.c:452:gf_store_handle_retrieve] 0-:
                      Returning -1
                    [1970-01-01 15:19:43.961108] D [MSGID: 0]
                      [glusterd-store.c:2169:glusterd_retrieve_op_version]
                      0-management: Unable to get store handle!
                    [1970-01-01 15:19:43.961216] E [MSGID: 101032]
                      [store.c:447:gf_store_handle_retrieve] 0-: Path
                      corresponding to /var/lib/glusterd/glusterd.info. [No
                      such file or directory]
                    [1970-01-01 15:19:43.961325] D [MSGID: 0]
                      [store.c:452:gf_store_handle_retrieve] 0-:
                      Returning -1
                    [1970-01-01 15:19:43.961428] D [MSGID: 0]
                      [glusterd-store.c:2345:glusterd_retrieve_uuid]
                      0-management: Unable to get storehandle!
                    [1970-01-01 15:19:43.961523] D [MSGID: 0]
                      [glusterd-store.c:2366:glusterd_retrieve_uuid]
                      0-management: Returning -1
                    [1970-01-01 15:19:43.961617] I [MSGID: 106514]
[glusterd-store.c:2304:glusterd_restore_op_version] 0-management:
                      Detected new install. Setting op-version to
                      maximum : 50000
                    [1970-01-01 15:19:43.962658] D [MSGID: 0]
                      [store.c:432:gf_store_handle_new] 0-: Returning 0
                    [1970-01-01 15:19:43.962750] D [MSGID: 0]
                      [store.c:452:gf_store_handle_retrieve] 0-:
                      Returning 0
                    [1970-01-01 15:19:43.963047] D [MSGID: 0]
                      [store.c:515:gf_store_iter_new] 0-: Returning with
                      0
                    [1970-01-01 15:19:43.963194] D [MSGID: 0]
                      [store.c:632:gf_store_iter_get_next] 0-: Returning
                      with 0
                    [1970-01-01 15:19:43.963318] D [MSGID: 0]
                      [store.c:632:gf_store_iter_get_next] 0-: Returning
                      with -1
                    [1970-01-01 15:19:43.963455] D [MSGID: 0]
                      [store.c:473:gf_store_handle_destroy] 0-:
                      Returning 0
                    [1970-01-01 15:19:43.963757] D [MSGID: 0]
                      [glusterd-store.c:3546:glusterd_store_retrieve_volumes]
                      0-management: Returning with 0
                    [1970-01-01 15:19:43.964159] D [MSGID: 0]
                      [glusterd-store.c:4662:glusterd_store_retrieve_peers]
                      0-management: Returning with 0
                    [1970-01-01 15:19:43.964471] I [MSGID: 106194]
[glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list]
                      0-management: No missed snaps list.
                    [1970-01-01 15:19:43.964580] D [MSGID: 0]
                      [glusterd-store.c:4104:glusterd_store_retrieve_snaps]
                      0-management: Returning with 0
                    [1970-01-01 15:19:43.964680] D [MSGID: 0]
                      [glusterd-store.c:4894:glusterd_restore]
                      0-management: Returning 0
                    [1970-01-01 15:19:43.965060] D [MSGID: 0]
                      [options.c:1225:xlator_option_init_int32]
                      0-management: option event-threads using set value
                      1
                    Final graph:
                    +------------------------------------------------------------------------------+
                      1: volume management
                      2:     type mgmt/glusterd
                      3:     option rpc-auth.auth-glusterfs on
                      4:     option rpc-auth.auth-unix on
                      5:     option rpc-auth.auth-null on
                      6:     option rpc-auth-allow-insecure on
                      7:     option transport.listen-backlog 10
                      8:     option event-threads 1
                      9:     option ping-timeout 0
                     10:     option transport.socket.read-fail-log
                      off
                     11:     option
                      transport.socket.keepalive-interval 2
                     12:     option transport.socket.keepalive-time
                      10
                     13:     option transport-type rdma
                     14:     option working-directory
                      /var/lib/glusterd
                     15: end-volume
                     16:  
                    +------------------------------------------------------------------------------+
                    [1970-01-01 15:19:43.966808] I [MSGID: 101190]
                      [event-epoll.c:622:event_dispatch_epoll_worker]
                      0-epoll: Started thread with index 1
                    [1970-01-01 15:19:44.840454] E
                      [rpcsvc.c:513:rpcsvc_request_create]
                      0-rpc-service: RPC version not supported (XID:
                      0x0, Ver: 0, Program: 0, ProgVers: 0, Proc: 2)
                      from trans (socket.management)
                    [1970-01-01 15:19:44.840884] D
                      [rpcsvc.c:1416:rpcsvc_error_reply] (-->
                      /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4]
                      (-->
                      /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4]
                      (-->
                      /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00]
                      (-->
                      /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224]
                      (-->
                      /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c]
                      ))))) 0-: sending a RPC error reply
                    [1970-01-01 15:19:44.841055] D
                      [logging.c:1805:gf_log_flush_extra_msgs]
                      0-logging-infra: Log buffer size reduced. About to
                      flush 5 extra log messages
                    [1970-01-01 15:19:44.841156] D
                      [logging.c:1808:gf_log_flush_extra_msgs]
                      0-logging-infra: Just flushed 5 extra log messages
                    pending frames:
                    patchset: git://git.gluster.org/glusterfs.git
                    signal received: 11
                    time of crash: 
                    1970-01-01 15:19:44
                    configuration details:
                    argp 1
                    backtrace 1
                    dlfcn 1
                    libpthread 1
                    llistxattr 1
                    setfsid 1
                    spinlock 1
                    epoll.h 1
                    xattr.h 1
                    st_atim.tv_nsec 1
                    package-string: glusterfs 5.0
                    /usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4]
                    /usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0]
                    ./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec]
                    linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478]
                    /lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc]
                    /lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c]
                    /lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4]
                    /lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0]
                    /lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120]
                    /usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c]
                    /usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c]
                    /usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614]
                    /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00]
                    /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224]
                    /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c]
                    /usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c]
                    /usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c]
                    /usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc]
                    /lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30]
                    /lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c]
                    ---------
                    Segmentation fault (core dumped)
                    

                    Could you  please help me, what actually the
                      problem?
                    

                    -- 

                    
                        Regards

                        Abhishek Paliwal

                      
              _______________________________________________

              Gluster-users mailing list

              Gluster-users@xxxxxxxxxxx

              https://lists.gluster.org/mailman/listinfo/gluster-users
          
          
          -- 

          
              Thanks,

              
              Sanju

            
      -- 

      
          Regards

          Abhishek Paliwal

        
      _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
    
  
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users