Re: Crash in glusterd!!!

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

[2017-12-01 14:10:13.060193] I [MSGID: 100030] [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd       version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)

From the above I can see you're running a way too old gluster version which has been EOLed. Please upgrade your cluster to the latest supported versions.

On Wed, Dec 6, 2017 at 5:39 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
I hope these logs were sufficient... please let me know if you require more logs.

On Wed, Dec 6, 2017 at 3:26 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi Atin,

Please find the backtrace and logs files attached here.

Also, below are the BT from core.

(gdb) bt

#0  0x00003fff8834b898 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#1  0x00003fff88350fd0 in __GI_abort () at abort.c:89

 

[**ALERT: The abort() might not be exactly invoked from the following function line.

                If the trail function contains multiple abort() calls, then you should cross check by other means to get correct abort() call location.

                This is due to the optimized compilation which hides the debug info for multiple abort() calls in a given function.

                Refer TR HU16995 for more information]

 

#2  0x00003fff8838be04 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175

#3  0x00003fff8839aba8 in malloc_printerr (action="" out>, str=0x3fff8847e498 "double free or corruption (!prev)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5007

#4  0x00003fff8839ba40 in _int_free (av=0x3fff6c000020, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3868

#5  0x00003fff885e0814 in __gf_free (free_ptr=0x3fff6c045da0) at mem-pool.c:336

#6  0x00003fff849093c4 in glusterd_friend_sm () at glusterd-sm.c:1295

#7  0x00003fff84901a58 in __glusterd_handle_incoming_unfriend_req (req=0x3fff8481c06c) at glusterd-handler.c:2606

#8  0x00003fff848fb870 in glusterd_big_locked_handler (req=0x3fff8481c06c, actor_fn=@0x3fff84a43e70: 0x3fff84901830 <__glusterd_handle_incoming_unfriend_req>) at glusterd-handler.c:83

#9  0x00003fff848fbd08 in glusterd_handle_incoming_unfriend_req (req=<optimized out>) at glusterd-handler.c:2615

#10 0x00003fff8854e87c in rpcsvc_handle_rpc_call (svc=0x10062fd0 <_GLOBAL__sub_I__ZN27UehChSwitchFachToDchC_ActorC2EP12RTControllerP10RTActorRef()+1148>, trans=<optimized out>, msg=0x3fff6c000920) at rpcsvc.c:705

#11 0x00003fff8854eb7c in rpcsvc_notify (trans=0x3fff74002210, mydata=<optimized out>, event=<optimized out>, data="" out>) at rpcsvc.c:799

#12 0x00003fff885514fc in rpc_transport_notify (this=<optimized out>, event=<optimized out>, data="" out>) at rpc-transport.c:546

#13 0x00003fff847fcd44 in socket_event_poll_in (this=this@entry=0x3fff74002210) at socket.c:2236

#14 0x00003fff847ff89c in socket_event_handler (fd=<optimized out>, idx=<optimized out>, data="" poll_in=<optimized out>, poll_out=<optimized out>, poll_err=<optimized out>) at socket.c:2349

#15 0x00003fff88616874 in event_dispatch_epoll_handler (event=0x3fff83d9d6a0, event_pool=0x10045bc0 <_GLOBAL__sub_I__ZN29DrhIfRhControlPdrProxyC_ActorC2EP12RTControllerP10RTActorRef()+116>) at event-epoll.c:575

#16 event_dispatch_epoll_worker (data="" <main_thread_func__()+1756>) at event-epoll.c:678

#17 0x00003fff884cfb10 in start_thread (arg=0x3fff83d9e160) at pthread_create.c:339

#18 0x00003fff88419c0c in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

 

(gdb) bt full

#0  0x00003fff8834b898 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

        r4 = 1560

        r7 = 16

        arg2 = 1560

        r5 = 6

        r8 = 0

        arg3 = 6

        r0 = 250

        r3 = 0

        r6 = 8

        arg1 = 0

        sc_err = <optimized out>

        sc_ret = <optimized out>

        pd = 0x3fff83d9e160

        pid = 0

---Type <return> to continue, or q <return> to quit---

        selftid = 1560

#1  0x00003fff88350fd0 in __GI_abort () at abort.c:89

        save_stage = 2

        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}

        sigs = {__val = {32, 0 <repeats 15 times>}}

 

[**ALERT: The abort() might not be exactly invoked from the following function line.

                If the trail function contains multiple abort() calls, then you should cross check by other means to get correct abort() call location.

                This is due to the optimized compilation which hides the debug info for multiple abort() calls in a given function.

                Refer TR HU16995 for more information]

 

#2  0x00003fff8838be04 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175

        ap = <optimized out>

        fd = <optimized out>

        _on_2_ = <optimized out>

        list = <optimized out>

        nlist = <optimized out>

        cp = <optimized out>

        written = <optimized out>

#3  0x00003fff8839aba8 in malloc_printerr (action="" out>, str=0x3fff8847e498 "double free or corruption (!prev)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5007

        buf = "00003fff6c045d60"

        cp = <optimized out>

        ar_ptr = <optimized out>

        ptr = <optimized out>

        str = 0x3fff8847e498 "double free or corruption (!prev)"

        action = "">

#4  0x00003fff8839ba40 in _int_free (av=0x3fff6c000020, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3868

        size = <optimized out>

        fb = <optimized out>

        nextchunk = <optimized out>

        nextsize = <optimized out>

        nextinuse = <optimized out>

        prevsize = <optimized out>

        bck = <optimized out>

        fwd = <optimized out>

        errstr = <optimized out>

        locked = <optimized out>

        __func__ = "_int_free"

#5  0x00003fff885e0814 in __gf_free (free_ptr=0x3fff6c045da0) at mem-pool.c:336

        ptr = 0x3fff6c045d60

        mem_acct = <optimized out>

        header = 0x3fff6c045d60

        free_ptr = 0x3fff6c045da0

#6  0x00003fff849093c4 in glusterd_friend_sm () at glusterd-sm.c:1295

        event = 0x3fff6c045da0

        tmp = 0x3fff6c045da0

        ret = <optimized out>

---Type <return> to continue, or q <return> to quit---

        handler = @0x3fff84a44038: 0x3fff84906750 <glusterd_ac_friend_remove>

        state = 0x3fff84a390c0 <glusterd_state_befriended>

        peerinfo = <optimized out>

        event_type = GD_FRIEND_EVENT_REMOVE_FRIEND

        is_await_conn = <optimized out>

        quorum_action = <optimized out>

        old_state = GD_FRIEND_STATE_BEFRIENDED

        this = <optimized out>

        priv = 0x3fff84748050

        __FUNCTION__ = "glusterd_friend_sm"

#7  0x00003fff84901a58 in __glusterd_handle_incoming_unfriend_req (req=0x3fff8481c06c) at glusterd-handler.c:2606

        ret = 0

        friend_req = {uuid = "\231\214R¦\177\223I\216\236Õ\214dÎöy¡", hostname = 0x3fff6c028ef0 "", port = 0, vols = {vols_len = 0, vols_val = 0x0}}

        remote_hostname = "10.32.0.48", '\000' <repeats 98 times>

        __FUNCTION__ = "__glusterd_handle_incoming_unfriend_req"

#8  0x00003fff848fb870 in glusterd_big_locked_handler (req=0x3fff8481c06c, actor_fn=@0x3fff84a43e70: 0x3fff84901830 <__glusterd_handle_incoming_unfriend_req>) at glusterd-handler.c:83

        priv = 0x3fff84748050

        ret = -1

#9  0x00003fff848fbd08 in glusterd_handle_incoming_unfriend_req (req=<optimized out>) at glusterd-handler.c:2615

No locals.

#10 0x00003fff8854e87c in rpcsvc_handle_rpc_call (svc=0x10062fd0 <_GLOBAL__sub_I__ZN27UehChSwitchFachToDchC_ActorC2EP12RTControllerP10RTActorRef()+1148>, trans=<optimized out>, msg=0x3fff6c000920) at rpcsvc.c:705

        actor = 0x3fff84a38860 <gd_svc_peer_actors+192>

        actor_fn = @0x3fff84a43ab0: 0x3fff848fbcf0 <glusterd_handle_incoming_unfriend_req>

        req = 0x3fff8481c06c

        ret = -1

        port = <optimized out>

        unprivileged = <optimized out>

        reply = <optimized out>

        drc = <optimized out>

        __FUNCTION__ = "rpcsvc_handle_rpc_call"

#11 0x00003fff8854eb7c in rpcsvc_notify (trans=0x3fff74002210, mydata=<optimized out>, event=<optimized out>, data="" out>) at rpcsvc.c:799

        ret = -1

        msg = <optimized out>

        new_trans = 0x0

        svc = <optimized out>

        listener = 0x0

        __FUNCTION__ = "rpcsvc_notify"

#12 0x00003fff885514fc in rpc_transport_notify (this=<optimized out>, event=<optimized out>, data="" out>) at rpc-transport.c:546

        ret = -1

        __FUNCTION__ = "rpc_transport_notify"

#13 0x00003fff847fcd44 in socket_event_poll_in (this=this@entry=0x3fff74002210) at socket.c:2236

        ret = <optimized out>

        pollin = 0x3fff6c000920

        priv = 0x3fff74002d50

#14 0x00003fff847ff89c in socket_event_handler (fd=<optimized out>, idx=<optimized out>, data="" poll_in=<optimized out>, poll_out=<optimized out>, poll_err=<optimized out>) at socket.c:2349

---Type <return> to continue, or q <return> to quit---

        this = 0x3fff74002210

        priv = 0x3fff74002d50

        ret = <optimized out>

        __FUNCTION__ = "socket_event_handler"

#15 0x00003fff88616874 in event_dispatch_epoll_handler (event=0x3fff83d9d6a0, event_pool=0x10045bc0 <_GLOBAL__sub_I__ZN29DrhIfRhControlPdrProxyC_ActorC2EP12RTControllerP10RTActorRef()+116>) at event-epoll.c:575

        handler = @0x3fff8481a620: 0x3fff847ff6f0 <socket_event_handler>

        gen = 1

        slot = 0x100803f0 <_GLOBAL__sub_I__ZN24RoamIfFroRrcRoExtAttribDC2Ev()+232>

        data = "" out>

        ret = -1

        fd = 8

        ev_data = 0x3fff83d9d6a8

        idx = 7

#16 event_dispatch_epoll_worker (data="" <main_thread_func__()+1756>) at event-epoll.c:678

        event = {events = 1, data = "" = 0x700000001, fd = 7, u32 = 7, u64 = 30064771073}}

        ret = <optimized out>

        ev_data = 0x100bb4a0 <main_thread_func__()+1756>

        event_pool = 0x10045bc0 <_GLOBAL__sub_I__ZN29DrhIfRhControlPdrProxyC_ActorC2EP12RTControllerP10RTActorRef()+116>

        myindex = <optimized out>

        timetodie = 0

        __FUNCTION__ = "event_dispatch_epoll_worker"

#17 0x00003fff884cfb10 in start_thread (arg=0x3fff83d9e160) at pthread_create.c:339

        pd = 0x3fff83d9e160

        now = <optimized out>

        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-6868946778599096053, 70366736145408, -6868946778421678961, 0, 0, 70366652919808, 70366661304864, 8388608, 70366736105504, 269202592, 70367897957648, 70366736131032, 70366737568040, 3, 0, 70366736131048, 70367897957296, 70367897957352, 4001536, 70366736106520, 70366661302080, -3187654076, 0 <repeats 42 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = "" = 0x0, cleanup = 0x0, canceltype = 0}}}

        not_first_call = <optimized out>

        pagesize_m1 = <optimized out>

        sp = <optimized out>

        freesize = <optimized out>

        __PRETTY_FUNCTION__ = "start_thread"

#18 0x00003fff88419c0c in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

No locals.


Regards,
Abhishek

On Wed, Dec 6, 2017 at 3:21 PM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:
Without the glusterd log file and the core file or the backtrace I can't comment anything.

On Wed, Dec 6, 2017 at 3:09 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Any suggestion....

On Dec 6, 2017 11:51, "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> wrote:
Hi Team,

We are getting the crash in glusterd after start of it. When I tried to debug in brick logs we are getting below errors:

[2017-12-01 14:10:14.684122] E [MSGID: 100018] [glusterfsd.c:1960:glusterfs_pidfile_update] 0-glusterfsd: pidfile /system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid lock failed [Resource temporarily unavailable]
:
:
:
[2017-12-01 14:10:16.862903] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=18: key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:16.862985] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 92: FTRUNCATE 1 (934f08b7-e3b5-4690-84fc-742a4b1fb78b)==> (No space left on device) [No space left on device]
[2017-12-01 14:10:16.907037] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:16.907108] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 35: FTRUNCATE 0 (109d6537-a1ec-4556-8ce1-04c365c451eb)==> (No space left on device) [No space left on device]
[2017-12-01 14:10:16.947541] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:16.947623] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 70: FTRUNCATE 0 (8f9c8054-b0d7-4b93-a95b-cd3ab249c56d)==> (No space left on device) [No space left on device]
[2017-12-01 14:10:16.968515] E [MSGID: 113001] [posix.c:4616:_posix_remove_xattr] 0-c_glusterfs-posix: removexattr failed on /opt/lvmdir/c2/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/configuration (for trusted.glusterfs.dht) [No space left on device]
[2017-12-01 14:10:16.968589] I [MSGID: 115058] [server-rpc-fops.c:740:server_removexattr_cbk] 0-c_glusterfs-server: 90: REMOVEXATTR <gfid:a240d2fd-869c-408d-9b95-62ee1bff074e> (a240d2fd-869c-408d-9b95-62ee1bff074e) of key  ==> (No space left on device) [No space left on device]
[2017-12-01 14:10:17.039815] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:17.039900] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 152: FTRUNCATE 0 (d67bcfcd-ff19-4b58-9823-46d6cce9ace3)==> (No space left on device) [No space left on device]
[2017-12-01 14:10:17.048767] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:17.048874] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 163: FTRUNCATE 0 (0e3ee6ad-408b-4fcf-a1a7-4262ec113316)==> (No space left on device) [No space left on device]
[2017-12-01 14:10:17.075007] E [MSGID: 113001] [posix.c:4616:_posix_remove_xattr] 0-c_glusterfs-posix: removexattr failed on /opt/lvmdir/c2/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/java (for trusted.glusterfs.dht) [No space left on device]

Also, we are having the lack disk space.

Could any one please explain me what glusterd is doing in brick so that it is causing of its crash.

Please find the brick logs in attachment.

Thanks in advance!!!
--
Regards
Abhishek Paliwal

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel




--




Regards
Abhishek Paliwal



--




Regards
Abhishek Paliwal

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel
[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux