Re: Glusterfsd crashed with SIGSEGV

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi Abhishek,

Few more questions,


On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi Amar,

Below are the requested logs

pabhishe@arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so
not a dynamic executable

pabhishe@arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so
not a dynamic executable


Can you please add a * at the end, so it gets the linked library list from the actual files (ideally this is a symlink, but I expected it to resolve like in Fedora).

 
root@128:/# gdb /usr/sbin/glusterd core.1099 
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-wrs-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/glusterd...(no debugging symbols found)...done.
[New LWP 1109]
[New LWP 1101]
[New LWP 1105]
[New LWP 1110]
[New LWP 1099]
[New LWP 1107]
[New LWP 1119]
[New LWP 1103]
[New LWP 1112]
[New LWP 1116]
[New LWP 1104]
[New LWP 1239]
[New LWP 1106]
[New LWP 1111]
[New LWP 1108]
[New LWP 1117]
[New LWP 1102]
[New LWP 1118]
[New LWP 1100]
[New LWP 1114]
[New LWP 1113]
[New LWP 1115]

warning: Could not load shared library symbols for linux-vdso64.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id gv0.128.224.95.140.tmp-bric'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00003fffb76a6d48 in _int_malloc (av=av@entry=0x3fffa8000020, bytes=bytes@entry=36) at malloc.c:3327
3327 {
[Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))]
(gdb) bt full

This is backtrace of one particular thread. I need output of command

(gdb) thread apply all bt full


Also, considering this is a crash in the malloc library call itself, would like to know the details of OS, Kernel version and gcc versions.
 
Regards,
Amar

#0  0x00003fffb76a6d48 in _int_malloc (av=av@entry=0x3fffa8000020, bytes=bytes@entry=36) at malloc.c:3327
        nb = <optimized out>
        idx = <optimized out>
        bin = <optimized out>
        victim = <optimized out>
        size = <optimized out>
        victim_index = <optimized out>
        remainder = <optimized out>
        remainder_size = <optimized out>
        block = <optimized out>
        bit = <optimized out>
        map = <optimized out>
        fwd = <optimized out>
        bck = <optimized out>
        errstr = 0x0
        __func__ = "_int_malloc"
#1  0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921
        ar_ptr = 0x3fffa8000020
        victim = <optimized out>
        hook = <optimized out>
        __func__ = "__libc_malloc"
#2  0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len=<optimized out>) at xdr_sizeof.c:89
        len = 36
        xdrs = 0x3fffb1686d20
#3  0x00003fffb7842488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#4  0x00003fffb7842e84 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#5  0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81099f0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j"
        stat = <optimized out>
#6  0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81099f0, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#7  0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#8  0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8109870, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271"
        stat = <optimized out>
#9  0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8109870, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81096f0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa81097a0 "\241X\372!\216\256=\342"
        stat = <optimized out>
---Type <return> to continue, or q <return> to quit---
#12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81096f0, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8109570, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa8109620 "\265\205\003Vu'\002L"
        stat = <optimized out>
#15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8109570, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81093f0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa81094a0 "\200L\027F'\177\366D"
        stat = <optimized out>
#18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81093f0, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8109270, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa8109320 "\217{dK(\001E\220"
        stat = <optimized out>
#21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8109270, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81090f0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005"
        stat = <optimized out>
#24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81090f0, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8108f70, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa8109020 "\260.\025\b\244\352IT"
        stat = <optimized out>
#27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8108f70, obj_size=<optimized out>, 
    xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
        more_data = 1
#28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
No symbol table info available.
#29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8108df0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
        loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\"
---Type <return> to continue, or q <return> to quit---


Regards,
Abhishek

On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan <atumball@xxxxxxxxxx> wrote:
Hi Abhishek,

Can you check and get back to us?

```
bash# ldd /usr/lib64/libglusterfs.so
bash# ldd /usr/lib64/libgfrpc.so

```

Also considering you have the core, can you do `(gdb) thr apply all bt full`  and pass it on?

Thanks & Regards,
Amar

On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi Team,

COuld you please provide some pointer to debug it further.

Regards,
Abhishek

On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
Hi Team,

I am using Glusterfs 5.4, where after setting the gluster mount point when trying to access it, glusterfsd is getting crashed and mount point through the "Transport endpoint is not connected error.

Here I are the gdb log for the core file

warning: Could not load shared library symbols for linux-vdso64.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id gv0.128.224.95.140.tmp-bric'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00003fff95ab1d48 in _int_malloc (av=av@entry=0x3fff7c000020, bytes=bytes@entry=36) at malloc.c:3327
3327 {
[Current thread is 1 (Thread 0x3fff90394160 (LWP 811))]
(gdb) 
(gdb) 
(gdb) bt
#0  0x00003fff95ab1d48 in _int_malloc (av=av@entry=0x3fff7c000020, bytes=bytes@entry=36) at malloc.c:3327
#1  0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at malloc.c:2921
#2  0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, len=<optimized out>) at xdr_sizeof.c:89
#3  0x00003fff95c4d488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0
#4  0x00003fff95c4de84 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#5  0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c132020, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#6  0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c132020, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#7  0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#8  0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131ea0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#9  0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131ea0, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131d20, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131d20, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131ba0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131ba0, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131a20, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131a20, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c1318a0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c1318a0, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131720, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131720, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c1315a0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c1315a0, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131420, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131420, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135
#31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0
#32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c1312a0, size=<optimized out>, proc=<optimized out>) at xdr_ref.c:84
#33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c1312a0, obj_size=<optimized out>, 
    xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135

Frames are getting repeated, could any one please me.
--
Regards
Abhishek Paliwal


--




Regards
Abhishek Paliwal
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
Amar Tumballi (amarts)


--




Regards
Abhishek Paliwal


--




Regards
Abhishek Paliwal


--
Amar Tumballi (amarts)
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux