Re: core dump when do "gluster volume quota foo list --xml"

[Raghavendra Talur <raghavendra.talur@xxxxxxxxx>]

Sorry for top posting but I think this might be a different issue.

Dang,

If it was working well for you and started crashing recently after doing a system update,

it might be because of glibc checks.

This is the line from your output which gives the hint

>*** glibc detected *** gluster: double free or corruption (out):

As Niels suggests 3.4.5 is too old and you should upgrade to supported version.

However, if you want a quick workaround for now, you should disable glibc checks

for double free.

I found this stack overflow link for doing that.

http://stackoverflow.com/questions/6750815/how-to-turn-off-glibc-run-time-protections

Let us know if it works.

Thanks,

Raghavendra Talur

On Sat, Jun 13, 2015 at 1:30 PM, Niels de Vos <ndevos@xxxxxxxxxx> wrote:

On Sat, Jun 13, 2015 at 10:24:34AM +0530, Vijaikumar M wrote:
> Hi Dang,
>
> This issue has been fixed in 3.7.0:
> bug: https://bugzilla.redhat.com/show_bug.cgi?id=1185259
> patch: http://review.gluster.org/#/c/9481/

I found https://bugzilla.redhat.com/1200297 which is doing a backport to
3.6.x. Should this not get fixed in 3.5.x too? If it applies there,
please send a backport for it.

Dang, 3.4.x is not getting any updates anymore. We are providing updated
releases for the most current version (3.7) and the two previous ones
(3.6 and 3.5).

HTH,
Niels

>
>
> Thanks,
> Vijay
>
>
> On Thursday 11 June 2015 12:48 PM, Dang Zhiqiang wrote:
> >Hi,
> >
> >    core dump when do "gluster volume quota foo list --xml", how to fix
> >it?
> >
> >env:
> >glusterfs 3.4.5
> >centos 6.5
> >
> >log:
> ># gluster volume quota foo list --xml
> ><?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> ><cliOutput>
> >  <opRet>0</opRet>
> >  <opErrno>115</opErrno>
> >  <opErrstr/>
> >  <volQuota>
> >    <quota>
> >      <path>/music</path>
> >      <limit>50MB</limit>
> >      <size>0Bytes</size>
> >    </quota>
> >    <quota>
> >      <path>/film</path>
> >      <limit>60MB</limit>
> >      <size>9.6MB</size>
> >    </quota>
> >    <quota>
> >      <path>/</path>
> >      <limit>200MB</limit>
> >      <size>19.3MB</size>
> >    </quota>
> >  </volQuota>
> ></cliOutput>
> >*** glibc detected *** gluster: double free or corruption (out):
> >0x000000000203c0c0 ***
> >======= Backtrace: =========
> >/lib64/libc.so.6[0x3d53a75e76]
> >/lib64/libc.so.6[0x3d53a789b3]
> >gluster(cli_xml_output_vol_quota_limit_list+0x141)[0x427161]
> >gluster(gf_cli_quota_cbk+0x3f3)[0x415f93]
> >/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x347f80d6f5]
> >/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11f)[0x347f80ec6f]
> >/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x347f80a4e8]
> >/usr/lib64/glusterfs/3.4.5/rpc-transport/socket.so(+0x9216)[0x7f0d425df216]
> >/usr/lib64/glusterfs/3.4.5/rpc-transport/socket.so(+0xac3d)[0x7f0d425e0c3d]
> >/usr/lib64/libglusterfs.so.0[0x347f45e907]
> >gluster(main+0x355)[0x408f35]
> >/lib64/libc.so.6(__libc_start_main+0xfd)[0x3d53a1ed5d]
> >gluster[0x4083c9]
> >======= Memory map: ========
> >00400000-0043b000 r-xp 00000000 fd:00 40468
> >/usr/sbin/gluster
> >0063b000-0063f000 rw-p 0003b000 fd:00 40468
> >/usr/sbin/gluster
> >01fe0000-0204d000 rw-p 00000000 00:00 0                [heap]
> >347f400000-347f47a000 r-xp 00000000 fd:00 40465
> >/usr/lib64/libglusterfs.so.0.0.0
> >347f47a000-347f67a000 ---p 0007a000 fd:00 40465
> >/usr/lib64/libglusterfs.so.0.0.0
> >347f67a000-347f67c000 rw-p 0007a000 fd:00 40465
> >/usr/lib64/libglusterfs.so.0.0.0
> >347f67c000-347f67f000 rw-p 00000000 00:00 0
> >347f800000-347f814000 r-xp 00000000 fd:00 40466
> >/usr/lib64/libgfrpc.so.0.0.0
> >347f814000-347fa14000 ---p 00014000 fd:00 40466
> >/usr/lib64/libgfrpc.so.0.0.0
> >347fa14000-347fa15000 rw-p 00014000 fd:00 40466
> >/usr/lib64/libgfrpc.so.0.0.0
> >347fc00000-347fc15000 r-xp 00000000 fd:00 40467
> >/usr/lib64/libgfxdr.so.0.0.0
> >347fc15000-347fe15000 ---p 00015000 fd:00 40467
> >/usr/lib64/libgfxdr.so.0.0.0
> >347fe15000-347fe16000 rw-p 00015000 fd:00 40467
> >/usr/lib64/libgfxdr.so.0.0.0
> >3d53200000-3d53220000 r-xp 00000000 fd:00 40276
> >/lib64/ld-2.12.so
> >3d5341f000-3d53420000 r--p 0001f000 fd:00 40276
> >/lib64/ld-2.12.so
> >3d53420000-3d53421000 rw-p 00020000 fd:00 40276
> >/lib64/ld-2.12.so
> >3d53421000-3d53422000 rw-p 00000000 00:00 0
> >3d53600000-3d53602000 r-xp 00000000 fd:00 40285
> >/lib64/libdl-2.12.so
> >3d53602000-3d53802000 ---p 00002000 fd:00 40285
> >/lib64/libdl-2.12.so
> >3d53802000-3d53803000 r--p 00002000 fd:00 40285
> >/lib64/libdl-2.12.so
> >3d53803000-3d53804000 rw-p 00003000 fd:00 40285
> >/lib64/libdl-2.12.so
> >3d53a00000-3d53b8a000 r-xp 00000000 fd:00 40277
> >/lib64/libc-2.12.so
> >3d53b8a000-3d53d8a000 ---p 0018a000 fd:00 40277
> >/lib64/libc-2.12.so
> >3d53d8a000-3d53d8e000 r--p 0018a000 fd:00 40277
> >/lib64/libc-2.12.so
> >3d53d8e000-3d53d8f000 rw-p 0018e000 fd:00 40277
> >/lib64/libc-2.12.so
> >3d53d8f000-3d53d94000 rw-p 00000000 00:00 0
> >3d53e00000-3d53e17000 r-xp 00000000 fd:00 11764
> >/lib64/libpthread-2.12.so
> >3d53e17000-3d54017000 ---p 00017000 fd:00 11764
> >/lib64/libpthread-2.12.so
> >3d54017000-3d54018000 r--p 00017000 fd:00 11764
> >/lib64/libpthread-2.12.so
> >3d54018000-3d54019000 rw-p 00018000 fd:00 11764
> >/lib64/libpthread-2.12.so
> >3d54019000-3d5401d000 rw-p 00000000 00:00 0
> >3d54200000-3d54222000 r-xp 00000000 fd:00 5114
> >/lib64/libncurses.so.5.7
> >3d54222000-3d54421000 ---p 00022000 fd:00 5114
> >/lib64/libncurses.so.5.7
> >3d54421000-3d54422000 rw-p 00021000 fd:00 5114
> >/lib64/libncurses.so.5.7
> >3d54600000-3d54615000 r-xp 00000000 fd:00 40286
> >/lib64/libz.so.1.2.3
> >3d54615000-3d54814000 ---p 00015000 fd:00 40286
> >/lib64/libz.so.1.2.3
> >3d54814000-3d54815000 r--p 00014000 fd:00 40286
> >/lib64/libz.so.1.2.3
> >3d54815000-3d54816000 rw-p 00015000 fd:00 40286
> >/lib64/libz.so.1.2.3
> >3d54a00000-3d54a83000 r-xp 00000000 fd:00 40287
> >/lib64/libm-2.12.so
> >3d54a83000-3d54c82000 ---p 00083000 fd:00 40287
> >/lib64/libm-2.12.so
> >3d54c82000-3d54c83000 r--p 00082000 fd:00 40287
> >/lib64/libm-2.12.so
> >3d54c83000-3d54c84000 rw-p 00083000 fd:00 40287
> >/lib64/libm-2.12.so
> >3d55200000-3d5521d000 r-xp 00000000 fd:00 40290
> >/lib64/libselinux.so.1
> >3d5521d000-3d5541c000 ---p 0001d000 fd:00 40290
> >/lib64/libselinux.so.1
> >3d5541c000-3d5541d000 r--p 0001c000 fd:00 40290
> >/lib64/libselinux.so.1
> >3d5541d000-3d5541e000 rw-p 0001d000 fd:00 40290
> >/lib64/libselinux.so.1
> >3d5541e000-3d5541f000 rw-p 00000000 00:00 0
> >3d55600000-3d55616000 r-xp 00000000 fd:00 12401
> >/lib64/libresolv-2.12.so
> >3d55616000-3d55816000 ---p 00016000 fd:00 12401
> >/lib64/libresolv-2.12.so
> >3d55816000-3d55817000 r--p 00016000 fd:00 12401
> >/lib64/libresolv-2.12.so
> >3d55817000-3d55818000 rw-p 00017000 fd:00 12401
> >/lib64/libresolv-2.12.so
> >3d55818000-3d5581a000 rw-p 00000000 00:00 0
> >3d55a00000-3d55a16000 r-xp 00000000 fd:00 40318
> >/lib64/libgcc_s-4.4.7-20120601.so.1
> >3d55a16000-3d55c15000 ---p 00016000 fd:00 40318
> >/lib64/libgcc_s-4.4.7-20120601.so.1
> >3d55c15000-3d55c16000 rw-p 00015000 fd:00 40318
> >/lib64/libgcc_s-4.4.7-20120601.so.1
> >3d55e00000-3d55e1d000 r-xp 00000000 fd:00 24839
> >/lib64/libtinfo.so.5.7
> >3d55e1d000-3d5601d000 ---p 0001d000 fd:00 24839
> >/lib64/libtinfo.so.5.7
> >3d5601d000-3d56021000 rw-p 0001d000 fd:00 24839
> >/lib64/libtinfo.so.5.7Aborted (core dumped
> >
> >debug:
> >Core was generated by `gluster --mode=script v quota foo list --xml'.
> >Program terminated with signal 6, Aborted.
> >#0  0x0000003d53a32635 in raise () from /lib64/libc.so.6
> >Missing separate debuginfos, use: debuginfo-install
> >glibc-2.12-1.132.el6_5.4.x86_64 keyutils-libs-1.4-4.el6.x86_64
> >krb5-libs-1.10.3-15.el6_5.1.x86_64 libcom_err-1.41.12-18.el6_5.1.x86_64
> >libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64
> >libxml2-2.7.6-14.el6_5.2.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64
> >openssl-1.0.1e-16.el6_5.15.x86_64 readline-6.0-4.el6.x86_64
> >zlib-1.2.3-29.el6.x86_64
> >(gdb) bt
> >#0  0x0000003d53a32635 in raise () from /lib64/libc.so.6
> >#1  0x0000003d53a33e15 in abort () from /lib64/libc.so.6
> >#2  0x0000003d53a70547 in __libc_message () from /lib64/libc.so.6
> >#3  0x0000003d53a75e76 in malloc_printerr () from /lib64/libc.so.6
> >#4  0x0000003d53a789b3 in _int_free () from /lib64/libc.so.6
> >#5  0x0000000000427161 in cli_xml_output_vol_quota_limit_list
> >(volname=<value optimized out>, limit_list=0xcc3bb0
> >"/music:50MB,/film:60MB,/:200MB", op_ret=30,
> >    op_errno=115, op_errstr=<value optimized out>) at
> >cli-xml-output.c:2843
> >#6  0x0000000000415f93 in gf_cli_quota_cbk (req=<value optimized out>,
> >iov=<value optimized out>, count=<value optimized out>, myframe=<value
> >optimized out>)
> >    at cli-rpc-ops.c:2487
> >#7  0x000000347f80d6f5 in rpc_clnt_handle_reply (clnt=0xcb3b10,
> >pollin=0xcc3d00) at rpc-clnt.c:771
> >#8  0x000000347f80ec6f in rpc_clnt_notify (trans=<value optimized out>,
> >mydata=0xcb3b40, event=<value optimized out>, data="" optimized out>)
> >at rpc-clnt.c:891
> >#9  0x000000347f80a4e8 in rpc_transport_notify (this=<value optimized
> >out>, event=<value optimized out>, data="" optimized out>) at
> >rpc-transport.c:497
> >#10 0x00007fa4f14fe216 in socket_event_poll_in (this=0xcbd130) at
> >socket.c:2118
> >#11 0x00007fa4f14ffc3d in socket_event_handler (fd=<value optimized out>,
> >idx=<value optimized out>, data="" poll_in=1, poll_out=0,
> >poll_err=0) at socket.c:2230
> >#12 0x000000347f45e907 in event_dispatch_epoll_handler
> >(event_pool=0xc8be90) at event-epoll.c:384
> >#13 event_dispatch_epoll (event_pool=0xc8be90) at event-epoll.c:445
> >#14 0x0000000000408f35 in main (argc=<value optimized out>, argv=<value
> >optimized out>) at cli.c:620
> >(gdb) l *0x0000000000427161
> >0x427161 is in cli_xml_output_vol_quota_limit_list
> >(cli-xml-output.c:2844).
> >2839
> >2840       ret = cli_end_xml_output (writer, doc);
> >2841
> >2842out:

> >*2843        GF_FREE (size_str);*

> >2844       gf_log ("cli", GF_LOG_DEBUG, "Returning %d", ret);
> >2845       return ret;
> >2846#else
> >2847       return 0;
> >2848#endif
> >
> >
> >
> >_______________________________________________
> >Gluster-users mailing list
> >Gluster-users@xxxxxxxxxxx
> >http://www.gluster.org/mailman/listinfo/gluster-users
>

> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

--

Raghavendra Talur 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users