On Mon, Dec 3, 2018 at 8:25 AM Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx> wrote:
On Sat, Dec 1, 2018 at 11:02 AM Milind Changire <mchangir@xxxxxxxxxx> wrote:failed brick-mux-regression job:patch:Does this happen only with the above patch? Does brick-mux regression succeed on current master without this patch? Wondering whether the parallelism introduced by bumping up event-threads to 2, is opening up some races in multiplexed environment (though there were always more than one event-thread when more than one brick is multiplexed).
Also, is this bug locally reproducible on your setup if you run test following test with brick-mux enabled (with and without your patch)?
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
_______________________________________________stack trace:$ gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/glfs_epoll000-964.core' ./build/install/sbin/glusterfsd
GNU gdb (GDB) Fedora 8.2-4.fc29
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/install/sbin/glusterfsd...done.
[New LWP 970]
[New LWP 992]
[New LWP 993]
[New LWP 1005]
[New LWP 1241]
[New LWP 964]
[New LWP 968]
[New LWP 996]
[New LWP 995]
[New LWP 994]
[New LWP 967]
[New LWP 969]
[New LWP 1003]
[New LWP 1181]
[New LWP 1242]
[New LWP 966]
[New LWP 965]
[New LWP 999]
[New LWP 1000]
[New LWP 1002]
[New LWP 989]
[New LWP 990]
[New LWP 991]
[New LWP 971]
warning: Ignoring non-absolute filename: <./lib64/libz.so.1>
Missing separate debuginfo for ./lib64/libz.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/ea/8e45dc8e395cc5e26890470112d97a1f1e0b65.debug
warning: Ignoring non-absolute filename: <./lib64/libuuid.so.1>
Missing separate debuginfo for ./lib64/libuuid.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/71/de190dc0c93504abacc17b9747cd772a1e4b0d.debug
warning: Ignoring non-absolute filename: <./lib64/libm.so.6>
Missing separate debuginfo for ./lib64/libm.so.6
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f4/cae74047f9aa2d5a71fdec67c4285d75753eba.debug
warning: Ignoring non-absolute filename: <./lib64/librt.so.1>
Missing separate debuginfo for ./lib64/librt.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d3/3989ec31efe745eb0d3b68a92d19e77d7ddfda.debug
warning: Ignoring non-absolute filename: <./lib64/libdl.so.2>
Missing separate debuginfo for ./lib64/libdl.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/5c/db5a56336e7e2bd14ffa189411e44a834afcd8.debug
warning: Ignoring non-absolute filename: <./lib64/libpthread.so.0>
Missing separate debuginfo for ./lib64/libpthread.so.0
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f4/c04bce85d2d269d0a2af4972fc69805b50345b.debug
warning: Expected absolute pathname for libpthread in the inferior, but got ./lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Ignoring non-absolute filename: <./lib64/libcrypto.so.10>
Missing separate debuginfo for ./lib64/libcrypto.so.10
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/67/ceb4edd36bfe0eb31cd92da2694aca5377a599.debug
warning: Ignoring non-absolute filename: <./lib64/libc.so.6>
Missing separate debuginfo for ./lib64/libc.so.6
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/cb/4b7554d1adbef2f001142dd6f0a5139fc9aa69.debug
warning: Ignoring non-absolute filename: <./lib64/ld-linux-x86-64.so.2>
Missing separate debuginfo for ./lib64/ld-linux-x86-64.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d2/66b1f6650927e18108323bcca8f7b68e68eb92.debug
warning: Ignoring non-absolute filename: <./lib64/libssl.so.10>
Missing separate debuginfo for ./lib64/libssl.so.10
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/64/68a4e28a19cdd885a3cbc30e009589ca4c2e92.debug
warning: Ignoring non-absolute filename: <./lib64/libgssapi_krb5.so.2>
Missing separate debuginfo for ./lib64/libgssapi_krb5.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/16/fe0dc6cefc5f444bc876516d02efe9cc2d432f.debug
warning: Ignoring non-absolute filename: <./lib64/libkrb5.so.3>
Missing separate debuginfo for ./lib64/libkrb5.so.3
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d1/cd1b94855a85fbc735c745db39bc096f7d8cc3.debug
warning: Ignoring non-absolute filename: <./lib64/libcom_err.so.2>
Missing separate debuginfo for ./lib64/libcom_err.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/2c/7ef64ef0c5af8bcfa8f9e628e5605a7d8c52d3.debug
warning: Ignoring non-absolute filename: <./lib64/libk5crypto.so.3>
Missing separate debuginfo for ./lib64/libk5crypto.so.3
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/a2/0f715c514b3ea873f4cc77d585a50cb670e266.debug
warning: Ignoring non-absolute filename: <./lib64/libkrb5support.so.0>
Missing separate debuginfo for ./lib64/libkrb5support.so.0
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9f/82b06ce44724a8ace827a2c95a9a76619ea314.debug
warning: Ignoring non-absolute filename: <./lib64/libkeyutils.so.1>
Missing separate debuginfo for ./lib64/libkeyutils.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/2e/01d5ac08c1280d013aab96b292ac58bc30a263.debug
warning: Ignoring non-absolute filename: <./lib64/libresolv.so.2>
Missing separate debuginfo for ./lib64/libresolv.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/2b/dc2b6ff0b2c204cce34d139a9eada0272eb070.debug
warning: Ignoring non-absolute filename: <./lib64/libselinux.so.1>
Missing separate debuginfo for ./lib64/libselinux.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/1b/0c0dc4f9054c5922082921c1447739f27ac8f4.debug
warning: Ignoring non-absolute filename: <./lib64/libpcre.so.1>
Missing separate debuginfo for ./lib64/libpcre.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9c/a3d11f018beeb719cdb34be800bf1641350d0a.debug
warning: Ignoring non-absolute filename: <./lib64/libgcc_s.so.1>
Missing separate debuginfo for ./lib64/libgcc_s.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/53/1aa1391ea4e1489d5ef11aa5dc2ffd9e2bdfee.debug
warning: Ignoring non-absolute filename: <./lib64/libnss_files.so.2>
Missing separate debuginfo for ./lib64/libnss_files.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/52/ed84166576539e50b20c4edf9a104c1e04ae45.debug
warning: Ignoring non-absolute filename: <./lib64/libaio.so.1>
Missing separate debuginfo for ./lib64/libaio.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/62/bdce334f05919d9568bf4c44e4bc470dcf8c56.debug
warning: Ignoring non-absolute filename: <./lib64/libacl.so.1>
Missing separate debuginfo for ./lib64/libacl.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/7f/39882fc0b80be53790c2eac307d39f7de1ad6e.debug
warning: Ignoring non-absolute filename: <./lib64/libattr.so.1>
Missing separate debuginfo for ./lib64/libattr.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/26/17ecc6738047e207ae3add990bd6a34d11b265.debug
warning: Expected absolute pathname for libpthread in the inferior, but got ./lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/build/install/sbin/glusterfsd -s builder105.cloud.gluster.org --volfile-id pat'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000040bc33 in STACK_DESTROY (stack=0x3) at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/stack.h:182
182 LOCK(&stack->pool->lock);
[Current thread is 1 (LWP 970)]
(gdb) bt
#0 0x000000000040bc33 in STACK_DESTROY (stack=0x3) at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/stack.h:182
#1 0x000000000041286e in mgmt_pmap_signin_cbk (req=0x7f9af0349768, iov=0x7f9af03497a0, count=1, myframe=0x7f9af0141648)
at /home/jenkins/root/workspace/regression-on-demand-multiplex/glusterfsd/src/glusterfsd-mgmt.c:2829
#2 0x00007f9b0dc5c59e in rpc_clnt_handle_reply (clnt=0x7fa5b0, pollin=0x7f9af0008750)
at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-lib/src/rpc-clnt.c:755
#3 0x00007f9b0dc5cac7 in rpc_clnt_notify (trans=0x7fa910, mydata=0x7fa5e0, event=RPC_TRANSPORT_MSG_RECEIVED, data=""> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-lib/src/rpc-clnt.c:922
#4 0x00007f9b0dc58b23 in rpc_transport_notify (this=0x7fa910, event=RPC_TRANSPORT_MSG_RECEIVED, data=""> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-lib/src/rpc-transport.c:525
#5 0x00007f9b02a6a6a6 in socket_event_poll_in (this=0x7fa910, notify_handled=true)
at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-transport/socket/src/socket.c:2530
#6 0x00007f9b02a6b687 in socket_event_handler (fd=16, idx=8, gen=10, data="" poll_in=1, poll_out=0, poll_err=0)
at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-transport/socket/src/socket.c:2931
#7 0x00007f9b0df146c1 in event_dispatch_epoll_handler (event_pool=0x7b0c30, event=0x7f9b00fa1ea0)
at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/event-epoll.c:591
#8 0x00007f9b0df149ba in event_dispatch_epoll_worker (data=""> at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/event-epoll.c:668
#9 0x00007f9b0cce5e25 in start_thread () from ./lib64/libpthread.so.0
#10 0x00007f9b0c5aebad in clone () from ./lib64/libc.so.6
(gdb) up
#1 0x000000000041286e in mgmt_pmap_signin_cbk (req=0x7f9af0349768, iov=0x7f9af03497a0, count=1, myframe=0x7f9af0141648)
at /home/jenkins/root/workspace/regression-on-demand-multiplex/glusterfsd/src/glusterfsd-mgmt.c:2829
2829 STACK_DESTROY(frame->root);
(gdb) p *frame
$1 = {root = 0x3, parent = 0x300000001, frames = {next = 0xffffffffffffffff, prev = 0x7f9af01416a8}, local = 0x7f9af0003958, this = 0x0,
ret = 0x0, ref_count = 0, lock = {spinlock = 0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
__spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x7f9af0003958}},
__size = '\000' <repeats 32 times>, "X9\000\360\232\177\000", __align = 0}}, cookie = 0x0, complete = 232, op = 32666, begin = {
tv_sec = 0, tv_nsec = 140303428253256}, end = {tv_sec = 140303428244368, tv_nsec = 245890744},
wind_from = 0x1 <error: Cannot access memory at address 0x1>, wind_to = 0x0, unwind_from = 0x0, unwind_to = 0x0}
(gdb)
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-devel
