Re: glusterd stuck for glusterfs with version 3.12.15

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

Gluster brick multiplex is not enabled in my env.

My point is that : socket notify another around of poll in too early actually this round of poll in is not finished, thread 9 has not finished poll in while thread 8 has begun another round of poll in, thread 8 has called iobref_unref and iobref_unref does not release

iobref->lock , so thread 9 may block.

 

cynthia

From: Sanju Rakonde <srakonde@xxxxxxxxxx>
Sent: Tuesday, April 09, 2019 3:08 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx>
Cc: Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx>; gluster-devel@xxxxxxxxxxx
Subject: Re: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15

 

Hello,

 

I'm unable to figure out the issue by just looking at the backtrace. You might be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1650115. If you come across the same problem in future, please capture pstack output and share it with us.

 

On Mon, Apr 8, 2019 at 2:31 PM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx> wrote:

Hi,

The env is not there anymore, but I have collected the thread stack trace of glusterd with command thread apply all bt

Using host libthread_db library "/lib64/libthread_db.so.1".

0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0

Missing separate debuginfos, use: dnf debuginfo-install rcp-pack-glusterfs-1.12.0_1_gc999db1-RCP2.wf29.x86_64

(gdb) thread apply all bt

 

Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):

#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0

#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0

#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944

#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123

#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322

#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2471

#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583

#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data="" at event-epoll.c:659

#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):

#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0

#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0

#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201

#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2480

#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583

#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data="" at event-epoll.c:659

#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 7 (Thread 0x7f9ee49b3700 (LWP 1931)):

#0  0x00007f9ee9fd45bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007f9ee5e651b9 in hooks_worker (args=0x1813000) at glusterd-hooks.c:529

#2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 6 (Thread 0x7f9ee692e700 (LWP 1762)):

#0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808e00) at syncop.c:603

#2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808e00) at syncop.c:695

#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 5 (Thread 0x7f9ee712f700 (LWP 1761)):

---Type <return> to continue, or q <return> to quit---

#0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808a40) at syncop.c:603

#2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808a40) at syncop.c:695

#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 4 (Thread 0x7f9ee7930700 (LWP 1760)):

#0  0x00007f9ee98725d0 in nanosleep () from /lib64/libc.so.6

#1  0x00007f9ee98724aa in sleep () from /lib64/libc.so.6

#2  0x00007f9eeb247fdf in pool_sweeper (arg=0x0) at mem-pool.c:481

#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 3 (Thread 0x7f9ee8131700 (LWP 1759)):

#0  0x00007f9ee97e3d7c in sigtimedwait () from /lib64/libc.so.6

#1  0x00007f9ee9fd8bac in sigwait () from /lib64/libpthread.so.0

#2  0x0000000000409ed7 in glusterfs_sigwaiter ()

#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 2 (Thread 0x7f9ee8932700 (LWP 1758)):

#0  0x00007f9ee9fd83b0 in nanosleep () from /lib64/libpthread.so.0

#1  0x00007f9eeb224545 in gf_timer_proc (data="" at timer.c:164

#2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 1 (Thread 0x7f9eeb707780 (LWP 1757)):

#0  0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0

#1  0x00007f9eeb282b09 in event_dispatch_epoll (event_pool=0x17feb00) at event-epoll.c:746

#2  0x00007f9eeb246786 in event_dispatch (event_pool=0x17feb00) at event.c:124

#3  0x000000000040ab95 in main ()

(gdb)

(gdb)

(gdb) q!

A syntax error in _expression_, near `'.

(gdb) quit

 

From: Sanju Rakonde <srakonde@xxxxxxxxxx>
Sent: Monday, April 08, 2019 4:58 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx>
Cc: Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx>; gluster-devel@xxxxxxxxxxx
Subject: Re: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15

 

Can you please capture output of "pstack $(pidof glusterd)" and send it to us? We need to capture this information when glusterd is struck.

 

On Mon, Apr 8, 2019 at 8:05 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx> wrote:

Hi glusterfs experts,

Good day!

In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)

In iobref_destroy, I think it is better to add destroy lock.

Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

 

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

 

* socket: fix issue on concurrent handle of a socket

 

 

 

GDB INFO:

Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think

Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):

#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0

#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0

#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944

#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123

#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322

#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2471

#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583

#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data="" at event-epoll.c:659

#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):

#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0

#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0

#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201

#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2480

#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583

#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data="" at event-epoll.c:659

#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

 

(gdb) thread 9

[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]

#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0

(gdb) bt

#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0

#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0

#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944

#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123

#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322

#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2471

#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583

#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data="" at event-epoll.c:659

#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0

#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) frame 2

#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944

944 iobuf.c: No such file or directory.

(gdb) print *iobref

$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,

        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},

      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000О\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

(gdb) quit

A debugging session is active.

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel


 

--

Thanks,

Sanju


 

--

Thanks,

Sanju

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux