Re: 389 Deadlock

Rich Megginson <rmeggins@xxxxxxxxxx> · Mon, 22 Jul 2013 15:52:04 -0600



    On 07/22/2013 03:38 PM, Jeffrey Dunham
      wrote:

    
          Thanks for the info.

          
          I have loaded 389-ds-base-debuginfo and attached the stack
          trace.  Hopefully this will be more telling.

        
    Excellent.  Yes, very nice.

    This looks like https://fedorahosted.org/389/ticket/47375

    
    This is due to be fixed in 1.2.11.22

    
        Thanks.

      
        On Mon, Jul 22, 2013 at 8:18 AM, Rich
          Megginson <rmeggins@xxxxxxxxxx>
          wrote:

          
                On 07/22/2013 09:16 AM, Jeffrey Dunham wrote:

                
                    Uploaded the stacktrace as specified by http://port389.org/wiki/FAQ#Debugging_Hangs

                    
                    I will look into upgrading to 1.2.11, but it would
                    be to know if the issue will actually be fixed
                    there.  Does anything stand out here?

                  
              No.  Not sure what happened, but this stack trace is
              useless.  Looks like you don't have the right versions of
              the debuginfo packages.  Please check the directions at http://port389.org/wiki/FAQ#Debugging_Crashes
              and Debugging_Hangs
              
                
                      On Mon, Jul 22, 2013 at
                        8:02 AM, Rich Megginson <rmeggins@xxxxxxxxxx>
                        wrote:

                        
                              On 07/22/2013 08:46 AM, Jeffrey
                                Dunham wrote:

                              
                                Here's thread 31: 

                                  (gdb) thread 31

                                  [Switching to thread 31 (Thread
                                  0x2b0e92901940 (LWP 10739))]#0 
                                  0x00002b0df2847654 in __lll_lock_wait
                                  () from /lib64/libpthread.so.0

                                  (gdb) bt

                                  #0  0x00002b0df2847654 in
                                  __lll_lock_wait () from
                                  /lib64/libpthread.so.0

                                  #1  0x00002b0df2842f80 in _L_lock_1233
                                  () from /lib64/libpthread.so.0

                                  #2  0x00002b0df2842f03 in
                                  pthread_mutex_lock () from
                                  /lib64/libpthread.so.0

                                  #3  0x00002b0df2204289 in PR_Lock ()
                                  from /usr/lib64/libnspr4.so

                                  #4  0x0000000000411e88 in
                                  ber_sockbuf_free ()

                                  #5  0x00002b0df04d652a in ?? () from
                                  /usr/lib64/dirsrv/libslapd.so.0

                                  #6  0x00002b0df04d6f1e in
                                  send_ldap_result_ext () from
                                  /usr/lib64/dirsrv/libslapd.so.0

                                  #7  0x00002b0df04d7421 in
                                  send_ldap_result () from
                                  /usr/lib64/dirsrv/libslapd.so.0

                                  #8  0x00002b0df04c58c4 in
                                  slapi_send_ldap_result () from
                                  /usr/lib64/dirsrv/libslapd.so.0

                                  #9  0x000000000042524e in ?? ()

                                  #10 0x00002b0df04c70f7 in
                                  plugin_call_exop_plugins () from
                                  /usr/lib64/dirsrv/libslapd.so.0

                                  #11 0x000000000041841c in
                                  ber_sockbuf_free ()

                                  #12 0x00000000004137ad in
                                  ber_sockbuf_free ()

                                  #13 0x00002b0df22098cd in ?? () from
                                  /usr/lib64/libnspr4.so

                                  #14 0x00002b0df284083d in start_thread
                                  () from /lib64/libpthread.so.0

                                  #15 0x00002b0df2b2af8d in clone ()
                                  from /lib64/libc.so.6

                                
                            For hangs/deadlocks, please follow this:

                            
                            http://port389.org/wiki/FAQ#Debugging_Hangs

                            
                            1.2.11 has improved deadlock detection and
                            handling
                            
                              
                                    On Mon, Jul
                                      22, 2013 at 7:42 AM, Ludwig
                                      Krispenz <lkrispen@xxxxxxxxxx>
                                      wrote:

                                      
                                         could you add
                                          the bt for thread 31 ?
                                          
                                            
                                              On 07/22/2013 03:58
                                                PM, Jeffrey Dunham
                                                wrote:

                                              
                                                  Hello,

                                                    
                                                    Starting a day ago
                                                    we have been
                                                    noticing deadlocks
                                                    on select 389
                                                    servers.  It does
                                                    not happen on all of
                                                    them (10% of servers
                                                    have been affected
                                                    currently, and no
                                                    repeat offenders).

                                                    
                                                  Just looking for
                                                    any hints on where
                                                    to look or what
                                                    could be causing
                                                    this issue.

                                                  
                                                  Version:
                                                    389-Directory/1.2.10.14

                                                    
                                                    From GDB:

                                                    0x00002b0df2847654
                                                    in __lll_lock_wait
                                                    () from
                                                    /lib64/libpthread.so.0

                                                    (gdb) info threads

                                                      40 Thread
                                                    0x2b0e86c91940 (LWP
                                                    10689) 
                                                    0x00002b0df2b24122
                                                    in select () from
                                                    /lib64/libc.so.6

                                                      39 Thread
                                                    0x2b0e87692940 (LWP
                                                    10690) 
                                                    0x00002b0df2b24122
                                                    in select () from
                                                    /lib64/libc.so.6

                                                      38 Thread
                                                    0x2b0e88093940 (LWP
                                                    10691) 
                                                    0x00002b0df2b24122
                                                    in select () from
                                                    /lib64/libc.so.6

                                                      37 Thread
                                                    0x2b0e88a94940 (LWP
                                                    10692) 
                                                    0x00002b0df2b24122
                                                    in select () from
                                                    /lib64/libc.so.6

                                                      36 Thread
                                                    0x2b0e89495940 (LWP
                                                    10734) 
                                                    0x00002b0df2845019
                                                    in pthread_cond_wait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      35 Thread
                                                    0x2b0e89e96940 (LWP
                                                    10735) 
                                                    0x00002b0df2845019
                                                    in pthread_cond_wait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      34 Thread
                                                    0x2b0e8a897940 (LWP
                                                    10736) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      33 Thread
                                                    0x2b0e8b298940 (LWP
                                                    10737) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      32 Thread
                                                    0x2b0e91f00940 (LWP
                                                    10738) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      31 Thread
                                                    0x2b0e92901940 (LWP
                                                    10739) 
                                                    0x00002b0df2847654
                                                    in __lll_lock_wait
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      30 Thread
                                                    0x2b0e93302940 (LWP
                                                    10740) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      29 Thread
                                                    0x2b0e93d03940 (LWP
                                                    10741) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      28 Thread
                                                    0x2b0e94704940 (LWP
                                                    10742) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      27 Thread
                                                    0x2b0e95105940 (LWP
                                                    10743) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      26 Thread
                                                    0x2b0e95b06940 (LWP
                                                    10744) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      25 Thread
                                                    0x2b0e96507940 (LWP
                                                    10745) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      24 Thread
                                                    0x2b0e96f08940 (LWP
                                                    10746) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      23 Thread
                                                    0x2b0e97909940 (LWP
                                                    10747) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      22 Thread
                                                    0x2b0e9830a940 (LWP
                                                    10748) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      21 Thread
                                                    0x2b0e98d0b940 (LWP
                                                    10749) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      20 Thread
                                                    0x2b0e9970c940 (LWP
                                                    10750) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      19 Thread
                                                    0x2b0e9a10d940 (LWP
                                                    10751) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      18 Thread
                                                    0x2b0e9ab0e940 (LWP
                                                    10752) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      17 Thread
                                                    0x2b0e9b50f940 (LWP
                                                    10753) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      16 Thread
                                                    0x2b0e9bf10940 (LWP
                                                    10754) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      15 Thread
                                                    0x2b0e9c911940 (LWP
                                                    10755) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      14 Thread
                                                    0x2b0e9d312940 (LWP
                                                    10756) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      13 Thread
                                                    0x2b0e9dd13940 (LWP
                                                    10757) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      12 Thread
                                                    0x2b0e9e714940 (LWP
                                                    10758) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      11 Thread
                                                    0x2b0e9f115940 (LWP
                                                    10759) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      10 Thread
                                                    0x2b0e9fb16940 (LWP
                                                    10760) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      9 Thread
                                                    0x2b0ea0517940 (LWP
                                                    10761) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      8 Thread
                                                    0x2b0ea0f18940 (LWP
                                                    10762) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      7 Thread
                                                    0x2b0ea1919940 (LWP
                                                    10763) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      6 Thread
                                                    0x2b0ea231a940 (LWP
                                                    10764) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      5 Thread
                                                    0x2b0ea2d1b940 (LWP
                                                    10765) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      4 Thread
                                                    0x2b0ea371c940 (LWP
                                                    10766) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      3 Thread
                                                    0x2b0ea411d940 (LWP
                                                    10767) 
                                                    0x00002b0df2845280
                                                    in pthread_cond_timedwait@@GLIBC_2.3.2
                                                    () from
                                                    /lib64/libpthread.so.0

                                                      2 Thread
                                                    0x2b0ea4b1e940 (LWP
                                                    10768) 
                                                    0x00002b0df2b24122
                                                    in select () from
                                                    /lib64/libc.so.6

                                                    * 1 Thread
                                                    0x2b0df452a7e0 (LWP
                                                    10666) 
                                                    0x00002b0df2847654
                                                    in __lll_lock_wait
                                                    () from
                                                    /lib64/libpthread.so.0

                                                    (gdb) bt

                                                    #0 
                                                    0x00002b0df2847654
                                                    in __lll_lock_wait
                                                    () from
                                                    /lib64/libpthread.so.0

                                                    #1 
                                                    0x00002b0df2842f80
                                                    in _L_lock_1233 ()
                                                    from
                                                    /lib64/libpthread.so.0

                                                    #2 
                                                    0x00002b0df2842f03
                                                    in
                                                    pthread_mutex_lock
                                                    () from
                                                    /lib64/libpthread.so.0

                                                    #3 
                                                    0x00002b0df2204289
                                                    in PR_Lock () from
                                                    /usr/lib64/libnspr4.so

                                                    #4 
                                                    0x0000000000415955
                                                    in ber_sockbuf_free
                                                    ()

                                                    #5 
                                                    0x00000000004166f1
                                                    in ber_sockbuf_free
                                                    ()

                                                    #6 
                                                    0x000000000041d7d4
                                                    in ?? ()

                                                    #7 
                                                    0x00002b0df2a739c4
                                                    in __libc_start_main
                                                    () from
                                                    /lib64/libc.so.6

                                                    #8 
                                                    0x000000000040c9b9
                                                    in ber_sockbuf_free
                                                    ()

                                                    #9 
                                                    0x00007fff5cd57a98
                                                    in ?? ()

                                                    #10
                                                    0x0000000000000000
                                                    in ?? ()

                                                    
                                                --
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
                                              
                                          
                                        --

                                        389 users mailing list

                                        389-users@xxxxxxxxxxxxxxxxxxxxxxx

                                        https://admin.fedoraproject.org/mailman/listinfo/389-users

                                      
                                  --
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
                                
                                
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users