Re: 389 Deadlock

Rich Megginson <rmeggins@xxxxxxxxxx> · Mon, 22 Jul 2013 09:18:13 -0600



    On 07/22/2013 09:16 AM, Jeffrey Dunham
      wrote:

    
        Uploaded the stacktrace as specified by http://port389.org/wiki/FAQ#Debugging_Hangs

        
        I will look into upgrading to 1.2.11, but it would be to know if
        the issue will actually be fixed there.  Does anything stand out
        here?

      
    No.  Not sure what happened, but this stack trace is useless.  Looks
    like you don't have the right versions of the debuginfo packages. 
    Please check the directions at
    http://port389.org/wiki/FAQ#Debugging_Crashes and Debugging_Hangs

    
        On Mon, Jul 22, 2013 at 8:02 AM, Rich
          Megginson <rmeggins@xxxxxxxxxx>
          wrote:

          
                On 07/22/2013 08:46 AM, Jeffrey Dunham wrote:

                
                  Here's thread 31: 

                    (gdb) thread 31

                    [Switching to thread 31 (Thread 0x2b0e92901940 (LWP
                    10739))]#0  0x00002b0df2847654 in __lll_lock_wait ()
                    from /lib64/libpthread.so.0

                    (gdb) bt

                    #0  0x00002b0df2847654 in __lll_lock_wait () from
                    /lib64/libpthread.so.0

                    #1  0x00002b0df2842f80 in _L_lock_1233 () from
                    /lib64/libpthread.so.0

                    #2  0x00002b0df2842f03 in pthread_mutex_lock () from
                    /lib64/libpthread.so.0

                    #3  0x00002b0df2204289 in PR_Lock () from
                    /usr/lib64/libnspr4.so

                    #4  0x0000000000411e88 in ber_sockbuf_free ()

                    #5  0x00002b0df04d652a in ?? () from
                    /usr/lib64/dirsrv/libslapd.so.0

                    #6  0x00002b0df04d6f1e in send_ldap_result_ext ()
                    from /usr/lib64/dirsrv/libslapd.so.0

                    #7  0x00002b0df04d7421 in send_ldap_result () from
                    /usr/lib64/dirsrv/libslapd.so.0

                    #8  0x00002b0df04c58c4 in slapi_send_ldap_result ()
                    from /usr/lib64/dirsrv/libslapd.so.0

                    #9  0x000000000042524e in ?? ()

                    #10 0x00002b0df04c70f7 in plugin_call_exop_plugins
                    () from /usr/lib64/dirsrv/libslapd.so.0

                    #11 0x000000000041841c in ber_sockbuf_free ()

                    #12 0x00000000004137ad in ber_sockbuf_free ()

                    #13 0x00002b0df22098cd in ?? () from
                    /usr/lib64/libnspr4.so

                    #14 0x00002b0df284083d in start_thread () from
                    /lib64/libpthread.so.0

                    #15 0x00002b0df2b2af8d in clone () from
                    /lib64/libc.so.6

                  
              For hangs/deadlocks, please follow this:

              
              http://port389.org/wiki/FAQ#Debugging_Hangs

              
              1.2.11 has improved deadlock detection and handling
              
                
                      On Mon, Jul 22, 2013 at
                        7:42 AM, Ludwig Krispenz <lkrispen@xxxxxxxxxx>
                        wrote:

                        
                           could
                            you add the bt for thread 31 ?
                            
                              
                                On 07/22/2013 03:58 PM, Jeffrey
                                  Dunham wrote:

                                
                                    Hello,

                                      
                                      Starting a day ago we have been
                                      noticing deadlocks on select 389
                                      servers.  It does not happen on
                                      all of them (10% of servers have
                                      been affected currently, and no
                                      repeat offenders).

                                      
                                    Just looking for any hints on
                                      where to look or what could be
                                      causing this issue.

                                    
                                    Version: 389-Directory/1.2.10.14

                                      
                                      From GDB:

                                      0x00002b0df2847654 in
                                      __lll_lock_wait () from
                                      /lib64/libpthread.so.0

                                      (gdb) info threads

                                        40 Thread 0x2b0e86c91940 (LWP
                                      10689)  0x00002b0df2b24122 in
                                      select () from /lib64/libc.so.6

                                        39 Thread 0x2b0e87692940 (LWP
                                      10690)  0x00002b0df2b24122 in
                                      select () from /lib64/libc.so.6

                                        38 Thread 0x2b0e88093940 (LWP
                                      10691)  0x00002b0df2b24122 in
                                      select () from /lib64/libc.so.6

                                        37 Thread 0x2b0e88a94940 (LWP
                                      10692)  0x00002b0df2b24122 in
                                      select () from /lib64/libc.so.6

                                        36 Thread 0x2b0e89495940 (LWP
                                      10734)  0x00002b0df2845019 in pthread_cond_wait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        35 Thread 0x2b0e89e96940 (LWP
                                      10735)  0x00002b0df2845019 in pthread_cond_wait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        34 Thread 0x2b0e8a897940 (LWP
                                      10736)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        33 Thread 0x2b0e8b298940 (LWP
                                      10737)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        32 Thread 0x2b0e91f00940 (LWP
                                      10738)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        31 Thread 0x2b0e92901940 (LWP
                                      10739)  0x00002b0df2847654 in
                                      __lll_lock_wait () from
                                      /lib64/libpthread.so.0

                                        30 Thread 0x2b0e93302940 (LWP
                                      10740)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        29 Thread 0x2b0e93d03940 (LWP
                                      10741)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        28 Thread 0x2b0e94704940 (LWP
                                      10742)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        27 Thread 0x2b0e95105940 (LWP
                                      10743)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        26 Thread 0x2b0e95b06940 (LWP
                                      10744)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        25 Thread 0x2b0e96507940 (LWP
                                      10745)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        24 Thread 0x2b0e96f08940 (LWP
                                      10746)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        23 Thread 0x2b0e97909940 (LWP
                                      10747)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        22 Thread 0x2b0e9830a940 (LWP
                                      10748)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        21 Thread 0x2b0e98d0b940 (LWP
                                      10749)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        20 Thread 0x2b0e9970c940 (LWP
                                      10750)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        19 Thread 0x2b0e9a10d940 (LWP
                                      10751)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        18 Thread 0x2b0e9ab0e940 (LWP
                                      10752)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        17 Thread 0x2b0e9b50f940 (LWP
                                      10753)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        16 Thread 0x2b0e9bf10940 (LWP
                                      10754)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        15 Thread 0x2b0e9c911940 (LWP
                                      10755)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        14 Thread 0x2b0e9d312940 (LWP
                                      10756)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        13 Thread 0x2b0e9dd13940 (LWP
                                      10757)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        12 Thread 0x2b0e9e714940 (LWP
                                      10758)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        11 Thread 0x2b0e9f115940 (LWP
                                      10759)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        10 Thread 0x2b0e9fb16940 (LWP
                                      10760)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        9 Thread 0x2b0ea0517940 (LWP
                                      10761)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        8 Thread 0x2b0ea0f18940 (LWP
                                      10762)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        7 Thread 0x2b0ea1919940 (LWP
                                      10763)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        6 Thread 0x2b0ea231a940 (LWP
                                      10764)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        5 Thread 0x2b0ea2d1b940 (LWP
                                      10765)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        4 Thread 0x2b0ea371c940 (LWP
                                      10766)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        3 Thread 0x2b0ea411d940 (LWP
                                      10767)  0x00002b0df2845280 in pthread_cond_timedwait@@GLIBC_2.3.2
                                      () from /lib64/libpthread.so.0

                                        2 Thread 0x2b0ea4b1e940 (LWP
                                      10768)  0x00002b0df2b24122 in
                                      select () from /lib64/libc.so.6

                                      * 1 Thread 0x2b0df452a7e0 (LWP
                                      10666)  0x00002b0df2847654 in
                                      __lll_lock_wait () from
                                      /lib64/libpthread.so.0

                                      (gdb) bt

                                      #0  0x00002b0df2847654 in
                                      __lll_lock_wait () from
                                      /lib64/libpthread.so.0

                                      #1  0x00002b0df2842f80 in
                                      _L_lock_1233 () from
                                      /lib64/libpthread.so.0

                                      #2  0x00002b0df2842f03 in
                                      pthread_mutex_lock () from
                                      /lib64/libpthread.so.0

                                      #3  0x00002b0df2204289 in PR_Lock
                                      () from /usr/lib64/libnspr4.so

                                      #4  0x0000000000415955 in
                                      ber_sockbuf_free ()

                                      #5  0x00000000004166f1 in
                                      ber_sockbuf_free ()

                                      #6  0x000000000041d7d4 in ?? ()

                                      #7  0x00002b0df2a739c4 in
                                      __libc_start_main () from
                                      /lib64/libc.so.6

                                      #8  0x000000000040c9b9 in
                                      ber_sockbuf_free ()

                                      #9  0x00007fff5cd57a98 in ?? ()

                                      #10 0x0000000000000000 in ?? ()

                                      
                                  --
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
                                
                            
                          --

                          389 users mailing list

                          389-users@xxxxxxxxxxxxxxxxxxxxxxx

                          https://admin.fedoraproject.org/mailman/listinfo/389-users

                        
                    --
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
                  
                  
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users