Re: Libvirt 4.2.0 hang on destination on live migration cancel

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

For anyone who might find this thread from online searching, I was able to determine this commit fixes the above issue:

https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=dddcb601ebf97ef222a03bb27b2357e831e8a0cc


On Tue, Oct 2, 2018 at 8:42 AM Scott Sullivan <scottgregorysullivan@xxxxxxxxx> wrote:
I'm trying the following command on the source machine:

virsh migrate --live --copy-storage-all  --verbose TEST qemu+ssh://10.30.76.66/system

If I ssh into the destination machine when this command is running, I can see NBD copying data as expected, and if I wait long enough it completes and succeeds. 

However if I ctrl+c this command above before it completes, it causes virsh commands (like virsh list) to just hang seemingly forever on the destination machine.

Here's the libvirt debug entries when I run virsh list in this state:

[root@host ~]# export LIBVIRT_DEBUG=1 
[root@host ~]# virsh list                                    
2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register drivers
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddbb60 name=Test 
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering Test as driver 0 
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddc660 name=ESX
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote               
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering remote as driver 2
2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterDefaultImpl:280 : registering default event implementation
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 : Used 0 handle slots, adding at least 10 more
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0                                                                                                
2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil) ff=(nil)                               
2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 : addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500 removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660 updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 : Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0
2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 : EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380 opaque=0x7ffe5b15e700 ff=(nil)
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 : name=<null>, auth=0x7f70c1ddea00, flags=0x0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a3f0 classname=virConnect                                                              2018-10-02 05:14:05.054+0000: 16208: debug : virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop
2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation
2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 : Loading config file '/etc/libvirt/libvirt.conf'
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 1
2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 : filename=/etc/libvirt/libvirt.conf
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1
2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6
2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 : Get value string (nil) 0
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 : no name, allowing driver auto-select
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 0 (Test) ...
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 0 Test returned DECLINED
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 1 (ESX) ...
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 1 ESX returned DECLINED
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 2 (remote) ...
2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 : Auto-probe remote URI
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding with name =
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 : Connecting with transport 1
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : Proceeding with sockname /var/run/libvirt/libvirt-sock
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:641 : path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null>
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:705 : connect() succeeded
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 : localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83aa20 classname=virNetSocket
2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 : RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ae20 classname=virNetClient
2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 : RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83aa20
2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting
2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0 opaque=0x5559fa83aa20 ff=0x7f70c19b5320
2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 : client=0x5559fa83ae20, interval=-1, count=0
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b120 classname=virKeepAlive
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s)
2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 : RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b340 classname=virConnectCloseCallbackData
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b340
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b040 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ac00 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a9b0 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b040
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ac00
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83a9b0
2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying authentication
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=1 d=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 : msg=0x5559fa83b720 tracked=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageEncodePayload:389 : Encode length as 28
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=2 timeout=-1
2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 : RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 vers=1 proc=66 type=0 status=0 serial=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : New call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 dispatch=(nil)
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0
2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=2 events=0
2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s)
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=0 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1

And here's some information from gbd when its in this state:

[root@host ~]# ps axf | grep libvirtd
17010 pts/2    S+     0:00          \_ grep libvirtd
15555 ?        Sl     0:00 libvirtd --daemon
[root@host ~]# gdb -p $(pgrep libvirtd)

(gdb) t a a bt                                                                             
                                                  
Thread 17 (Thread 0x2ae648a10700 (LWP 15559)):                                             
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                              
                                             
Thread 16 (Thread 0x2ae648c11700 (LWP 15560)):                                    
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0             
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                       
Thread 15 (Thread 0x2ae648e12700 (LWP 15561)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                       
Thread 14 (Thread 0x2ae649013700 (LWP 15562)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                                                             
Thread 13 (Thread 0x2ae649214700 (LWP 15563)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0        
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                                                            
Thread 12 (Thread 0x2ae649415700 (LWP 15564)):
---Type <return> to continue, or q <return> to quit---                                 
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                       
                                      
Thread 11 (Thread 0x2ae649616700 (LWP 15565)):                                 
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                       
---Type <return> to continue, or q <return> to quit---
                                                                                   
Thread 10 (Thread 0x2ae649817700 (LWP 15566)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0            
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                                                                   
Thread 9 (Thread 0x2ae649a18700 (LWP 15567)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
---Type <return> to continue, or q <return> to quit---                        
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                     
                                    
Thread 8 (Thread 0x2ae649c19700 (LWP 15568)):                                 
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                            
                                           
Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)):                                       
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0              
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                                                                      
Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0                 
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6
                                                                                
Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)):
---Type <return> to continue, or q <return> to quit---                                     
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                             
                                            
Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)):                                       
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6                                   
---Type <return> to continue, or q <return> to quit---                           

Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)):
#0  0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)):
#0  0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0
#1  0x00002ae63e9caef0 in udev_monitor_receive_device () from /lib64/libudev.so.0
#2  0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized out>) at node_device/node_device_udev.c:1623
#3  0x00002ae63b2c0075 in virThreadHelper (data="" optimized out>) at util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)):
#0  0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized out>, fd=19, events=<value optimized out>, data="" optimized out>) at node_device/node_device_udev.c:1666
#4  0x00002ae63b257564 in virEventPollDispatchHandles () at util/vireventpoll.c:508
#5  virEventPollRunOnce () at util/vireventpoll.c:657
#6  0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327
#7  0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at rpc/virnetdaemon.c:858
#8  0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value optimized out>) at remote/remote_daemon.c:1475
(gdb)

This issue I know is fixed in 4.7.0 because I run this version on my CentOS 7 machines and don't have this issue. However on my CentOS 6 machines I have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on CentOS 6 is troublesome though, because of the requirement of gnutls >= 3.2.

I'm hoping with the above output someone might be able to point me in the right direction to a commit that resolved this between 4.2.0 and 4.7.0, or a possible way to prevent this hang in 4.2.0 as described above ?

Thanks in advance
_______________________________________________
libvirt-users mailing list
libvirt-users@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvirt-users
[Index of Archives]     [Virt Tools]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux