On Mon, Dec 20, 2010 at 10:37:56AM +0100, Jiri Denemark wrote: > in your first email, you said you can't even run virsh list while migration is > running, right? yup -- when i start my migration using my python application, 'virsh list' no longer works from the command-line. i will try setting up a simple test-case i can use with 'virsh migrate' and let you know the results. > So could you make sure you have debug info libvirt packages installed > or in case you compile libvirt yourself that you compile with -g > (should be there by default) and don't strip binaries. I am using the rpm packages distributed from the website, which I've installed on my debian box using alien. I've attached two gdb traces -- one for when a migration is just running, and another where I've also got a blocked 'virsh list' from the command line. It appears there's no difference between them (except in the data counts in the thread actually doing the migration). I've also attached a trace for the 'virsh' process itself. Thanks for answering, hopefully we can get to the bottom of this. --Igor
Thread 7 (Thread 0x7f6358f33710 (LWP 24325)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00007f635cb560e9 in _L_lock_953 () from /lib/libpthread.so.0
#2 0x00007f635cb55f0b in __pthread_mutex_lock (mutex=0x7f6350015910) at pthread_mutex_lock.c:61
#3 0x000000000043d44d in qemuDriverLock (conn=0x7f635009b8f0) at qemu/qemu_driver.c:144
#4 qemudClose (conn=0x7f635009b8f0) at qemu/qemu_driver.c:4500
#5 0x00007f636054a97b in virReleaseConnect (conn=0x7f635009b8f0) at datatypes.c:269
#6 0x00007f636054b028 in virUnrefConnect (conn=0x7f635009b8f0) at datatypes.c:321
#7 0x00007f63605525c8 in virConnectClose (conn=0x7f635009b8f0) at libvirt.c:1548
#8 0x000000000041941f in qemudFreeClient (opaque=0xc636f0) at libvirtd.c:2284
#9 qemudRunLoop (opaque=0xc636f0) at libvirtd.c:2358
#10 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#11 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7f634effd710 (LWP 24476)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74ab0) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Current language: auto
The current source language is "auto; currently asm".
Thread 5 (Thread 0x7f634e7fc710 (LWP 24478)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74ac8) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7f634dffb710 (LWP 24489)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74ae0) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f634d7fa710 (LWP 24494)):
#0 0x00007f635cb5b0bd in read () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f636051683e in read (fd=54, buf=0x13bddb0, count=32768) at /usr/include/bits/unistd.h:45
#2 saferead (fd=54, buf=0x13bddb0, count=32768) at util/util.c:95
#3 0x0000000000447e03 in doTunnelSendAll (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0,
dom_xml=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0,
resource=0, dom=<value optimized out>) at qemu/qemu_driver.c:11389
#4 doTunnelMigrate (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0, dom_xml=<value optimized out>,
uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0, dom=<value optimized out>)
at qemu/qemu_driver.c:11604
#5 0x0000000000448631 in doPeer2PeerMigrate (dom=0xc63970, cookie=<value optimized out>,
cookielen=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0,
resource=0) at qemu/qemu_driver.c:11746
#6 qemudDomainMigratePerform (dom=0xc63970, cookie=<value optimized out>, cookielen=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0)
at qemu/qemu_driver.c:11816
#7 0x00007f6360555b16 in virDomainMigratePerform (domain=0xc63970, cookie=0x0, cookielen=0,
uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, bandwidth=0) at libvirt.c:3931
#8 0x0000000000425782 in remoteDispatchDomainMigratePerform (server=<value optimized out>,
client=<value optimized out>, conn=0x7f63501472a0, hdr=<value optimized out>, rerr=0x7f634d7f9be0,
args=0x7f634d7f9d20, ret=0x7f634d7f9cc0) at remote.c:1852
#9 0x000000000042b417 in remoteDispatchClientCall (server=0xc636f0, client=0xc66fc0, msg=0xf6d650)
at dispatch.c:530
#10 remoteDispatchClientRequest (server=0xc636f0, client=0xc66fc0, msg=0xf6d650) at dispatch.c:408
#11 0x000000000041aaa8 in qemudWorker (data=0xc74af8) at libvirtd.c:1586
#12 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#13 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f635672e710 (LWP 24853)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74a68) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f6360ce07e0 (LWP 24324)):
#0 0x00007f635cb54be5 in pthread_join (threadid=140064670824208, thread_return=0x0) at pthread_join.c:89
#1 0x000000000041d6dd in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:3304
Current language: auto
The current source language is "auto; currently c".
Thread 7 (Thread 0x7f6358f33710 (LWP 24325)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00007f635cb560e9 in _L_lock_953 () from /lib/libpthread.so.0
#2 0x00007f635cb55f0b in __pthread_mutex_lock (mutex=0x7f6350015910) at pthread_mutex_lock.c:61
#3 0x000000000043d44d in qemuDriverLock (conn=0x7f635009b8f0) at qemu/qemu_driver.c:144
#4 qemudClose (conn=0x7f635009b8f0) at qemu/qemu_driver.c:4500
#5 0x00007f636054a97b in virReleaseConnect (conn=0x7f635009b8f0) at datatypes.c:269
#6 0x00007f636054b028 in virUnrefConnect (conn=0x7f635009b8f0) at datatypes.c:321
#7 0x00007f63605525c8 in virConnectClose (conn=0x7f635009b8f0) at libvirt.c:1548
#8 0x000000000041941f in qemudFreeClient (opaque=0xc636f0) at libvirtd.c:2284
#9 qemudRunLoop (opaque=0xc636f0) at libvirtd.c:2358
#10 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#11 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7f634effd710 (LWP 24476)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74ab0) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Current language: auto
The current source language is "auto; currently asm".
Thread 5 (Thread 0x7f634e7fc710 (LWP 24478)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74ac8) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7f634dffb710 (LWP 24489)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74ae0) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f634d7fa710 (LWP 24494)):
#0 0x00007f635cb5b0bd in read () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f636051683e in read (fd=54, buf=0x13b5db0, count=65536) at /usr/include/bits/unistd.h:45
#2 saferead (fd=54, buf=0x13b5db0, count=65536) at util/util.c:95
#3 0x0000000000447e03 in doTunnelSendAll (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0,
dom_xml=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0,
resource=0, dom=<value optimized out>) at qemu/qemu_driver.c:11389
#4 doTunnelMigrate (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0, dom_xml=<value optimized out>,
uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0, dom=<value optimized out>)
at qemu/qemu_driver.c:11604
#5 0x0000000000448631 in doPeer2PeerMigrate (dom=0xc63970, cookie=<value optimized out>,
cookielen=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0,
resource=0) at qemu/qemu_driver.c:11746
#6 qemudDomainMigratePerform (dom=0xc63970, cookie=<value optimized out>, cookielen=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0)
at qemu/qemu_driver.c:11816
#7 0x00007f6360555b16 in virDomainMigratePerform (domain=0xc63970, cookie=0x0, cookielen=0,
uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, bandwidth=0) at libvirt.c:3931
#8 0x0000000000425782 in remoteDispatchDomainMigratePerform (server=<value optimized out>,
client=<value optimized out>, conn=0x7f63501472a0, hdr=<value optimized out>, rerr=0x7f634d7f9be0,
args=0x7f634d7f9d20, ret=0x7f634d7f9cc0) at remote.c:1852
#9 0x000000000042b417 in remoteDispatchClientCall (server=0xc636f0, client=0xc66fc0, msg=0xf6d650)
at dispatch.c:530
#10 remoteDispatchClientRequest (server=0xc636f0, client=0xc66fc0, msg=0xf6d650) at dispatch.c:408
#11 0x000000000041aaa8 in qemudWorker (data=0xc74af8) at libvirtd.c:1586
#12 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#13 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f635672e710 (LWP 24853)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
at util/threads-pthread.c:105
#2 0x000000000041aa15 in qemudWorker (data=0xc74a68) at libvirtd.c:1565
#3 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f6360ce07e0 (LWP 24324)):
#0 0x00007f635cb54be5 in pthread_join (threadid=140064670824208, thread_return=0x0) at pthread_join.c:89
#1 0x000000000041d6dd in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:3304
Current language: auto
The current source language is "auto; currently c".
Thread 1 (Thread 0x7f849b1687e0 (LWP 24979)):
#0 0x00007f84975490d8 in *__GI___poll (fds=0x7fff70e53d90, nfds=2, timeout=<value optimized out>)
at ../sysdeps/unix/sysv/linux/poll.c:83
#1 0x00007f849abf68e2 in remoteIOEventLoop (conn=0x171a1f0, priv=0x7f849afb2010, flags=3,
thiscall=0x7f84949d2010) at remote/remote_driver.c:10129
#2 remoteIO (conn=0x171a1f0, priv=0x7f849afb2010, flags=3, thiscall=0x7f84949d2010)
at remote/remote_driver.c:10370
#3 0x00007f849abf731c in call (conn=0x171a1f0, priv=0x7f849afb2010, flags=3, proc_nr=<value optimized out>,
args_filter=0x7f849757d240 <xdr_void>, args=<value optimized out>,
ret_filter=0x7f849ac0c000 <xdr_remote_auth_list_ret>, ret=0x7fff70e54400 "")
at remote/remote_driver.c:10455
#4 0x00007f849ac070ff in remoteAuthenticate (conn=<value optimized out>, priv=0x7f849afb2010,
auth=0x7f849af53340, flags=0) at remote/remote_driver.c:6853
#5 doRemoteOpen (conn=<value optimized out>, priv=0x7f849afb2010, auth=0x7f849af53340, flags=0)
at remote/remote_driver.c:861
#6 0x00007f849ac08ee0 in remoteOpen (conn=0x171a1f0, auth=0x7f849af53340, flags=<value optimized out>)
at remote/remote_driver.c:1103
#7 0x00007f849abd6c19 in do_open (name=0x0, auth=0x7f849af53340, flags=0) at libvirt.c:1259
#8 0x00007f849abd8518 in virConnectOpenAuth (name=0x0, auth=0x7f849af53340, flags=0) at libvirt.c:1514
#9 0x000000000041ee17 in vshInit (argc=<value optimized out>, argv=0x7fff70e54af8) at virsh.c:11167
#10 main (argc=<value optimized out>, argv=0x7fff70e54af8) at virsh.c:11819
Attachment:
signature.asc
Description: Digital signature
