В письме от Четверг, 20-июн-2013 13:25:06 пользователь Michal Privoznik написал: > On 20.06.2013 12:56, Alexandr wrote: > > On 20.06.2013 10:44, Michal Privoznik wrote: > >> On 20.06.2013 09:40, Александр wrote: > >>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали: > >>>> On 19.06.2013 21:24, Александр wrote: > >>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik > >>> > >>> написал: > >>>>>> On 18.06.2013 22:21, Александр wrote: > >>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal > >>>>>>> Privoznik > >>>>>>> > >>>>>>> написал: > >>>>>>>> On 18.06.2013 07:14, Александр wrote: > >>>>>>>>> good day, i am trying to create domain via virDomainDefineXML. > >>>>>>>>> and my > >>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon with > >>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and > >>>>>>>>> restarting > >>>>>>>>> it solveing hang (i mean after restart i able to connect with > >>>>>>>>> virt-manager), i using following domain xml > >>>>>>>>> > >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> > >>>>>>>>> <domain type="kvm"> > >>>>>>>>> > >>>>>>>>> <tittle>Automatically created domain with default > >>>>>>>>> configuration</tittle> > >>>>>>>>> <name>vm1</name> > >>>>>>>>> <description>no description</description> > >>>>>>>>> <os> > >>>>>>>>> > >>>>>>>>> <type>hvm</type> > >>>>>>>>> > >>>>>>>>> <boot dev="cdrom"/> > >>>>>>>>> <boot dev="hd"/> > >>>>>>>>> > >>>>>>>>> </os> > >>>>>>>>> > >>>>>>>>> <vcpu>1</vcpu> > >>>>>>>>> <memory unit="MB">512</memory> > >>>>>>>>> <cpu model="host-model"> > >>>>>>>>> > >>>>>>>>> <topology sockets="1" cores="1" threads="1"/> > >>>>>>>>> > >>>>>>>>> </cpu> > >>>>>>>>> > >>>>>>>>> <clock offset="localtime"/> > >>>>>>>>> <devices> > >>>>>>>>> > >>>>>>>>> <graphics type="vnc" autoport="yes" password="admin"/> > >>>>>>>>> > >>>>>>>>> <channel type="unix"> > >>>>>>>>> > >>>>>>>>> <source mode="bind" path="/var/run/vm1.sock"/> > >>>>>>>>> > >>>>>>>>> <target type="virtio" name="org.guest_agent.0"/> > >>>>>>>>> > >>>>>>>>> </channel> > >>>>>>>>> > >>>>>>>>> </devices> > >>>>>>>>> > >>>>>>>>> </domain> > >>>>>>>>> > >>>>>>>>> i have this calls on stack > >>>>>>>>> > >>>>>>>>> 0 0x00007ffff48c1750 __poll_nocancel > >>>>>>>>> 1 0x00007ffff649e5ca virNetClientIOEventLoop > >>>>>>>> > >>>>>>>> This is just probably a client waiting for the corresponding > >>>>>>>> libvirtd > >>>>>>>> to > >>>>>>>> provide a reply message. Are you able to get libvirtd debug logs > >>>>>>>> so we > >>>>>>>> can see what's going on? > >>>>>>>> > >>>>>>>> http://wiki.libvirt.org/page/DebugLogs > >>>>>>>> > >>>>>>>> BTW, what's the libvirt version you are running? > >>>>>>>> > >>>>>>>> Michal > >>>>>>>> > >>>>>>>>> 2 0x00007ffff649ed96 virNetClientSendInternal > >>>>>>>>> 3 0x00007ffff64a0443 virNetClientSendWithReply > >>>>>>>>> 4 0x00007ffff64a0daa virNetClientProgramCall > >>>>>>>>> 5 0x00007ffff6478566 callFull.isra.2 > >>>>>>>>> 6 0x00007ffff6487fa2 remoteDomainDefineXML > >>>>>>>>> 7 0x00007ffff6449d2f virDomainDefineXML > >>>>>>>>> 8 0x0000000000426941 virt_domain::create > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 > >>>>>>>>> 9 0x000000000040c1a9 main > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 > >>>>>>>>> > >>>>>>>>> what i am doing wrong ? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> libvirt-users mailing list > >>>>>>>>> libvirt-users@xxxxxxxxxx > >>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users > >>>>>>> > >>>>>>> i have libvirt version 1.0.6, debug log in attachment. > >>>>>> > >>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is > >>>>>> stuck. Can you attach to the libvirtd while it's being stuck and > >>>>>> run 't > >>>>>> a a bt' and post the result? I wonder if we have a deadlock > >>>>>> somewhere. > >>>>>> > >>>>>> Michal > >>>>> > >>>>> if i correctly understand you, you need debugger backtrace after > >>>>> libvirtd > >>>>> stuck ?, if so, here is bt from gdb > >>>> > >>>> You understand correctly. However, simple 'bt' doesn't show what's > >>>> going > >>>> on in other threads - libvirtd is multi threaded. > >>>> > >>>>> (gdb) bt > >>>>> #0 0x00007f00c1ae876d in poll () from /lib64/libc.so.6 > >>>>> #1 0x00007f00c203cfcb in virEventPollRunOnce () from > >>>>> /usr/lib64/libvirt.so.0 #2 0x00007f00c203ba4d in > >>>>> virEventRunDefaultImpl > >>>>> () from > >>>>> /usr/lib64/libvirt.so.0 > >>>>> #3 0x00007f00c214a5cd in virNetServerRun () from > >>>>> /usr/lib64/libvirt.so.0 > >>>>> #4 0x00007f00c2aea08a in main () > >>>> > >>>> So all we see here is the libvirtd event loop. That's why we need 't > >>>> a a > >>>> bt' which is 'thread apply all backtrace' which will run 'bt' over all > >>>> threads. > >>>> > >>>> Michal > >>> > >>> how can i make it ? > >> > >> The very same way as you did 'bt'. Just instead of typing 'bt' you'll > >> type 't a a bt'. > >> > >> Michal > > > > here is backtrace from all threads: > > > > (gdb) t a a bt > > > > Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)): > > #0 0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0 > > #1 0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0 > > #2 0x00007fdd944b7caa in pthread_mutex_lock () from > > /lib64/libpthread.so.0 > > Bingo! So we've a deadlock here. Somebody has forgotten to unlock a > domain. So as we go over the domain list to check for name (if it exists > already or not), we deadlock. The process goes like this: > > 1) if adding a domain the the internal list (I should write rather hash > table where UUID is the key, and the pointer to the internal structure > representing domain is the value), we search for UUID. Since this is the > key, we don't have to lock anything (besides the hash table itself). > > 2) Okay, no matching UUID. But what about domain name? We go through the > table again, over each element, and: > 2a) lock the element > 2b) compare the names > 2c) unlock the element > > And since a domain withing the table is locked (and all none of the > threads is using it) we deadlock in step 2a) as there's no one to come > and unlock the domain. Sigh. Hopefully, I'll have some time to dig into > the log you've provided and get more precise info. > > > #3 0x00007fdd94773861 in virDomainObjListSearchName () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0 > > #5 0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from > > /usr/lib64/libvirt.so.0 > > #6 0x00007fdd9478cd47 in virDomainObjListAdd () from > > /usr/lib64/libvirt.so.0 > > #7 0x00007fdd875a311c in qemuDomainDefineXML () from > > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > > #8 0x00007fdd947ead2f in virDomainDefineXML () from > > /usr/lib64/libvirt.so.0 > > #9 0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper () > > #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from > > /usr/lib64/libvirt.so.0 > > #11 0x00007fdd94844068 in virNetServerHandleJob () from > > /usr/lib64/libvirt.so.0 > > #12 0x00007fdd94765795 in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > ---Type <return> to continue, or q <return> to quit--- > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 2 (Thread 0x7fdd89379700 (LWP 20330)): > > #0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 1 (Thread 0x7fdd95166740 (LWP 20320)): > > #0 0x00007fdd941e376d in poll () from /lib64/libc.so.6 > > #1 0x00007fdd94737fcb in virEventPollRunOnce () from > > /usr/lib64/libvirt.so.0 > > #2 0x00007fdd94736a4d in virEventRunDefaultImpl () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0 > > #4 0x00007fdd951e508a in main () > > > > and also from my program if it important: > > > > 0 0x00007ffff48c1750 __poll_nocancel > > 1 0x00007ffff649e5ca virNetClientIOEventLoop > > 2 0x00007ffff649ed96 virNetClientSendInternal > > 3 0x00007ffff64a0443 virNetClientSendWithReply > > 4 0x00007ffff64a0daa virNetClientProgramCall > > 5 0x00007ffff6478566 callFull.isra.2 > > 6 0x00007ffff6487fa2 remoteDomainDefineXML > > 7 0x00007ffff6449d2f virDomainDefineXML > > 8 0x0000000000426941 virt_domain::create > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 > > 9 0x000000000040c1a9 main > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 > > Michal thx for quick reply, i will wait.
Attachment:
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ libvirt-users mailing list libvirt-users@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvirt-users
