Hi John,
The issue mentioned is generated by a -1 returned by qemuDomainGetIOThreadsMon.
Further, it is caused by a -1 returned by virQEMUCapsGet:
6174 if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_OBJECT_IOTHREAD)) {
6175 virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
6176 _("IOThreads not supported with this binary"));
6177 goto endjob;
6178 }
By checking qemuCaps->flags, it seems the bit of QEMU_CAPS_OBJECT_IOTHREAD(176) is not set.
(gdb) call virBitmapFormat(qemuCaps->flags)
$4 = 0x7f08a4004cb0 "13,33,46,50,54-55,58,61-64,66-70,72-73,77-78,80,87-88,92-93,95-97,99,102-110,112,114,119-121,128-131,141-142,146,148-149,151-156,158-161,165,167,174-175,180,182,188,193-198,210,214,221-222,225,250,255"...
Breakpoint 1 at 0x7f088ef00880: file qemu/qemu_driver.c, line 5492.
(gdb) c
Continuing.
[Switching to Thread 0x7f08bcdcb700 (LWP 24711)]
Breakpoint 1, qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492
5492 {
Unprocessed breakpoint [1]
(gdb) finish
Run till exit from #0 qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492
0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885
20885 if ((niothreads = qemuDomainGetIOThreadsMon(driver, dom, &iothreads)) < 0)
Value returned is $1 = -1
(gdb) bt
#0 0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885
#1 0x00007f088eee70a9 in qemuDomainGetStats (flags=1, record=<synthetic pointer>, stats=255, dom=0x7f08740cd840, conn=0x7f08ac000c50) at qemu/qemu_driver.c:21062
#2 qemuConnectGetAllDomainStats (conn=0x7f08ac000c50, doms=<optimized out>, ndoms=<optimized out>, stats=255, retStats=0x7f08bcdcab10, flags=<optimized out>) at qemu/qemu_driver.c:21162
#3 0x00007f08cd147646 in virConnectGetAllDomainStats (conn=0x7f08ac000c50, stats=0, retStats=retStats@entry=0x7f08bcdcab10, flags=0) at libvirt-domain.c:11685
#4 0x0000562e2404d2c0 in remoteDispatchConnectGetAllDomainStats (server=0x562e24ee74a0, msg=<optimized out>, ret=0x7f08ac001ec0, args=0x7f08ac001f30, rerr=0x7f08bcdcac50, client=0x562e24ef5200) at remote/remote_daemon_dispatch.c:6665
#5 remoteDispatchConnectGetAllDomainStatsHelper (server=0x562e24ee74a0, client=0x562e24ef5200, msg=<optimized out>, rerr=0x7f08bcdcac50, args=0x7f08ac001f30, ret=0x7f08ac001ec0) at remote/remote_daemon_dispatch_stubs.h:743
#6 0x00007f08cd05dad5 in virNetServerProgramDispatchCall (msg=0x562e24ef5910, client=0x562e24ef5200, server=0x562e24ee74a0, prog=0x562e24ee6650) at rpc/virnetserverprogram.c:437
#7 virNetServerProgramDispatch (prog=0x562e24ee6650, server=server@entry=0x562e24ee74a0, client=0x562e24ef5200, msg=0x562e24ef5910) at rpc/virnetserverprogram.c:304
#8 0x00007f08cd063f7d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x562e24ee74a0) at rpc/virnetserver.c:144
#9 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562e24ee74a0) at rpc/virnetserver.c:165
#10 0x00007f08ccf95171 in virThreadPoolWorker (opaque=opaque@entry=0x562e24ee69a0) at util/virthreadpool.c:167
#11 0x00007f08ccf944f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#12 0x00007f08cb213dc5 in start_thread (arg=0x7f08bcdcb700) at pthread_create.c:308
#13 0x00007f08cab3b76d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb)
My domain configuration is：
[root@dl-c200 libvirt]# virsh list
v^H Id Name State
----------------------------------
1 ubuntu16.04-base running
[root@dl-c200 libvirt]# virsh dumpxml 1
<domain type='kvm' id='1'>
<name>ubuntu16.04-base</name>
<uuid>19d01bd1-ad54-4176-84bd-e510de08fe00</uuid>
<memory unit='KiB'>10485760</memory>
<currentMemory unit='KiB'>10485760</currentMemory>
<vcpu placement='static' current='4'>8</vcpu>
<cputune>
<cachetune vcpus='0' id='vcpus_0'>
<monitor level='3' vcpus='0'/>
</cachetune>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Broadwell</model>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='xsaveopt'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/tmp/kvm-donot-delete/vm-ubuntu16.04-base'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu'/>
<target dev='hda' bus='ide'/>
<readonly/>
<alias name='ide0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<alias name='usb'/>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<alias name='usb'/>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<alias name='usb'/>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:eb:24:d3'/>
<source bridge='virbr0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c157,c912</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c157,c912</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
I wonder if these message is enough for you to locate the root cause, I don’t have too much time in investigating this today, I'll try to create a fix tomorrow if you need.
Huaqiang
> -----Original Message-----
> From: Wang, Huaqiang
> Sent: Tuesday, November 27, 2018 11:21 AM
> To: 'John Ferlan' <jferlan@xxxxxxxxxx>; libvir-list@xxxxxxxxxx
> Subject: RE: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology
> (CMT)
>
>
>
> > -----Original Message-----
> > From: John Ferlan [mailto:jferlan@xxxxxxxxxx]
> > Sent: Tuesday, November 27, 2018 10:59 AM
> > To: Wang, Huaqiang <huaqiang.wang@xxxxxxxxx>; libvir-list@xxxxxxxxxx
> > Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology
> > (CMT)
> >
> >
> >
> > On 11/26/18 9:39 PM, Wang, Huaqiang wrote:
> > > Hi John,
> > >
> > > Really appreciate your hard work for the CMT series. Next I'll
> > > working on
> > the MBM.
> > >
> > > In testing the newly pushed code, I find a problem:
> > >
> > > <error message>
> > > [david@dl-c200 ~]$ sudo virsh domstats
> > > error: An error occurred, but the cause is unknown </error message>
> > >
> >
> > I couldn't reproduced in a quick test here. Can you get a thread trace
> > of the failure?
> >
> > What I usually do, build libvirt, then in a terminal session at the
> > top of the git tree "./run gdb src/libvirtd" (dbg> r)... THen in
> > another terminal session run the virsh command and when the libvirtd
> session stops do a "t a a bt"
> > (thread apply all backtrace)...
> >
> > John
>
> I'll trace the error.
>
> Thanks.
> Huaqiang
>
> >
> > (done for the night)
> >
> > > seems it is caused by qemuDomainGetStatsIOThread not by the new
> > series.
> > > What I test is return 0 immediately at top of
> > > qemuDomainGetStatsIOThread, the command 'virsh domstats' reports
> > the cache statistics normally.
> > >
> > > BR
> > > Huaqiang
> > >
> > >> -----Original Message-----
> > >> From: John Ferlan [mailto:jferlan@xxxxxxxxxx]
> > >> Sent: Tuesday, November 27, 2018 9:49 AM
> > >> To: Wang, Huaqiang <huaqiang.wang@xxxxxxxxx>; libvir-
> > list@xxxxxxxxxx
> > >> Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring
> > Technology
> > >> (CMT)
> > >>
> > >>
> > >>
> > >> On 11/26/18 12:56 PM, Wang Huaqiang wrote:
> > >>> These patches are the remaining part for the CMT enabling series,
> > >>> and most of the series have been merged.
> > >>>
> > >>> This series is addressing John's review comments and suggestions,
> > >>> which are
> > >>> https://www.redhat.com/archives/libvir-list/2018-
> > >> November/msg00907.htm
> > >>> l
> > >>> https://www.redhat.com/archives/libvir-list/2018-
> > >> November/msg00510.htm
> > >>> l
> > >>> https://www.redhat.com/archives/libvir-list/2018-
> > >> November/msg00561.htm
> > >>> l
> > >>>
> > >>> Change lists:
> > >>> Changes in v10:
> > >>> -. Add tag (virResctrlMonitorType) in
> qemuDomainGetResctrlMonData,
> > >> thus
> > >>> qemuDomainGetResctrlMonData could be reused for MBM.
> > >>> -. Using VIR_APPEND_ELEMENT to append
> virQEMUResctrlMonDataPtr
> > >> list.
> > >>> -. Add qemuDomainFreeResctrlMonData.
> > >>> -. Add virResctrlMonitorFreeStats.
> > >>> -. Return a list of virResctrlMonitorStatsPtr instead of
> > >>> a virResctrlMonitorStats array in virResctrlMonitorGetStats.
> > >>>
> > >>> Changes in V9:
> > >>> -. Addressing code review comments form John.
> > >>> -. Refined the names for new data structure and new functions.
> > >>> -. Merged qemuDomainGetStatsCpuResMonitorPerTag and
> > >> qemuDomainGetStatsCpuResMonitor,
> > >>> and refined new function name based on the fact that we only
> > >>> support
> > >> cache monitor now.
> > >>> Wang Huaqiang (4):
> > >>> util: Return a list of pointer in virResctrlMonitorGetStats
> > >>> util: Add function to free monitor statistical data
> > >>> qemu: Report cache occupancy (CMT) with domstats
> > >>> docs: Updated news.xml for CMT
> > >>>
> > >>> docs/news.xml | 12 ++++
> > >>> src/libvirt-domain.c | 12 ++++
> > >>> src/libvirt_private.syms | 1 +
> > >>> src/qemu/qemu_driver.c | 183
> > >> ++++++++++++++++++++++++++++++++++++++++++++++-
> > >>> src/util/virresctrl.c | 26 +++++--
> > >>> src/util/virresctrl.h | 8 ++-
> > >>> tools/virsh.pod | 14 ++++
> > >>> 7 files changed, 248 insertions(+), 8 deletions(-)
> > >>>
> > >>
> > >> Reviewed-by: John Ferlan <jferlan@xxxxxxxxxx>
> > >> (series)
> > >>
> > >> and pushed,
> > >>
> > >> John
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list
