On Mon, May 11, 2015 at 10:52:46AM +1000, Tony Breeds wrote:
Hello all, This is with reference to OpenStack (nova) bug 1439280 [1]. The symptom is when nova(-compute) tries to launch an instance/VM it errors out with: "libvirtError: Requested operation is not valid: cpu affinity is not supported" This only happens with using qemu in TCG mode.
I can see where the error might have come from. QEMU doesn't have vcpu threads if ran in TCG mode, so if libvirt tries to pin some of them, this will happen. *But* upstream handles this perfectly. I just tried and it works. I'll explain why a few lines down the road.
After looking at the domain XML and the docs at [2] It seems to me the problem is either: 1) the libvirt driver in nova is generating valid XML that is an invalid domain description ; or 2) NUMA support in qemu (TCG) mode is broken.
I don't get what this has to do with NUMA, but anyway I just think this is a bug in older libvirt actually.
Exploring these options a little further: Option 1: the libvirt driver in nova is generating valid XML that is an invalid domain description ==== This is the relevant section of the domain XML --- <vcpu placement='static' cpuset='0-1'>1</vcpu> <cputune> <shares>1024</shares> </cputune> --- To my reading of [2] the domain XML should either avoid specifying the 'cpuset' in the vcpu node ; or include 'emulatorpin' in the cputune node.
Yes, that's because we can't differentiate which threads do what with TCG accel. Because of the you must only specify one pinning for all threads (I/O threads, CPU threads and the emulator thread), which is done exactly using the 'cpuset' attribute as that is valid for the whole machine. Thatnsk to that we don't have to differentiate anything and just use the cpuset for the whole machine -> no error should occur there.
Have I understood the documentation correctly? If so it would seem that the correct fix is in nova to teach the libvirt driver to generate the correct XML for this virtulisation type.
You understood the documentation correctly, it was just libvirt acting up a bit.
Option 2: NUMA support in qemu (TCG) mode is broken. ==== Is cpuset and NUMA architecture expected to work in qemu when running in TCG mode? If it is then clearly we need to dig into this and fix qemu.
I'm still not sure why are you bringing NUMA into the mix as there is
nothing NUMA related in the bug or this mail. But to complete my
answer, even though there is no practical benefit to using NUMA on TCG
accelerated QEMU machine, it should still work.
Here's the XML I just fired up using libvirt 1.2.15 just to try it
out. And it simply works.
<domain type='qemu'>
<name>dummy-tcg</name>
<uuid>946f637c-dcaa-4392-8926-b54476eddd43</uuid>
<memory unit='KiB'>131072</memory>
<currentMemory unit='KiB'>65536</currentMemory>
<vcpu placement='static' cpuset='3-5,7'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-2.3'>hvm</type>
<boot dev='hd'/>
</os>
<cpu>
<numa>
<cell id='0' cpus='0,2' memory='65536' unit='KiB'/>
<cell id='1' cpus='1,3' memory='65536' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
</controller>
<memballoon model='none'/>
</devices>
</domain>
Yours Tony. [1] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1439280 [2] https://libvirt.org/formatdomain.html#elementsCPUAllocation
Attachment:
signature.asc
Description: PGP signature
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list