Hello All,
Since 4.5.0-23.el7 version (Red Hat 7.7), when I launch pinned VM, libvirtd reset CPU affinity to all enabled in host CPUs, if it runs in custom cpuset.
I can't reproduce this behavior with 4.5.0-10.el7_6.12 with the same kernel version (Red Hat 7.7). Libvirt runs in a custom cpuset 'libvirt', where the number of available cpus is restricted to 0,2,4,6,8.
And this 'libvirt' cpuset is created in a system with total cpus number: 40 (all cpus are enabled in BIOS) and I have '0-39' range in /sys/fs/cgroup/cpuset/cpuset.cpus. When a VM with pinned vcpus is launched, VM XML config is in attachement: <vcpu placement='static'>2</vcpu> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='6'/> I have the following error: # virsh create /tmp/vm1_2vms-2cores_host1.xml error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied And in the debug log I can see, that /sys/fs/cgroup/cpuset/machine.slice was created with a proper cpuset list: 0,2,4,6,8.
This list at the beginning was successfully inherited and set in /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus.
But finally libvirtd tries to reset inherited cpus and set there: "0-39" 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/ 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/ 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/ 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus 2019-08-28 15:11:03.357+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0,2,4,6,8' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.mems' to '0-1' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.memory_migrate' to '1' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1126 : Done making controllers for group 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0-39' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied The same log snippet with 4.5.0-10.el7_6.12 version, everything works well: 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/ 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/ 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/ 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.cpus' to '0,2,4,6,8' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0-1' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.memory_migrate' to '1' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1126 : Done making controllers for group 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/tasks' to '27045' Would someone give me a hint, please, if we need to change settings lied to NUMA or vCPUs in VM XML config according to last changed in libvirtd ?
Please, find a bug report about this issue here: https://bugzilla.redhat.com/show_bug.cgi?id=1746517
I suppose, that this may be an impact of one of the following patches applied to 4.5.0-10.el7_6.12 source code version:
libvirt-qemu-Rework-setting-process-affinity.patchlibvirt-qemu-Fix-qemuProcessInitCpuAffinity.patchlibvirt-qemu-Fix-NULL-pointer-access-in-qemuProcessInitCpuAffinity.patchlibvirt-qemu-Fix-leak-in-qemuProcessInitCpuAffinity.patch
How reproducible: Always Steps to Reproduce: 1. create a cpuset with reduced cpus list and launch libvirtd in it 2. prepare VM XML with pinned vcpus and <numatune> setting 3. virsh create vm.xml Actual results: error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied Expected results: should work as with a previous version 4.5.0-10.el7_6.12. Additional info: # ps auxf | grep libvirt root 24571 2.3 0.0 1895564 28052 ? Ssl 16:52 0:00 /usr/sbin/libvirtd # grep Cpus_allowed /proc/24571/task/*/status /proc/24571/task/24571/status:Cpus_allowed: 00,00000554 /proc/24571/task/24571/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24572/status:Cpus_allowed: 00,00000554 /proc/24571/task/24572/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24573/status:Cpus_allowed: 00,00000554 /proc/24571/task/24573/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24574/status:Cpus_allowed: 00,00000554 /proc/24571/task/24574/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24575/status:Cpus_allowed: 00,00000554 .. for all threads Processor: "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz" Sockets: 2, Cores: 20, HyperThreads: 40 socket 0 socket 1 +---------------------+ +---------------------+ | c0 c1 | | c0 c1 | | +-------+ +-------+ | | +-------+ +-------+ | | | 0| 20| | 2| 22| | | | 1| 21| | 3| 23| | | +-------+ +-------+ | | +-------+ +-------+ | | c2 c3 | | c2 c3 | | +-------+ +-------+ | | +-------+ +-------+ | | | 4| 24| | 6| 26| | | | 5| 25| | 7| 27| | | +-------+ +-------+ | | +-------+ +-------+ | | c4 c8 | | c4 c8 | | +-------+ +-------+ | | +-------+ +-------+ | | | 8| 28| | 10| 30| | | | 9| 29| | 11| 31| | | +-------+ +-------+ | | +-------+ +-------+ | | c9 c10 | | c9 c10 | | +-------+ +-------+ | | +-------+ +-------+ | | | 12| 32| | 14| 34| | | | 13| 33| | 15| 35| | | +-------+ +-------+ | | +-------+ +-------+ | | c11 c12 | | c11 c12 | | +-------+ +-------+ | | +-------+ +-------+ | | | 16| 36| | 18| 38| | | | 17| 37| | 19| 39| | | +-------+ +-------+ | | +-------+ +-------+ | +---------------------+ +---------------------+ # cat /sys/fs/cgroup/cpuset/cpuset.cpus 0-39 # cat /sys/fs/cgroup/cpuset/libvirt/cpuset.cpus 2,4,6,8,10 # cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 0,2,4,6,8
<domain type='kvm'>
<name>vm1_2vms-2cores_host1</name> <!-- change the name according to your needs-->
<memory>1048576</memory> <!-- adapt to the desired memory size -->
<os>
<type>hvm</type>
<boot dev="hd"/>
</os>
<vcpu placement='static'>2</vcpu>
<vcpupin vcpu='0' cpuset='4'/>
<vcpupin vcpu='1' cpuset='6'/>
<cpu>
<numa>
<!-- adapt to the desired memory size -->
<cell id="0" cpus="1" memory="1048576" memAccess="shared"/>
</numa>
</cpu>
<numatune>
<!-- adapt to set the host node where hugepages are taken -->
<memory mode='strict' nodeset='0'/>
</numatune>
<features>
<acpi/>
</features>
<devices>
<disk type='file' device='disk'>
<source file='/var/lib/libvirt/images/vm1_2vms-2cores_host1-seed.iso'/>
<target dev='vdb' bus='virtio'/>
</disk>
<disk type='file' device='disk'>
<driver type='qcow2' cache='none'/>
<source file='/var/lib/libvirt/images/ubuntu-16.04.img'/>
<target dev='vda' bus='virtio'/>
</disk>
<interface type='user'>
<model type='virtio'/>
</interface>
<serial type='tcp'>
<source mode='bind' host='0.0.0.0' service='10000'/>
<protocol type='raw'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<source path='/dev/pts/4'/>
<target port='1'/>
<alias name='serial1'/>
</serial>
<console type='tcp'>
<source mode='bind' host='0.0.0.0' service='10000'/>
<protocol type='raw'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
</devices>
</domain>
_______________________________________________ libvirt-users mailing list libvirt-users@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvirt-users
