Hi Daniel,
On 09/05/2012 05:43 PM, Daniel P. Berrange wrote:
Your patch appears to work in some limited scenarios, but more
generally it will fail to work, and resulted in undesirable
behaviour.
Consider for example, if libvirtd is configured thus:
cd /sys/fs/cgroup/cpuset
mkdir demo
cd demo
echo 2-3> cpuset.cpus
echo 0> cpuset.mems
echo $$> tasks
/usr/sbin/libvirtd
ie, libvirtd is now running on cpus 2-3, in group 'demo'. VMs will
be created in
/sys/fs/cgroup/cpuset/demo/libvirt/qemu/$VMNAME
Your patch attempts to set the cpuset.cpus on 'libvirt/qemu/$VMNAME'
but ignores the fact that there could be many higher directories
(eg demo here) that need setting. libvirtd, however, should not be
responsible for / allowed to change settings in parent cgroups from
where it was started. ie in this example, libvirtd should *not*
touch the 'demo' cgroup.
Yes, I didn't realize this situation. Thanks for remind me. :)
So consider systemd starting tasks, giving them custom cgroups.
Now systemd also has to listen for netlink events and reset the
cpuset masks.
Things are even worse if the admin has temporarily offlined all the
cpus that are associated with the current cpuset. When this happens
the kernel throws libvirtd and all its VMs out of their current
cgroups and dumps them up in a parent cgroup (potentially even the
root group). This is really awful.
Agreed. :)
IMHO, execution of those tasks should simply be paused (same way that
the 'freezer' cgroup pauses tasks). The admin can then either move
the tasks to an alternate cgroup, or change the cpuset mask to allow
them to continue running.
The kernel's current behaviour of pushing all tasks up into a parent
cgroup is just crazy - it is just throwing away the users requested
cpu mask forever :-(
If I want to solve the start failure problem, what should I do ?
I maintain the problems we see with cpuset controller cannot be reasonably
solved by libvirtd, or userspace in general. The kernel behaviour is just
flawed. If the kernel won't fix it, then we should recommend people not
to use the cpuset cgroup at all, and just rely on our sched_setaffinity
support instead.
I like the sched_setaffinity idea. Let's just temporarily shut off
cpuset cgroup in libvirt, shall we ?
Since cpuset cgroup was turned on when I was working on the emulator-pin
job, I will shut if off and improve all these with sched_setaffinity().
And I will send new patches soon. Thanks. :)
Daniel
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list