On Mon, 21 Dec 2015 12:44:32 -0500 John Ferlan <jferlan@xxxxxxxxxx> wrote: > > > On 12/21/2015 03:36 AM, Henning Schild wrote: > > On Mon, 14 Dec 2015 16:27:54 -0500 > > John Ferlan <jferlan@xxxxxxxxxx> wrote: > > > >> > >> > >> On 11/13/2015 11:56 AM, Henning Schild wrote: > >>> Hi, > >>> > >>> i already explained some of the cgroup problems in some detail so > >>> i will not do that again. > >>> https://www.redhat.com/archives/libvir-list/2015-October/msg00876.html > >>> > >>> I managed to solve some of the problems in the current codebase, > >>> and am now sharing the patches. But they are really just half of > >>> what i had to change to get libvirt to behave in a system with > >>> isolated cpus. > >>> > >>> Other changes/hacks i am not sending here because they do not work > >>> for the general case: > >>> - create machine.slice before starting libvirtd (smaller than > >>> root) ... and hope it wont grow > >>> - disabling cpuset.cpus inheritance in libvirtd > >>> - allowing only xml with fully specified cputune > >>> - set machine cpuset to (vcpupins | emulatorpin) > >>> > >>> I am not sure how useful the individual fixes are, i am sending > >>> them as concrete examples for the problems i described earlier. > >>> And i am hoping that will start a discussion. > >>> > >>> Henning > >>> > >>> Henning Schild (3): > >>> util: cgroups do not implicitly add task to new machine cgroup > >>> qemu: do not put a task into machine cgroup > >>> qemu cgroups: move new threads to new cgroup after cpuset is set > >>> up > >>> > >>> src/lxc/lxc_cgroup.c | 6 ++++++ > >>> src/qemu/qemu_cgroup.c | 23 ++++++++++++++--------- > >>> src/util/vircgroup.c | 22 ---------------------- > >>> 3 files changed, 20 insertions(+), 31 deletions(-) > >>> > >> > >> > >> The updated code looks fine to me - although it didn't directly > >> git am -3 to top of tree - I was able to make a few adjustments to > >> get things merged... Since no one has objected to this ordering > >> change - I've pushed. > > > > Sorry the patches where still based on v1.2.19. Thanks for the merge > > and accepting them! > > > > No problem - although it seems they've generated a regression in the > virttest memtune test suite. I'm 'technically' on vacation for the > next couple of weeks; however, I think/perhaps the problem is a > result of this patch and the change to adding the task to the cgroup > at the end of the for loop, but perhaps the following code causes the > control to jump back to the top of the loop: > > if (!cpumap) > continue; > > if (qemuSetupCgroupCpusetCpus(cgroup_vcpu, cpumap) < 0) > goto cleanup; > > not allowing the > > > /* move the thread for vcpu to sub dir */ > if (virCgroupAddTask(cgroup_vcpu, > qemuDomainGetVcpuPid(vm, i)) < 0) > goto cleanup; > > to be executed. > > The code should probably change to be (like IOThreads): > > if (cpumap && > qemuSetupCgroupCpusetCpus(cgroup_vcpu, cpumap) < 0) > goto cleanup; > > > As for the rest, I suspect things will be quite quiet around here over > the next couple of weeks. A discussion to perhaps start in the new > year. Same here. I will have a look at that regression after my vacation, should it still be there. Henning > John > > > > Wrong operation ordering within libvirt cgroups (like the ones > > fixed by the patches) could still push tasks onto dedicated cpus. > > And more importantly other cgroups users can still grab the > > dedicated cpus as well. The only reliable solution to prevent that > > seems to be making use of the "exclusive" feature of cpusets. And > > that would imply changing the cgroups layout of libvirt again. > > Because sets can not be partially exclusive and libvirt deals with > > dedicated cpus and shared ones. > > How to deal with these problems is a discussion that i wanted to get > > started with this patch-series. It would be nice to receive general > > comments on that. How should we proceed here? I could maybe write an > > RFC mail describing the problems again and suggesting changes to > > libvirt on a conceptual basis. > > > > But until then maybe people responsible for cgroups in libvirt (Paul > > and Martin?) can again look at > > https://www.redhat.com/archives/libvir-list/2015-October/msg00876.html > > There i described how naive use of cgoups can place tasks on cpus > > that are supposed to be isolated/dedicated/exclusive. Even if > > libvirt does not make these mistakes it should protect itself > > against docker, systemd, ... > > > > Henning > > > > -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list