Hello, On Thu, Mar 10, 2022 at 01:47:34PM +0800, Tianchen Ding wrote: > If we want to build group balancer in userspace, we need: > 1) gather load info from each rq periodically > 2) make decision to set cpuset.cpus of each cgroup > > However, there're some problems about this way. > > For 1), we need to consider how frequently collecting these info, which may > impact performance and accuracy. If the load changes hugely right after we > get it once, our data are expired and then the decision may be wrong. (If we > are in kernel, faster action can be taken.) We now have a pretty well established way to transport data to userspace at really low overhead. If you piggy back on bpf interfaces, they can usually be pretty unintrusive and low effort as long as you have the right kind of data aggregated already, which shouldn't be that difficult here. > We believe 2) is harder. The specific policy may be complex and alter > according to different scenes. There's not a general method. > e.g., with 16cpus and 4 cgroups, how to decide when we set one of them > 0-3(when busy)or 0-7(when some of other cgroups are idle)? If there are much > more threads in cgroupA than cgroupB/C/D , and we want to satisfy cgroupA as > far as possible(on the premise of fairness of B/C/D)， dynamically > enlarging(when B/C/D partly idle) and shrinking(when B/C/D busy) cpuset of > cgroupA requires complex policy. In this example, fairness and performance > can be provided by existing scheduler, but when it comes to grouping hot > cache or decreasing competion, both scheduler in kernel and action in > userspace are hard to solve. So, I get that it's not easy. In fact, we don't even yet know how to properly compare loads across groups of CPUs - simple sums that you're using break down when there are big gaps in weight numbers across tasks and can become meaningless in the presence of CPU affinities. They can still work when the configuration is fairly homogeenous and controlled but the standard should be far higher for something we bake into the kernel and expose userland-visible interface for. > What's more, in many cloud computing scenes, there may be hundreds or > thousands of containers, which are much larger than partition number. These > containers may be dynamically created and destroyed at any time. Making > policy to manage them from userspace will not be practical. > > These problems become easy when going to kernelspace. We get info directly > from scheduler, and help revising its decision at some key points, or do > some support work(e.g., task migration if possible). I don't think they become necessarily easy. Sure, you can hack up something which works for some cases by poking into existing code; however, the bar for acceptance is also way higher for a kernel interface - it should be generic, consistent with other interfaces (I won't go into cgroup interface issues here), and work orthogonally with other kernel features (ie. task / group weights should work in an explainable way). I don't think the proposed patches are scoring high in those axes. I'm not against the goal here. Given that cgroups express the logical structure of applications running on the system, it does make sense to factor that into scheduling decisions. However, what's proposed seems too premature and I have a hard time seeing why this level of functionality would be difficult to be implement from userspace with some additions in terms of visibility which is really easy to do these days. Thanks. -- tejun
