(2012/04/20 16:37), Ying Han wrote: > On Thu, Apr 19, 2012 at 3:33 PM, Johannes Weiner <hannes@xxxxxxxxxxx> wrote: >> On Thu, Apr 19, 2012 at 10:47:27AM -0700, Ying Han wrote: >>> On Thu, Apr 19, 2012 at 10:04 AM, Michal Hocko <mhocko@xxxxxxx> wrote: >>>> On Wed 18-04-12 11:00:40, Ying Han wrote: >>>>> On Wed, Apr 18, 2012 at 5:24 AM, Johannes Weiner <hannes@xxxxxxxxxxx> wrote: >>>>>> On Tue, Apr 17, 2012 at 09:37:46AM -0700, Ying Han wrote: >>>>>>> The "soft_limit" was introduced in memcg to support over-committing the >>>>>>> memory resource on the host. Each cgroup configures its "hard_limit" where >>>>>>> it will be throttled or OOM killed by going over the limit. However, the >>>>>>> cgroup can go above the "soft_limit" as long as there is no system-wide >>>>>>> memory contention. So, the "soft_limit" is the kernel mechanism for >>>>>>> re-distributing system spare memory among cgroups. >>>>>>> >>>>>>> This patch reworks the softlimit reclaim by hooking it into the new global >>>>>>> reclaim scheme. So the global reclaim path including direct reclaim and >>>>>>> background reclaim will respect the memcg softlimit. >>>>>>> >>>>>>> v3..v2: >>>>>>> 1. rebase the patch on 3.4-rc3 >>>>>>> 2. squash the commits of replacing the old implementation with new >>>>>>> implementation into one commit. This is to make sure to leave the tree >>>>>>> in stable state between each commit. >>>>>>> 3. removed the commit which changes the nr_to_reclaim for global reclaim >>>>>>> case. The need of that patch is not obvious now. >>>>>>> >>>>>>> Note: >>>>>>> 1. the new implementation of softlimit reclaim is rather simple and first >>>>>>> step for further optimizations. there is no memory pressure balancing between >>>>>>> memcgs for each zone, and that is something we would like to add as follow-ups. >>>>>>> >>>>>>> 2. this patch is slightly different from the last one posted from Johannes >>>>>>> http://comments.gmane.org/gmane.linux.kernel.mm/72382 >>>>>>> where his patch is closer to the reverted implementation by doing hierarchical >>>>>>> reclaim for each selected memcg. However, that is not expected behavior from >>>>>>> user perspective. Considering the following example: >>>>>>> >>>>>>> root (32G capacity) >>>>>>> --> A (hard limit 20G, soft limit 15G, usage 16G) >>>>>>> --> A1 (soft limit 5G, usage 4G) >>>>>>> --> A2 (soft limit 10G, usage 12G) >>>>>>> --> B (hard limit 20G, soft limit 10G, usage 16G) >>>>>>> >>>>>>> Under global reclaim, we shouldn't add pressure on A1 although its parent(A) >>>>>>> exceeds softlimit. This is what admin expects by setting softlimit to the >>>>>>> actual working set size and only reclaim pages under softlimit if system has >>>>>>> trouble to reclaim. >>>>>> >>>>>> Actually, this is exactly what the admin expects when creating a >>>>>> hierarchy, because she defines that A1 is a child of A and is >>>>>> responsible for the memory situation in its parent. >>>> >>>> Hmm, I guess that both approaches have cons and pros. >>>> * Hierarchical soft limit reclaim - reclaim the whole subtree of the over >>>> soft limit memcg >>>> + it is consistent with the hard limit reclaim >>> Not sure why we want them to be consistent. Soft_limit is serving >>> different purpose and the one of the main purpose is to preserve the >>> working set of the cgroup. >> >> I'd argue, given the history of cgroups, one of the main purposes is >> having a machine of containers where you overcommit their hard limit >> and set the soft limit accordingly to provide fairness. >> >> Yes, we don't want to reclaim hierarchies that are below their soft >> limit as long as there are some in excess, of course. This is a flaw >> and needs fixing. But it's something completely different than >> changing how the soft limit is defined and suddenly allow child >> groups, which you may not trust, to override rules defined by parental >> groups. >> >> It bothers me that we should add something that will almost certainly >> bite us in the future while we are discussing on the cgroups list what >> would stand in the way of getting sane hierarchy semantics across >> controllers to provide consistency, nesting, etc. > > I understand the concern here and I don't want the soft_limit reclaim > to be far away from the other part of the cgroup design down to the > road. On the other hand, I don't think the current implementation is > against the hierarchy semantics totally. See the comment below :) > >> >> To support a single use case, which I feel we still have not discussed >> nearly enough to justify this change. >> >> For example, I get that you want 'meta-groups' that group together >> subgroups for common accounting and hard limiting. But I don't see >> why such meta-groups have their own processes. Conceptually, I mean, >> how does a process fit into A? Is it superior to the tasks in A1 and >> A2? Why can't it live in A3? > > For user processes, I can see that is totally feasible to live in A3. > The case I was thinking is kernel threads, which 1) we don't want to > limit their memory usage 2) they serve for the whole group unlike > individual jobs. Of course, we could say that putting those kernel > thread in A3 and leave the cgroup to unlimited, but not sure if we > should constrain ourselves not having any processes running under A. > >> >> So here is a proposal: >> >> Would it make sense to try to keep those meta groups always free of >> their own memory so that they don't /need/ soft limits with weird >> semantics? E.g. immediately free the unused memory on rmdir, OR add >> mechanisms to migrate the memory to a dedicated group: >> >> A >> A1 (soft-limited) >> A2 (soft-limited) >> B >> unused (soft-limited) >> >> Move all leftover memory from finished jobs to this 'unused' group. >> You could set its soft limit to 0 so that it sticks around only until >> you actually need the memory for something else. >> >> Then you would get the benefits of accounting and limiting A1 and A2 >> under a single umbrella without the need for a soft limit in A. We >> could keep the consistent semantics for soft limits, because you would >> only have to set it on leaf nodes. >> >> Wouldn't this work for you? > > To be frankly, this sounds a lot of extra work for admin to manage the > system and we still can not prevent page being landed on A totally. > > Back to the current proposal, there are two concerns that I can tell by far: > > 1. skipping "not trust" cgroup in case it sets its soft_limit very high: > Here, we don't skip the "not trust" cgroup always. We do reclaim from > them if not enough progress made from other cgroups above the > softlimit. So, I don't see a problem here. > > 2. not reclaiming based on hierarchy: > Here I am not checking the ancestor's soft_limit in > should_reclaim_mem_cgroup(). And it will only make difference if A is > under soft_limit and A1 is above soft_limit. Now you do agree that we > shouldn't reclaim from those under softlimit groups if there are > cgroup exeed their softlimit. Then it leads me to think something like > the following: > > 1. for priority > DEF_PRIORITY - 3, only reclaim memcg above their softlimit > 2. for priority <= DEF_PRIORITY - 3, besides 1), also look at memcg's > ancestor. reclaim memcgs whose ancestor above soft_limit > 3. for priority == 0, reclaim everything. > > Then it has the guarantee of the softlimit at certain level while also > considers the hierarchy reclaim if the first few rounds doesn't > fulfill the request. > seems complicated. I vote for " Hierarchical soft limit reclaim ". If you need smart victim selection under hierarchy, please implement victim scheduler which choose A2 rather than A and A1. I think you can do it. Thanks, -Kame -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>