Without considering the manually triggered OOM, if no victim found in system OOM, the system will be deadlocked on memory, however if no victim found in memcg OOM, it can charge successfully and runs well. This behavior in memcg oom is not proper because that can prevent the memcg from being limited. Take an easy example. $ cd /sys/fs/cgroup/foo/ $ echo $$ > cgroup.procs $ echo 200M > memory.max $ cat memory.max 209715200 $ echo -1000 > /proc/$$/oom_score_adj Then, let's run a memhog task in memcg foo, which will allocate 1G memory and keeps running. $ /home/yafang/test/memhog & Then memory.current will be greater than memory.max. Run bellow command in another shell. $ cat /sys/fs/cgroup/foo/memory.current 1097228288 The tasks which have already allocated memory and won't allocate new memory still runs well. This behavior makes nonsense. This patch is to improve it. If no victim found in memcg oom, we should force the current task to wait until there's available pages. That is similar with the behavior in memcg1 when oom_kill_disable is set. Patch #1 and #2 are the preparation of patch #3. Yafang Shao (3): mm: change the return type of out_of_memory() mm, memcg: introduce a new helper task_in_memcg_oom_set() memcg oom: bail out from the charge path if no victim found include/linux/memcontrol.h | 30 ++++++++++++++++++ include/linux/oom.h | 9 +++++- include/linux/sched.h | 1 + mm/memcontrol.c | 63 ++++++++++++++++++++------------------ mm/oom_kill.c | 36 +++++++++++++++++----- mm/page_alloc.c | 3 +- 6 files changed, 104 insertions(+), 38 deletions(-) -- 2.18.2