On Tue 21-06-16 13:46:43, Michal Hocko wrote: > On Tue 21-06-16 20:03:17, Tetsuo Handa wrote: > > Michal Hocko wrote: > > > On Mon 13-06-16 13:19:43, Michal Hocko wrote: > > > [...] > > > > I am trying to remember why we are disabling oom killer before kernel > > > > threads are frozen but not really sure about that right away. > > > > > > OK, I guess I remember now. Say that a task would depend on a freezable > > > kernel thread to get to do_exit (stuck in wait_event etc...). We would > > > simply get stuck in oom_killer_disable for ever. So we need to address > > > it a different way. > > > > > > One way would be what you are proposing but I guess it would be more > > > systematic to never call exit_oom_victim on a remote task. After [1] we > > > have a solid foundation to rely only on MMF_REAPED even when TIF_MEMDIE > > > is set. It is more code than your patch so I can see a reason to go with > > > yours if the following one seems too large or ugly. > > > > > > [1] http://lkml.kernel.org/r/1466426628-15074-1-git-send-email-mhocko@xxxxxxxxxx > > > > > > What do you think about the following? > > > > I'm OK with not clearing TIF_MEMDIE from a remote task. But this patch is racy. > > > > > @@ -567,40 +612,23 @@ static void oom_reap_task(struct task_struct *tsk) > > > while (attempts++ < MAX_OOM_REAP_RETRIES && !__oom_reap_task(tsk)) > > > schedule_timeout_idle(HZ/10); > > > > > > - if (attempts > MAX_OOM_REAP_RETRIES) { > > > - struct task_struct *p; > > > + tsk->oom_reaper_list = NULL; > > > > > > + if (attempts > MAX_OOM_REAP_RETRIES) { > > > > attempts > MAX_OOM_REAP_RETRIES would mean that down_read_trylock() > > continuously failed. But it does not guarantee that the offending task > > shall not call up_write(&mm->mmap_sem) and arrives at mmput() from exit_mm() > > (as well as other threads which are blocked at down_read(&mm->mmap_sem) in > > exit_mm() by the offending task arrive at mmput() from exit_mm()) when the > > OOM reaper was preempted at this point. > > > > Therefore, find_lock_task_mm() in requeue_oom_victim() could return NULL and > > the OOM reaper could fail to set MMF_OOM_REAPED (and find_lock_task_mm() in > > oom_scan_process_thread() could return NULL and the OOM killer could fail to > > select next OOM victim as well) when __mmput() got stuck. > > Fair enough. As this would break no-lockup requirement we cannot go that > way. Let me think about it more. Hmm, what about the following instead. It is rather a workaround than a full flaged fix but it seems much more easier and shouldn't introduce new issues. --- >From 86bf010d2a6086491bb356494fab0e0fca80dee9 Mon Sep 17 00:00:00 2001 From: Michal Hocko <mhocko@xxxxxxxx> Date: Mon, 20 Jun 2016 17:28:18 +0200 Subject: [PATCH] oom, suspend: fix oom_reaper vs. oom_killer_disable race Tetsuo has reported the following potential oom_killer_disable vs. oom_reaper race: (1) freeze_processes() starts freezing user space threads. (2) Somebody (maybe a kenrel thread) calls out_of_memory(). (3) The OOM killer calls mark_oom_victim() on a user space thread P1 which is already in __refrigerator(). (4) oom_killer_disable() sets oom_killer_disabled = true. (5) P1 leaves __refrigerator() and enters do_exit(). (6) The OOM reaper calls exit_oom_victim(P1) before P1 can call exit_oom_victim(P1). (7) oom_killer_disable() returns while P1 not yet finished (8) P1 perform IO/interfere with the freezer. This situation is unfortunate. We cannot move oom_killer_disable after all the freezable kernel threads are frozen because the oom victim might depend on some of those kthreads to make a forward progress to exit so we could deadlock. It is also far from trivial to teach the oom_reaper to not call exit_oom_victim() because then we would lose a guarantee of the OOM killer and oom_killer_disable forward progress because exit_mm->mmput might block and never call exit_oom_victim. It seems the easiest way forward is to workaround this race by calling try_to_freeze_tasks again after oom_killer_disable. This will make sure that all the tasks are frozen or it bails out. Fixes: 449d777d7ad6 ("mm, oom_reaper: clear TIF_MEMDIE for all tasks queued for oom_reaper") Reported-by: Tetsuo Handa <penguin-kernel@xxxxxxxxxxxxxxxxxxx> Signed-off-by: Michal Hocko <mhocko@xxxxxxxx> --- kernel/power/process.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/power/process.c b/kernel/power/process.c index df058bed53ce..0c2ee9761d57 100644 --- a/kernel/power/process.c +++ b/kernel/power/process.c @@ -146,6 +146,18 @@ int freeze_processes(void) if (!error && !oom_killer_disable()) error = -EBUSY; + /* + * There is a hard to fix race between oom_reaper kernel thread + * and oom_killer_disable. oom_reaper calls exit_oom_victim + * before the victim reaches exit_mm so try to freeze all the tasks + * again and catch such a left over task. + */ + if (!error) { + pr_info("Double checking all user space processes after OOM killer disable... "); + error = try_to_freeze_tasks(true); + pr_cont("\n"); + } + if (error) thaw_processes(); return error; -- 2.8.1 -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>