I am sending this as a follow up to yet-another timeout based proposal by Tetsuo because I was accused that I am pushing for a solution which I am not working on. This is a very coarse implementation of the idea I've had before. Please note that I haven't tested it yet. It is mostly to show the direction I would wish to go for. I have already explained why I hate timeout based retry loop and why I believe a more direct feedback based approach is much better. The locking protocol between the oom_reaper and the exit path is as follows. All parts which cannot race should use the exclusive lock on the exit path. Once the exit path has passed the moment when no blocking locks are taken then it clears mm->mmap under the exclusive lock. It is trivial to use a MMF_$FOO for this purpose if people think this is safer or better for any other reason. The oom proper is the only one which sets MMF_OOM_SKIP with these patches which is IMHO better because it is much easier understand the interaction this way. Last but not least, this is a core of the implementation. We might want to tune the number of oom_reaper retries or to think about a more effective tear down when there are multiple oom victims in the queue. I would simply care about those later once we see a clear evidence that this is needed. Ideally with a workload description and some numbers. If this looks like a feasible idea I can spend on this more time and turn it into something meargable.
