On Wed, Nov 30, 2022 at 3:29 PM Roman Gushchin <roman.gushchin@xxxxxxxxx> wrote: > > On Wed, Nov 30, 2022 at 03:01:58PM +0800, chengkaitao wrote: > > From: chengkaitao <pilgrimtao@xxxxxxxxx> > > > > We created a new interface <memory.oom.protect> for memory, If there is > > the OOM killer under parent memory cgroup, and the memory usage of a > > child cgroup is within its effective oom.protect boundary, the cgroup's > > tasks won't be OOM killed unless there is no unprotected tasks in other > > children cgroups. It draws on the logic of <memory.min/low> in the > > inheritance relationship. > > > > It has the following advantages, > > 1. We have the ability to protect more important processes, when there > > is a memcg's OOM killer. The oom.protect only takes effect local memcg, > > and does not affect the OOM killer of the host. > > 2. Historically, we can often use oom_score_adj to control a group of > > processes, It requires that all processes in the cgroup must have a > > common parent processes, we have to set the common parent process's > > oom_score_adj, before it forks all children processes. So that it is > > very difficult to apply it in other situations. Now oom.protect has no > > such restrictions, we can protect a cgroup of processes more easily. The > > cgroup can keep some memory, even if the OOM killer has to be called. > > It reminds me our attempts to provide a more sophisticated cgroup-aware oom > killer. The problem is that the decision which process(es) to kill or preserve > is individual to a specific workload (and can be even time-dependent > for a given workload). So it's really hard to come up with an in-kernel > mechanism which is at the same time flexible enough to work for the majority > of users and reliable enough to serve as the last oom resort measure (which > is the basic goal of the kernel oom killer). > > Previously the consensus was to keep the in-kernel oom killer dumb and reliable > and implement complex policies in userspace (e.g. systemd-oomd etc). > > Is there a reason why such approach can't work in your case? > FWIW we run into similar issues and the systemd-oomd approach doesn't work reliably enough for us to disable the kernel oom-killer. The issue as I understand is when the machine is under heavy memory pressure our userspace oom-killer fails to run quickly enough to save the machine from getting completely stuck. Why our oom-killer fails to run is more nuanced. There are cases where it seems stuck to itself to acquire memory to do the oom-killing or stuck on some lock that needs to be released by a process that itself is stuck trying to acquire memory to release the lock, etc. When the kernel oom-killer does run we would like to shield the important jobs from it and kill the batch jobs or restartable processes instead. So we have a similar feature to what is proposed here internally. Our design is a bit different. For us we enable the userspace to completely override the oom_badness score pretty much: 1. Every process has /proc/pid/oom_score_badness which overrides the kernel's calculation if set. 2. Every memcg has a memory.oom_score_badness which indicates this memcg's oom importance. On global oom the kernel pretty much kills the baddest process in the badesset memcg, so we can 'protect' the important jobs from oom-killing that way. I haven't tried upstreaming this because I assume there would be little appetite for it in a general use case, but if the general use case is interesting for some it would be good to collaborate on some way for folks that enable the kernel oom-killer to shield certain jobs that are important. > Thanks! >
