On Fri, May 19, 2017 at 04:37:27AM +1000, Balbir Singh wrote: > On Fri, May 19, 2017 at 3:30 AM, Michal Hocko <mhocko@xxxxxxxxxx> wrote: > > On Thu 18-05-17 17:28:04, Roman Gushchin wrote: > >> Traditionally, the OOM killer is operating on a process level. > >> Under oom conditions, it finds a process with the highest oom score > >> and kills it. > >> > >> This behavior doesn't suit well the system with many running > >> containers. There are two main issues: > >> > >> 1) There is no fairness between containers. A small container with > >> a few large processes will be chosen over a large one with huge > >> number of small processes. > >> > >> 2) Containers often do not expect that some random process inside > >> will be killed. So, in general, a much safer behavior is > >> to kill the whole cgroup. Traditionally, this was implemented > >> in userspace, but doing it in the kernel has some advantages, > >> especially in a case of a system-wide OOM. > >> > >> To address these issues, cgroup-aware OOM killer is introduced. > >> Under OOM conditions, it looks for a memcg with highest oom score, > >> and kills all processes inside. > >> > >> Memcg oom score is calculated as a size of active and inactive > >> anon LRU lists, unevictable LRU list and swap size. > >> > >> For a cgroup-wide OOM, only cgroups belonging to the subtree of > >> the OOMing cgroup are considered. > > > > While this might make sense for some workloads/setups it is not a > > generally acceptable policy IMHO. We have discussed that different OOM > > policies might be interesting few years back at LSFMM but there was no > > real consensus on how to do that. One possibility was to allow bpf like > > mechanisms. Could you explore that path? > > I agree, I think it needs more thought. I wonder if the real issue is something > else. For example > > 1. Did we overcommit a particular container too much? Imagine, you have a machine with multiple containers, each with it's own process tree, and the machine is overcommited, i.e. sum of container's memory limits is larger the amount available RAM. In a case of a system-wide OOM some random container will be affected. Historically, this problem was solving by some user-space daemon, which was monitoring OOM events and cleaning up affected containers. But this approach can't solve the main problem: non-optimal selection of a victim. > 2. Do we need something like https://urldefense.proofpoint.com/v2/url?u=https-3A__lwn.net_Articles_604212_&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=9jV4id5lmsjFJj1kQjJk0auyQ3bzL27-f6Ur6ZNw36c&s=ElsS25CoZSPba6ke7O-EIsR7lN0psP6tDVyLnGqCMfs&e= to solve > the problem? I don't think it's related. > 3. We have oom notifiers now, could those be used (assuming you are interested > in non memcg related OOM's affecting a container They can be used to inform an userspace daemon about an already happened OOM, but they do not affect victim selection. > 4. How do we determine limits for these containers? From a fariness > perspective Limits are usually set from some high-level understanding of the nature of tasks which are working inside, but overcommiting the machine is a common place, I assume. Thank you! Roman -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
