On Tue 14-04-20 21:25:58, Andrew Morton wrote:
[...]
> > Upon upgrading to kernel 5.4, we see constant OOM kills in database containers
> > that are restoring from backups, with nearly no RSS memory usage. It appears
> > all the memory is consumed by file_dirty, with applications using minimal
> > memory. On kernel 4.14.146 and 4.19.75, we do not see this problem, so it
> > appears to be a new regression.
OK, this is interesting. Because the memcg oom handling has changed in
4.19. Older kernels triggered memcg oom only from the page fault path
while your stack trace is pointing to the write(2) syscall. But if you
do not see any problem with 4.19 then this is not it.
[...]
> > gfp_mask=0x101c4a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE)
[...]
> > memory: usage 1536000kB, limit 1536000kB, failcnt 0
> > memory+swap: usage 1536000kB, limit 1536000kB, failcnt 490221
> > kmem: usage 23164kB, limit 9007199254740988kB, failcnt 0
Based on the output I assume you are using cgroup v1
> > Memory cgroup stats for
> > /kubepods/burstable/pod6900693c-8b2c-4efe-ab52-26e4a6bd9e4c/83216944bb43baf32f0d43ef12c85ebaa2767b3f51846f5fa438bba00b4636d8:
> > anon 72507392
> > file 1474740224
> > kernel_stack 774144
> > slab 18673664
> > sock 0
> > shmem 0
> > file_mapped 0
> > file_dirty 1413857280
> > file_writeback 60555264
This seems to be the crux of the problem. You cannot swap out any memory
due to memory+swap limit and 95% of the file LRU is dirty. Quite a lot
of dirty memory to flush. This alone shouldn't be a disaster because
cgroup v1 does have a hack to throttle the memory reclaim in presence of
dirty/writeback pages. But note the gfp_mask for the allocation. It
says GFP_NOFS which means that we cannot apply the throttling and have
to give up. We used to retry the reclaim even though not much could be
done with a restricted allocation context and that led to lockups. Then
we have merged f9c645621a28 ("memcg, oom: don't require __GFP_FS when
invoking memcg OOM killer") in 5.4 and it has been marked for stable
trees (4.19+). And this is likely the primary culprit of the issue you
are seeing.
Now what to do about that. Reverting f9c645621a28 doesn't sound like a
feasible solution. We could try to put a sleep for restricted
allocations after memory reclaim fails but we know from the past
experience this is a bit fishy because a sleep without any feedback on
the flushing is just not going to work reliably.
Another possibility is to workaround the problem by configuration. You
can either try to use cgroup v2 which has much better memcg aware dirty
throttling implementation so such a large amount of dirty pages doesn't
accumulate in the first place or you can reconfigure global dirty
limits. I pressume you are using defaults for
/proc/sys/vm/dirty_{background_}ratio which is a percentage of the
available memory. I would recommend using their resp. *_bytes
alternatives and use something like 500M for background and 800M for
dirty_bytes. That should help in your current situation. The overall IO
throughput might be smaller so you might need to tune those values a
bit.
HTH
--
Michal Hocko
SUSE Labs
