> You can either try to use cgroup v2 which has much better memcg aware dirty
> throttling implementation so such a large amount of dirty pages doesn't
> accumulate in the first place
I'd love to use cgroup v2, however this is docker + kubernetes so that
would require a lot of changes on our end to make happen, given how
recently container runtimes gained cgroup v2 support.
> I pressume you are using defaults for
> /proc/sys/vm/dirty_{background_}ratio which is a percentage of the
> available memory. I would recommend using their resp. *_bytes
> alternatives and use something like 500M for background and 800M for
> dirty_bytes.
We're using the defaults right now, however, given that this is a
containerized environment, it's problematic to set these values too
low system-wide since the containers all have dedicated volumes with
varying performance (from as low as 100MB/sec to gigabyes). Looking
around, I see that there were patches in the past to set per-cgroup
vm.dirty settings, however it doesn't look like those ever made it
into the kernel unless I'm missing something. In practice, maybe 500M
and 800M wouldn't be so bad though and may improve latency in other
ways. The other problem is that this also sets an upper bound on the
minimum container size for anything that does do IO. That said, I'll
still I'll tune these settings in our infrastructure and see how
things go, but it sounds like something should be done inside the
kernel to help this situation, since it's so easy to trigger, but
looking at the threads that led to the commits you referenced, I can
see that this is complicated.
Thanks,
Paul
On Wed, Apr 15, 2020 at 2:51 AM Michal Hocko <mhocko@xxxxxxxxxx> wrote:
>
> On Tue 14-04-20 21:25:58, Andrew Morton wrote:
> [...]
> > > Upon upgrading to kernel 5.4, we see constant OOM kills in database containers
> > > that are restoring from backups, with nearly no RSS memory usage. It appears
> > > all the memory is consumed by file_dirty, with applications using minimal
> > > memory. On kernel 4.14.146 and 4.19.75, we do not see this problem, so it
> > > appears to be a new regression.
>
> OK, this is interesting. Because the memcg oom handling has changed in
> 4.19. Older kernels triggered memcg oom only from the page fault path
> while your stack trace is pointing to the write(2) syscall. But if you
> do not see any problem with 4.19 then this is not it.
>
> [...]
>
> > > gfp_mask=0x101c4a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE)
> [...]
> > > memory: usage 1536000kB, limit 1536000kB, failcnt 0
> > > memory+swap: usage 1536000kB, limit 1536000kB, failcnt 490221
> > > kmem: usage 23164kB, limit 9007199254740988kB, failcnt 0
>
> Based on the output I assume you are using cgroup v1
>
> > > Memory cgroup stats for
> > > /kubepods/burstable/pod6900693c-8b2c-4efe-ab52-26e4a6bd9e4c/83216944bb43baf32f0d43ef12c85ebaa2767b3f51846f5fa438bba00b4636d8:
> > > anon 72507392
> > > file 1474740224
> > > kernel_stack 774144
> > > slab 18673664
> > > sock 0
> > > shmem 0
> > > file_mapped 0
> > > file_dirty 1413857280
> > > file_writeback 60555264
>
> This seems to be the crux of the problem. You cannot swap out any memory
> due to memory+swap limit and 95% of the file LRU is dirty. Quite a lot
> of dirty memory to flush. This alone shouldn't be a disaster because
> cgroup v1 does have a hack to throttle the memory reclaim in presence of
> dirty/writeback pages. But note the gfp_mask for the allocation. It
> says GFP_NOFS which means that we cannot apply the throttling and have
> to give up. We used to retry the reclaim even though not much could be
> done with a restricted allocation context and that led to lockups. Then
> we have merged f9c645621a28 ("memcg, oom: don't require __GFP_FS when
> invoking memcg OOM killer") in 5.4 and it has been marked for stable
> trees (4.19+). And this is likely the primary culprit of the issue you
> are seeing.
>
> Now what to do about that. Reverting f9c645621a28 doesn't sound like a
> feasible solution. We could try to put a sleep for restricted
> allocations after memory reclaim fails but we know from the past
> experience this is a bit fishy because a sleep without any feedback on
> the flushing is just not going to work reliably.
>
> Another possibility is to workaround the problem by configuration. You
> can either try to use cgroup v2 which has much better memcg aware dirty
> throttling implementation so such a large amount of dirty pages doesn't
> accumulate in the first place or you can reconfigure global dirty
> limits. I pressume you are using defaults for
> /proc/sys/vm/dirty_{background_}ratio which is a percentage of the
> available memory. I would recommend using their resp. *_bytes
> alternatives and use something like 500M for background and 800M for
> dirty_bytes. That should help in your current situation. The overall IO
> throughput might be smaller so you might need to tune those values a
> bit.
>
> HTH
> --
> Michal Hocko
> SUSE Labs
