On Thu 09-04-20 12:17:33, Bruno Prémont wrote:
> On Thu, 9 Apr 2020 11:46:15 Michal Hocko <mhocko@xxxxxxxxxx> wrote:
> > [Cc Chris]
> >
> > On Thu 09-04-20 11:25:05, Bruno Prémont wrote:
> > > Hi,
> > >
> > > Upgrading from 5.1 kernel to 5.6 kernel on a production system using
> > > cgroups (v2) and having backup process in a memory.high=2G cgroup
> > > sees backup being highly throttled (there are about 1.5T to be
> > > backuped).
> >
> > What does /proc/sys/vm/dirty_* say?
>
> /proc/sys/vm/dirty_background_bytes:0
> /proc/sys/vm/dirty_background_ratio:10
> /proc/sys/vm/dirty_bytes:0
> /proc/sys/vm/dirty_expire_centisecs:3000
> /proc/sys/vm/dirty_ratio:20
> /proc/sys/vm/dirty_writeback_centisecs:500
Sorry, but I forgot ask for the total amount of memory. But it seems
this is 64GB and 10% dirty ration might mean a lot of dirty memory.
Does the same happen if you reduce those knobs to something smaller than
2G? _bytes alternatives should be useful for that purpose.
[...]
> > Is it possible that the reclaim is not making progress on too many
> > dirty pages and that triggers the back off mechanism that has been
> > implemented recently in 5.4 (have a look at 0e4b01df8659 ("mm,
> > memcg: throttle allocators when failing reclaim over memory.high")
> > and e26733e0d0ec ("mm, memcg: throttle allocators based on
> > ancestral memory.high").
>
> Could be though in that case it's throttling the wrong task/cgroup
> as far as I can see (at least from cgroup's memory stats) or being
> blocked by state external to the cgroup.
> Will have a look at those patches so get a better idea at what they
> change.
Could you check where is the task of your interest throttled?
/proc/<pid>/stack should give you a clue.
--
Michal Hocko
SUSE Labs
