On Wed, Nov 25, 2020 at 12:39:56PM +0100, Bruno Prémont wrote:
> Hello,
>
> On a production system I've encountered a rather harsh behavior from
> kernel in the context of memory cgroup (v2) after updating kernel from
> 5.7 series to 5.9 series.
>
>
> It seems like kernel is reclaiming file cache but leaving inode cache
> (reclaimable slabs) alone in a way that the server ends up trashing and
> maxing out on IO to one of its disks instead of doing actual work.
>
>
> My setup, server has 64G of RAM:
> root
> + system { min=0, low=128M, high=8G, max=8G }
> + base { no specific constraints }
> + backup { min=0, low=32M, high=2G, max=2G }
> + shell { no specific constraints }
> + websrv { min=0, low=4G, high=32G, max=32G }
> + website { min=0, low=16G, high=40T, max=40T }
> + website1 { min=0, low=64M, high=2G, max=2G }
> + website2 { min=0, low=64M, high=2G, max=2G }
> ...
> + remote { min=0, low=1G, high=14G, max=14G }
> + webuser1 { min=0, low=64M, high=2G, max=2G }
> + webuser2 { min=0, low=64M, high=2G, max=2G }
> ...
>
>
> When the server was struggling I've had mostly IO on disk hosting
> system processes and some cache files of websrv processes.
> It seems that running backup does make the issue much more probable.
>
> The processes in websrv are the most impacted by the trashing and this
> is the one with lots of disk cache and inode cache assigned to it.
> (note a helper running in websrv cgroup scan whole file system
> hierarchy once per hour and this keeps inode cache pretty filled.
> Dropping just file cache (about 10G) did not unlock situation but
> dropping reclaimable slabs (inode cache, about 30G) got the system back
> running.
>
>
>
> Some metrics I have collected during a trashing period (metrics
> collected at about 5min interval) - I don't have ful memory.stat
> unfortunately:
>
> system/memory.min 0 = 0
> system/memory.low 134217728 = 134217728
> system/memory.high 8589934592 = 8589934592
> system/memory.max 8589934592 = 8589934592
> system/memory.pressure
> some avg10=54.41 avg60=59.28 avg300=69.46 total=7347640237
> full avg10=27.45 avg60=22.19 avg300=29.28 total=3287847481
> ->
> some avg10=77.25 avg60=73.24 avg300=69.63 total=7619662740
> full avg10=23.04 avg60=25.26 avg300=27.97 total=3401421903
> system/memory.current 262533120 < 263929856
> system/memory.events.local
> low 5399469 = 5399469
> high 0 = 0
> max 112303 = 112303
> oom 0 = 0
> oom_kill 0 = 0
>
> system/base/memory.min 0 = 0
> system/base/memory.low 0 = 0
> system/base/memory.high max = max
> system/base/memory.max max = max
> system/base/memory.pressure
> some avg10=18.89 avg60=20.34 avg300=24.95 total=5156816349
> full avg10=10.90 avg60=8.50 avg300=11.68 total=2253916169
> ->
> some avg10=33.82 avg60=32.26 avg300=26.95 total=5258381824
> full avg10=12.51 avg60=13.01 avg300=12.05 total=2301375471
> system/base/memory.current 31363072 < 32243712
> system/base/memory.events.local
> low 0 = 0
> high 0 = 0
> max 0 = 0
> oom 0 = 0
> oom_kill 0 = 0
>
> system/backup/memory.min 0 = 0
> system/backup/memory.low 33554432 = 33554432
> system/backup/memory.high 2147483648 = 2147483648
> system/backup/memory.max 2147483648 = 2147483648
> system/backup/memory.pressure
> some avg10=41.73 avg60=45.97 avg300=56.27 total=3385780085
> full avg10=21.78 avg60=18.15 avg300=25.35 total=1571263731
> ->
> some avg10=60.27 avg60=55.44 avg300=54.37 total=3599850643
> full avg10=19.52 avg60=20.91 avg300=23.58 total=1667430954
> system/backup/memory.current 222130176 < 222543872
> system/backup/memory.events.local
> low 5446 = 5446
> high 0 = 0
> max 0 = 0
> oom 0 = 0
> oom_kill 0 = 0
>
> system/shell/memory.min 0 = 0
> system/shell/memory.low 0 = 0
> system/shell/memory.high max = max
> system/shell/memory.max max = max
> system/shell/memory.pressure
> some avg10=0.00 avg60=0.12 avg300=0.25 total=1348427661
> full avg10=0.00 avg60=0.04 avg300=0.06 total=493582108
> ->
> some avg10=0.00 avg60=0.00 avg300=0.06 total=1348516773
> full avg10=0.00 avg60=0.00 avg300=0.00 total=493591500
> system/shell/memory.current 8814592 < 8888320
> system/shell/memory.events.local
> low 0 = 0
> high 0 = 0
> max 0 = 0
> oom 0 = 0
> oom_kill 0 = 0
>
> website/memory.min 0 = 0
> website/memory.low 17179869184 = 17179869184
> website/memory.high 45131717672960 = 45131717672960
> website/memory.max 45131717672960 = 45131717672960
> website/memory.pressure
> some avg10=0.00 avg60=0.00 avg300=0.00 total=415009408
> full avg10=0.00 avg60=0.00 avg300=0.00 total=201868483
> ->
> some avg10=0.00 avg60=0.00 avg300=0.00 total=415009408
> full avg10=0.00 avg60=0.00 avg300=0.00 total=201868483
> website/memory.current 11811520512 > 11456942080
> website/memory.events.local
> low 11372142 < 11377350
> high 0 = 0
> max 0 = 0
> oom 0 = 0
> oom_kill 0 = 0
>
> remote/memory.min 0
> remote/memory.low 1073741824
> remote/memory.high 15032385536
> remote/memory.max 15032385536
> remote/memory.pressure
> some avg10=0.00 avg60=0.25 avg300=0.50 total=2017364408
> full avg10=0.00 avg60=0.00 avg300=0.01 total=738071296
> ->
> remote/memory.current 84439040 > 81797120
> remote/memory.events.local
> low 11372142 < 11377350
> high 0 = 0
> max 0 = 0
> oom 0 = 0
> oom_kill 0 = 0
>
> websrv/memory.min 0 = 0
> websrv/memory.low 4294967296 = 4294967296
> websrv/memory.high 34359738368 = 34359738368
> websrv/memory.max 34426847232 = 34426847232
> websrv/memory.pressure
> some avg10=40.38 avg60=62.58 avg300=68.83 total=7760096704
> full avg10=7.80 avg60=10.78 avg300=12.64 total=2254679370
> ->
> some avg10=89.97 avg60=83.78 avg300=72.99 total=8040513640
> full avg10=11.46 avg60=11.49 avg300=11.47 total=2300116237
> websrv/memory.current 18421673984 < 18421936128
> websrv/memory.events.local
> low 0 = 0
> high 0 = 0
> max 0 = 0
> oom 0 = 0
> oom_kill 0 = 0
>
>
>
> Is there something important I'm missing in my setup that could prevent
> things from starving?
>
> Did memory.low meaning change between 5.7 and 5.9? From behavior it
> feels as if inodes are not accounted to cgroup at all and kernel pushes
> cgroups down to their memory.low by killing file cache if there is not
> enough free memory to hold all promises (and not only when a cgroup
> tries to use up to its promised amount of memory).
> As system was trashing as much with 10G of file cache dropped
> (completely unused memory) as with it in use.
>
>
> I will try to create a test-case for it to reproduce it on a test
> machine an be able to verify a fix or eventually bisect to triggering
> patch though it this all rings a bell, please tell!
>
> Note until I have a test-case I'm reluctant to just wait [on
> production system] for next occurrence (usually at unpractical times) to
> gather some more metrics.
Hi Bruno!
Thank you for the report.
Can you, please, check if the following patch fixes the issue?
Thanks!
--
diff --git a/mm/slab.h b/mm/slab.h
index 6cc323f1313a..ef02b841bcd8 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -290,7 +290,7 @@ static inline struct obj_cgroup *memcg_slab_pre_alloc_hook(struct kmem_cache *s,
if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) {
obj_cgroup_put(objcg);
- return NULL;
+ return (struct obj_cgroup *)-1UL;
}
return objcg;
@@ -501,9 +501,13 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s,
return NULL;
if (memcg_kmem_enabled() &&
- ((flags & __GFP_ACCOUNT) || (s->flags & SLAB_ACCOUNT)))
+ ((flags & __GFP_ACCOUNT) || (s->flags & SLAB_ACCOUNT))) {
*objcgp = memcg_slab_pre_alloc_hook(s, size, flags);
+ if (unlikely(*objcgp == (struct obj_cgroup *)-1UL))
+ return NULL;
+ }
+
return s;
}
