On Wed, 2023-11-29 at 11:44 -0800, David Rientjes wrote: > On Tue, 28 Nov 2023, Joakim Tjernlund wrote: > > > I got this OOM I am trying to make sense of: > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT) > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace: > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info: > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB > > isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB > > active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294 > > The reserved_highatomic looks strange because > "free" - "reserved_highatomic" brings us very close to "min". > > So we'd have to look into whether reserved_highatomic is getting freed up > appropriately on such small systems. But I think the first order of Thanks for looking Do I need reserved_highatomic at all on a small system? What would happen if I just forced it to 0 ? > business would be to determine if you can reproduce this on a more recent > upstream kernel. v5.15 is over two years old now :/ 5.15.129 isn't that old, Aug. this year. I cannot update to latest and greatest on this system, will be a big effort. Jocke > > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0 > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > > 4220kB > > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages > > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM > > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly > > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages): > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install- > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip > > Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0 > > Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0 > > pgtables:604kB oom_score_adj:0 > > > > There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB > > Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ? > > > > This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP. > > Any clues? > > > > Jocke > > > >
