On Tue, 28 Nov 2023, Joakim Tjernlund wrote: > I got this OOM I am trying to make sense of: > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT) > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace: > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.807277] dump_backtrace+0x0/0x148 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.810979] show_stack+0x14/0x1c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.814296] dump_stack_lvl+0x64/0x7c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.817982] dump_stack+0x14/0x2c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.821299] dump_header+0x64/0x1fc > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.824790] oom_kill_process+0xc0/0x28c > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.828726] out_of_memory+0x2c8/0x2e0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.832474] __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.838050] __alloc_pages+0xcc/0xdc > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.841625] __page_cache_alloc+0x18/0x20 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.845652] pagecache_get_page+0x14c/0x1bc > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.849833] filemap_fault+0x1f4/0x390 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.853580] __do_fault+0x48/0x78 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.856911] __handle_mm_fault+0x35c/0x7c0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.861010] handle_mm_fault+0x2c/0xc4 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.864759] do_page_fault+0x224/0x350 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.868515] do_translation_fault+0x3c/0x58 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.872697] do_mem_abort+0x40/0xa4 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.876184] el0_ia+0x74/0xc8 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.879157] el0t_32_sync_handler+0xa8/0xe8 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.883338] el0t_32_sync+0x15c/0x160 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info: > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_file:2 inactive_file:3 isolated_file:3 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] unevictable:266 dirty:0 writeback:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] slab_reclaimable:371 slab_unreclaimable:2213 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] mapped:2 shmem:0 pagetables:302 bounce:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] kernel_misc_reclaimable:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.889318] free:1055 free_pcp:31 free_cma:0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB > isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB > active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294 The reserved_highatomic looks strange because "free" - "reserved_highatomic" brings us very close to "min". So we'd have to look into whether reserved_highatomic is getting freed up appropriately on such small systems. But I think the first order of business would be to determine if you can reproduce this on a more recent upstream kernel. v5.15 is over two years old now :/ > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0 > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = > 4220kB > Mar 9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly > Mar 9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved > Mar 9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages): > Mar 9 13:08:41 xr kern.info kernel: [ 2027.007805] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name > Mar 9 13:08:41 xr kern.info kernel: [ 2027.016418] [ 77] 0 77 380 23 28672 0 0 mdev > Mar 9 13:08:41 xr kern.info kernel: [ 2027.024511] [ 224] 0 224 364 9 28672 0 0 syslogd > Mar 9 13:08:41 xr kern.info kernel: [ 2027.032863] [ 227] 0 227 364 9 24576 0 0 klogd > Mar 9 13:08:41 xr kern.info kernel: [ 2027.041040] [ 246] 0 246 301 10 20480 0 0 dropbear > Mar 9 13:08:41 xr kern.info kernel: [ 2027.049480] [ 252] 0 252 337 38 28672 0 0 dhcpcd > Mar 9 13:08:41 xr kern.info kernel: [ 2027.057744] [ 263] 0 263 899 33 32768 0 0 watchdog > Mar 9 13:08:41 xr kern.info kernel: [ 2027.066183] [ 265] 0 265 741 33 28672 0 0 rpmsg_broker > Mar 9 13:08:41 xr kern.info kernel: [ 2027.074967] [ 267] 0 267 75718 1243 618496 0 0 waactrl-main > Mar 9 13:08:41 xr kern.info kernel: [ 2027.083751] [ 325] 0 325 1133 94 28672 0 0 xr-fm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.092449] [ 329] 0 329 797 40 32768 0 0 factory_reset > Mar 9 13:08:41 xr kern.info kernel: [ 2027.101322] [ 333] 0 333 1294 62 36864 0 0 xr-cm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.110019] [ 335] 0 335 707 32 32768 0 0 mmcu-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.118629] [ 337] 0 337 4054 425 40960 0 0 xr-swm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.127412] [ 341] 0 341 1129 95 32768 0 0 xr-pm-agent > Mar 9 13:08:41 xr kern.info kernel: [ 2027.136109] [ 359] 0 359 734 33 28672 0 0 process_supervi > Mar 9 13:08:41 xr kern.info kernel: [ 2027.145153] [ 368] 0 368 1310 198 32768 0 0 swupdate > Mar 9 13:08:41 xr kern.info kernel: [ 2027.153591] [ 382] 0 382 370 13 28672 0 0 sh > Mar 9 13:08:41 xr kern.info kernel: [ 2027.161509] [ 4007] 0 4007 306 13 20480 0 0 dropbear > Mar 9 13:08:41 xr kern.info kernel: [ 2027.169948] [ 4010] 0 4010 366 12 24576 0 0 sh > Mar 9 13:08:41 xr kern.info kernel: [ 2027.177865] [ 4034] 0 4034 833 52 28672 0 0 xr-swm-install- > Mar 9 13:08:41 xr kern.info kernel: [ 2027.186908] [ 388] 0 388 546 25 32768 0 0 ip > Mar 9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0 > Mar 9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0 > pgtables:604kB oom_score_adj:0 > > There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB > Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ? > > This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP. > Any clues? > > Jocke > >
