Re: Understanding an OOM

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, 28 Nov 2023, Joakim Tjernlund wrote:

> I got this OOM I am trying to make sense of:
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.783305] xr-pm-agent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.793434] CPU: 0 PID: 341 Comm: xr-pm-agent Not tainted 5.15.129-xr-linux #1
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.800653] Hardware name: infinera (DT)
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.804835] Call trace:
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.807277]  dump_backtrace+0x0/0x148
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.810979]  show_stack+0x14/0x1c
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.814296]  dump_stack_lvl+0x64/0x7c
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.817982]  dump_stack+0x14/0x2c
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.821299]  dump_header+0x64/0x1fc
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.824790]  oom_kill_process+0xc0/0x28c
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.828726]  out_of_memory+0x2c8/0x2e0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.832474]  __alloc_pages_slowpath.constprop.0+0x4f4/0x5b0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.838050]  __alloc_pages+0xcc/0xdc
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.841625]  __page_cache_alloc+0x18/0x20
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.845652]  pagecache_get_page+0x14c/0x1bc
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.849833]  filemap_fault+0x1f4/0x390
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.853580]  __do_fault+0x48/0x78
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.856911]  __handle_mm_fault+0x35c/0x7c0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.861010]  handle_mm_fault+0x2c/0xc4
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.864759]  do_page_fault+0x224/0x350
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.868515]  do_translation_fault+0x3c/0x58
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.872697]  do_mem_abort+0x40/0xa4
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.876184]  el0_ia+0x74/0xc8
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.879157]  el0t_32_sync_handler+0xa8/0xe8
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.883338]  el0t_32_sync+0x15c/0x160
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.887033] Mem-Info:
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318] active_anon:20 inactive_anon:2478 isolated_anon:0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318]  active_file:2 inactive_file:3 isolated_file:3
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318]  unevictable:266 dirty:0 writeback:0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318]  slab_reclaimable:371 slab_unreclaimable:2213
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318]  mapped:2 shmem:0 pagetables:302 bounce:0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318]  kernel_misc_reclaimable:0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.889318]  free:1055 free_pcp:31 free_cma:0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.923588] Node 0 active_anon:80kB inactive_anon:9912kB active_file:8kB inactive_file:12kB unevictable:1064kB isolated(anon):0kB
> isolated(file):12kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.947633] Normal free:4220kB min:128kB low:160kB high:192kB reserved_highatomic:4096KB active_anon:80kB inactive_anon:9912kB
> active_file:8kB inactive_file:12kB unevictable:1064kB writepending:0kB present:36864kB managed:294

The reserved_highatomic looks strange because 
"free" - "reserved_highatomic" brings us very close to "min".

So we'd have to look into whether reserved_highatomic is getting freed up 
appropriately on such small systems.  But I think the first order of 
business would be to determine if you can reproduce this on a more recent 
upstream kernel.  v5.15 is over two years old now :/

> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.973759] lowmem_reserve[]: 0 0
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.977089] Normal: 179*4kB (UMH) 222*8kB (UMH) 82*16kB (UMH) 13*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> 4220kB
> Mar  9 13:08:40 xr kern.warn kernel: [ 2026.989752] 272 total pagecache pages
> Mar  9 13:08:41 xr kern.warn kernel: [ 2026.993415] 9216 pages RAM
> Mar  9 13:08:41 xr kern.warn kernel: [ 2026.996124] 0 pages HighMem/MovableOnly
> Mar  9 13:08:41 xr kern.warn kernel: [ 2026.999960] 1855 pages reserved
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.003103] Tasks state (memory values in pages):
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.007805] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.016418] [     77]     0    77      380       23    28672        0             0 mdev
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.024511] [    224]     0   224      364        9    28672        0             0 syslogd
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.032863] [    227]     0   227      364        9    24576        0             0 klogd
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.041040] [    246]     0   246      301       10    20480        0             0 dropbear
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.049480] [    252]     0   252      337       38    28672        0             0 dhcpcd
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.057744] [    263]     0   263      899       33    32768        0             0 watchdog
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.066183] [    265]     0   265      741       33    28672        0             0 rpmsg_broker
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.074967] [    267]     0   267    75718     1243   618496        0             0 waactrl-main
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.083751] [    325]     0   325     1133       94    28672        0             0 xr-fm-agent
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.092449] [    329]     0   329      797       40    32768        0             0 factory_reset
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.101322] [    333]     0   333     1294       62    36864        0             0 xr-cm-agent
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.110019] [    335]     0   335      707       32    32768        0             0 mmcu-agent
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.118629] [    337]     0   337     4054      425    40960        0             0 xr-swm-agent
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.127412] [    341]     0   341     1129       95    32768        0             0 xr-pm-agent
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.136109] [    359]     0   359      734       33    28672        0             0 process_supervi
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.145153] [    368]     0   368     1310      198    32768        0             0 swupdate
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.153591] [    382]     0   382      370       13    28672        0             0 sh
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.161509] [   4007]     0  4007      306       13    20480        0             0 dropbear
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.169948] [   4010]     0  4010      366       12    24576        0             0 sh
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.177865] [   4034]     0  4034      833       52    28672        0             0 xr-swm-install-
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.186908] [    388]     0   388      546       25    32768        0             0 ip
> Mar  9 13:08:41 xr kern.info kernel: [ 2027.194824] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=waactrl-main,pid=267,uid=0
> Mar  9 13:08:41 xr kern.err kernel: [ 2027.203573] Out of memory: Killed process 267 (waactrl-main) total-vm:302872kB, anon-rss:4972kB, file-rss:0kB, shmem-rss:0kB, UID:0
> pgtables:604kB oom_score_adj:0
> 
> There is free mem(free:1055) but I note this too(0 pages HighMem/MovableOnly) and reserved_highatomic:4096KB
> Don't understand why I got reserved_highatomic memory and why there are 0 movable pages ?
> 
> This is an embedded small ARM64 kernel 5.15.129 (ARM32 musl user space) with no SWAP.
> Any clues?
> 
>  Jocke
> 
> 




[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux