Hey there, I'm seeing page allocation failures across the Cloudflare fleet, typically during the network RX path, when trying to allocate order-0 pages in interrupt context. The machines appear to be under memory pressure because the code that gets interrupted is shrink_folio_list(). Below is an example stacktrace. Does anyone have any pointers on how to dig into this some more? It appears as though the machines are not able to reclaim memory fast enough when under pressure. Happy to provide more metrics or stats on request. Thanks, Matt ----8<---- kswapd1: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0-7 CPU: 10 PID: 696 Comm: kswapd1 Kdump: loaded Tainted: G O 6.6.43-CUSTOM #1 Hardware name: MACHINE Call Trace: <IRQ> dump_stack_lvl+0x3c/0x50 warn_alloc+0x13a/0x1c0 __alloc_pages_slowpath.constprop.0+0xc9d/0xd10 ? srso_alias_return_thunk+0x5/0xfbef5 ? __alloc_pages_bulk+0x3a0/0x630 __alloc_pages+0x327/0x340 __napi_alloc_skb+0x16d/0x1f0 bnxt_rx_page_skb+0x96/0x1b0 [bnxt_en] bnxt_rx_pkt+0x201/0x15e0 [bnxt_en] ? skb_release_data+0x14f/0x1b0 __bnxt_poll_work+0x156/0x2b0 [bnxt_en] bnxt_poll+0xd9/0x1c0 [bnxt_en] ? srso_alias_return_thunk+0x5/0xfbef5 __napi_poll+0x2b/0x1b0 bpf_trampoline_6442524138+0x7d/0x1000 __napi_poll+0x5/0x1b0 net_rx_action+0x342/0x740 ? srso_alias_return_thunk+0x5/0xfbef5 handle_softirqs+0xcf/0x2b0 irq_exit_rcu+0x6c/0x90 sysvec_apic_timer_interrupt+0x72/0x90 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:queued_spin_lock_slowpath+0x260/0x2b0 Code: 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 c0 30 03 00 48 03 04 d5 a0 d7 10 9c 48 89 28 8b 45 08 85 c0 75 09 f3 90 8b 45 08 <85> c0 74 f7 48 8b 55 00 48 85 d2 74 83 0f 0d 0a e9 7b ff ff ff 65 RSP: 0018:ffffc9000f9cb768 EFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff88905a3a9880 RCX: 0000000000000001 RDX: 000000000000001b RSI: 0000000000700000 RDI: ffff88905a3a9880 RBP: ffff88902f5330c0 R08: ffffc9000f9cb750 R09: 0000000000000000 R10: 0000000000000000 R11: 0000603fce623320 R12: 00000000002c0000 R13: 0000000000000001 R14: 00000000002c0000 R15: ffff889062f84a00 zs_malloc+0x9d/0x520 [zsmalloc] ? srso_alias_return_thunk+0x5/0xfbef5 ? __zstd_compress+0x60/0xa0 [zstd] zram_submit_bio+0x8d1/0x9f0 [zram] ? srso_alias_return_thunk+0x5/0xfbef5 __submit_bio+0xaa/0x160 submit_bio_noacct_nocheck+0x145/0x380 ? submit_bio_noacct+0x24/0x4c0 submit_bio_wait+0x5b/0xc0 swap_writepage_bdev_sync+0xf8/0x170 ? __pfx_submit_bio_wait_endio+0x10/0x10 swap_writepage+0x36/0x80 pageout+0xc8/0x240 shrink_folio_list+0x489/0xd60 shrink_lruvec+0x5a8/0xc40 shrink_node+0x2c5/0x7a0 balance_pgdat+0x32d/0x740 kswapd+0x205/0x400 ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_kswapd+0x10/0x10 kthread+0xe8/0x120 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1b/0x30 </TASK> Mem-Info: active_anon:14289951 inactive_anon:25056935 isolated_anon:1577 active_file:3254095 inactive_file:3963476 isolated_file:1 unevictable:4 dirty:305545 writeback:132 slab_reclaimable:2916775 slab_unreclaimable:1689088 mapped:2592762 shmem:1980658 pagetables:530605 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:618653 free_pcp:129763 free_cma:0 Node 0 active_anon:6461468kB inactive_anon:11667080kB active_file:1971908kB inactive_file:2302944kB unevictable:0kB isolated(anon):960kB isolated(file):0kB mapped:1070000kB dirty:110140kB writeback:64kB shmem:842272kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:37624kB pagetables:235212kB sec_pagetables:0kB all_unreclaimable? no Node 1 active_anon:7027824kB inactive_anon:12544448kB active_file:1695500kB inactive_file:2093056kB unevictable:0kB isolated(anon):308kB isolated(file):0kB mapped:1694880kB dirty:163436kB writeback:24kB shmem:1090692kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:31860kB pagetables:231608kB sec_pagetables:0kB all_unreclaimable? no Node 2 active_anon:7168612kB inactive_anon:11850084kB active_file:1669812kB inactive_file:1870596kB unevictable:0kB isolated(anon):144kB isolated(file):0kB mapped:1420628kB dirty:105912kB writeback:24kB shmem:1092068kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:40220kB pagetables:263428kB sec_pagetables:0kB all_unreclaimable? no Node 3 active_anon:7160892kB inactive_anon:12851880kB active_file:1453156kB inactive_file:1884092kB unevictable:0kB isolated(anon):452kB isolated(file):0kB mapped:1199768kB dirty:124548kB writeback:72kB shmem:965128kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2048kB writeback_tmp:0kB kernel_stack:27124kB pagetables:284676kB sec_pagetables:0kB all_unreclaimable? no Node 4 active_anon:7505196kB inactive_anon:12764280kB active_file:1466756kB inactive_file:1878740kB unevictable:16kB isolated(anon):640kB isolated(file):0kB mapped:1170484kB dirty:136668kB writeback:44kB shmem:986212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2048kB writeback_tmp:0kB kernel_stack:32380kB pagetables:312216kB sec_pagetables:0kB all_unreclaimable? no Node 5 active_anon:7169752kB inactive_anon:12867040kB active_file:1769832kB inactive_file:1809448kB unevictable:0kB isolated(anon):1008kB isolated(file):0kB mapped:1589272kB dirty:128616kB writeback:112kB shmem:1108816kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:32784kB pagetables:278392kB sec_pagetables:0kB all_unreclaimable? no Node 6 active_anon:7333288kB inactive_anon:12854340kB active_file:1504536kB inactive_file:2096488kB unevictable:0kB isolated(anon):1336kB isolated(file):4kB mapped:1117792kB dirty:228512kB writeback:92kB shmem:958680kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:43852kB pagetables:254060kB sec_pagetables:0kB all_unreclaimable? no Node 7 active_anon:7332772kB inactive_anon:12828588kB active_file:1484880kB inactive_file:1918540kB unevictable:0kB isolated(anon):1460kB isolated(file):0kB mapped:1108224kB dirty:224348kB writeback:96kB shmem:878764kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2048kB writeback_tmp:0kB kernel_stack:35580kB pagetables:262828kB sec_pagetables:0kB all_unreclaimable? no Node 0 DMA free:11264kB boost:0kB min:48kB low:60kB high:72kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 2095 31529 31529 Node 0 DMA32 free:118988kB boost:0kB min:6832kB low:8976kB high:11120kB reserved_highatomic:0KB active_anon:445316kB inactive_anon:780792kB active_file:122148kB inactive_file:151592kB unevictable:0kB writepending:1464kB present:2735864kB managed:2145496kB mlocked:0kB bounce:0kB free_pcp:20468kB local_pcp:48kB free_cma:0kB lowmem_reserve[]: 0 0 29434 29434 Node 0 Normal free:266252kB boost:0kB min:95988kB low:126128kB high:156268kB reserved_highatomic:305152KB active_anon:6016024kB inactive_anon:10884436kB active_file:1849108kB inactive_file:2149856kB unevictable:0kB writepending:108740kB present:30670848kB managed:30141044kB mlocked:0kB bounce:0kB free_pcp:37432kB local_pcp:84kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 1 Normal free:290496kB boost:0kB min:105164kB low:138184kB high:171204kB reserved_highatomic:333824KB active_anon:7028084kB inactive_anon:12543028kB active_file:1694884kB inactive_file:2092728kB unevictable:0kB writepending:163200kB present:33552384kB managed:33022704kB mlocked:0kB bounce:0kB free_pcp:53668kB local_pcp:892kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 2 Normal free:295000kB boost:0kB min:105172kB low:138196kB high:171220kB reserved_highatomic:333824KB active_anon:7168872kB inactive_anon:11848752kB active_file:1668876kB inactive_file:1871016kB unevictable:0kB writepending:106604kB present:33554432kB managed:33024756kB mlocked:0kB bounce:0kB free_pcp:48468kB local_pcp:752kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 3 Normal free:308228kB boost:0kB min:105012kB low:137984kB high:170956kB reserved_highatomic:333824KB active_anon:7164068kB inactive_anon:12847600kB active_file:1453016kB inactive_file:1885952kB unevictable:0kB writepending:126480kB present:33553408kB managed:32974232kB mlocked:0kB bounce:0kB free_pcp:64400kB local_pcp:732kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 4 Normal free:271672kB boost:0kB min:105172kB low:138196kB high:171220kB reserved_highatomic:333824KB active_anon:7505196kB inactive_anon:12763688kB active_file:1465932kB inactive_file:1880212kB unevictable:16kB writepending:137892kB present:33554432kB managed:33024756kB mlocked:16kB bounce:0kB free_pcp:60204kB local_pcp:632kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 5 Normal free:291824kB boost:0kB min:105168kB low:138188kB high:171208kB reserved_highatomic:333824KB active_anon:7169428kB inactive_anon:12866872kB active_file:1769184kB inactive_file:1811512kB unevictable:0kB writepending:131024kB present:33553408kB managed:33023728kB mlocked:0kB bounce:0kB free_pcp:78708kB local_pcp:568kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 6 Normal free:310936kB boost:0kB min:105172kB low:138196kB high:171220kB reserved_highatomic:333824KB active_anon:7333792kB inactive_anon:12852816kB active_file:1503264kB inactive_file:2097500kB unevictable:0kB writepending:229284kB present:33554432kB managed:33024756kB mlocked:0kB bounce:0kB free_pcp:74936kB local_pcp:796kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 7 Normal free:309668kB boost:0kB min:105112kB low:138116kB high:171120kB reserved_highatomic:333824KB active_anon:7331892kB inactive_anon:12827964kB active_file:1484024kB inactive_file:1920356kB unevictable:0kB writepending:226576kB present:33541120kB managed:33005940kB mlocked:0kB bounce:0kB free_pcp:80748kB local_pcp:704kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB Node 0 DMA32: 2225*4kB (UME) 338*8kB (UME) 178*16kB (UME) 459*32kB (UME) 215*64kB (UME) 115*128kB (ME) 86*256kB (UME) 35*512kB (UME) 4*1024kB (UM) 6*2048kB (M) 1*4096kB (U) = 118036kB Node 0 Normal: 797*4kB (H) 871*8kB (H) 802*16kB (H) 804*32kB (H) 601*64kB (H) 310*128kB (H) 164*256kB (H) 67*512kB (H) 25*1024kB (H) 14*2048kB (H) 2*4096kB (H) = 265612kB Node 1 Normal: 507*4kB (H) 680*8kB (H) 682*16kB (H) 699*32kB (H) 589*64kB (H) 363*128kB (H) 211*256kB (H) 93*512kB (H) 37*1024kB (H) 13*2048kB (H) 0*4096kB = 291052kB Node 2 Normal: 598*4kB (H) 843*8kB (H) 740*16kB (H) 735*32kB (H) 507*64kB (H) 298*128kB (H) 175*256kB (H) 102*512kB (H) 37*1024kB (H) 21*2048kB (H) 1*4096kB (H) = 297104kB Node 3 Normal: 440*4kB (H) 509*8kB (H) 493*16kB (H) 559*32kB (H) 438*64kB (H) 304*128kB (H) 197*256kB (H) 126*512kB (H) 50*1024kB (H) 21*2048kB (H) 0*4096kB = 307704kB Node 4 Normal: 604*4kB (H) 716*8kB (H) 674*16kB (H) 819*32kB (H) 544*64kB (H) 303*128kB (H) 182*256kB (H) 74*512kB (H) 24*1024kB (H) 20*2048kB (H) 0*4096kB = 268752kB Node 5 Normal: 809*4kB (H) 873*8kB (H) 775*16kB (H) 749*32kB (H) 414*64kB (H) 254*128kB (H) 154*256kB (H) 90*512kB (H) 37*1024kB (H) 31*2048kB (H) 0*4096kB = 292476kB Node 6 Normal: 659*4kB (H) 689*8kB (H) 708*16kB (H) 851*32kB (H) 592*64kB (H) 386*128kB (H) 226*256kB (H) 91*512kB (H) 40*1024kB (H) 13*2048kB (H) 1*4096kB (H) = 310132kB Node 7 Normal: 898*4kB (H) 907*8kB (H) 893*16kB (H) 897*32kB (H) 597*64kB (H) 375*128kB (H) 203*256kB (H) 86*512kB (H) 29*1024kB (H) 20*2048kB (H) 0*4096kB = 306704kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 9214746 total pagecache pages 17797 pages in swap cache Free swap = 208645424kB Total swap = 263402492kB 67071581 pages RAM 0 pages HighMem/MovableOnly 1220888 pages reserved
