On 29/07/2024 15:40, Michal Hocko wrote: > On Mon 29-07-24 06:34:21, Zhijian Li (Fujitsu) wrote: >> >> >> on 7/29/2024 2:13 PM, Michal Hocko wrote: >>> On Mon 29-07-24 02:14:13, Zhijian Li (Fujitsu) wrote: >>>> >>>> On 29/07/2024 08:37, Li Zhijian wrote: >>>>> Michal, >>>>> >>>>> Sorry to the late reply. >>>>> >>>>> >>>>> On 26/07/2024 17:17, Michal Hocko wrote: >>>>>> On Fri 26-07-24 16:44:56, Li Zhijian wrote: >>>>>>> When a process is bound to a node that is being hot-removed, any memory >>>>>>> allocation attempts from that node should fail gracefully without >>>>>>> triggering the OOM-killer. However, the current behavior can cause the >>>>>>> oom-killer to be invoked, leading to the termination of processes on other >>>>>>> nodes, even when there is sufficient memory available in the system. >>>>>> But you said they are bound to the node that is offlined. >>>>>>> Prevent the oom-killer from being triggered by processes bound to a >>>>>>> node undergoing hot-remove operations. Instead, the allocation attempts >>>>>>> from the offlining node will simply fail, allowing the process to handle >>>>>>> the failure appropriately without causing disruption to the system. >>>>>> NAK. >>>>>> >>>>>> Also it is not really clear why process of offlining should behave any >>>>>> different from after the node is offlined. Could you describe an actual >>>>>> problem you are facing with much more details please? >>>>> We encountered that some processes(including some system critical services, for example sshd, rsyslogd, login) >>>>> were killed during our memory hot-remove testing. Our test program are described previous mail[1] >>>>> >>>>> In short, we have 3 memory nodes, node0 and node1 are DRAM, while node2 is CXL volatile memory that is onlined >>>>> to ZONE_MOVABLE. When we attempted to remove the node2, oom-killed was invoked to kill other processes >>>>> (sshd, rsyslogd, login) even though there is enough memory on node0+node1. >>> What are sizes of those nodes, how much memory does the testing program >>> consumes and do you have oom report without the patch applied? >>> >> node0: 4G, node1: 4G, node2: 2G >> >> my testing program will consume 64M memory. It's running on an *IDEL* >> system. >> >> [root@localhost guest]# numactl -H >> >> available: 3 nodes (0-2) >> node 0 cpus: 0 1 >> node 0 size: 3927 MB >> node 0 free: 3449 MB >> node 1 cpus: 2 3 >> node 1 size: 4028 MB >> node 1 free: 3614 MB >> node 2 cpus: >> node 2 size: 2048 MB >> node 2 free: 2048 MB >> node distances: >> node 0 1 2 >> 0: 10 20 20 >> 1: 20 10 20 >> 2: 20 20 10 >> >> >> An oom report is as following: >> [13853.707626] consume_std_pag invoked oom-killer: >> gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, >> oom_score_adj=0 >> [13853.708400] CPU: 1 PID: 274746 Comm: consume_std_pag Not tainted >> 6.10.0-rc2+ #160 >> [13853.708745] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS >> rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 >> [13853.709161] Call Trace: >> [13853.709161] <TASK> >> [13853.709161] dump_stack_lvl+0x64/0x80 >> [13853.709161] dump_header+0x44/0x1a0 >> [13853.709161] oom_kill_process+0xf8/0x200 >> [13853.709161] out_of_memory+0x110/0x590 >> [13853.709161] __alloc_pages_slowpath.constprop.92+0xb5f/0xd80 >> [13853.709161] __alloc_pages_noprof+0x354/0x380 >> [13853.709161] alloc_pages_mpol_noprof+0xe3/0x1f0 >> [13853.709161] vma_alloc_folio_noprof+0x5c/0xb0 >> [13853.709161] folio_prealloc+0x21/0x80 >> [13853.709161] do_pte_missing+0x695/0xa20 >> [13853.709161] ? __pte_offset_map+0x1b/0x180 >> [13853.709161] __handle_mm_fault+0x65f/0xc10 >> [13853.709161] ? sched_tick+0xd7/0x2b0 >> [13853.709161] handle_mm_fault+0x128/0x360 >> [13853.709161] do_user_addr_fault+0x309/0x810 >> [13853.709161] exc_page_fault+0x7e/0x180 >> [13853.709161] asm_exc_page_fault+0x26/0x30 >> [13853.709161] RIP: 0033:0x7f1d3ae2428a >> [13853.709161] Code: c5 fe 7f 07 c5 fe 7f 47 20 c5 fe 7f 47 40 c5 fe 7f >> 47 60 c5 f8 77 c3 66 0f 1f 84 00 00 00 00 00 40 0f b6 c6 48 89 d1 48 89 >> fa <f3> aa 48 89 d0 c5 f8 77 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 >> [13853.712991] RSP: 002b:00007ffe083a2388 EFLAGS: 00000206 >> [13853.712991] RAX: 0000000000000000 RBX: 00007ffe083a24d8 RCX: >> 0000000007eb8010 >> [13853.713915] Fallback order for Node 0: 0 1 >> [13853.713987] Fallback order for Node 1: 1 0 >> [13853.714006] Fallback order for Node 2: 0 1 >> [13853.714175] Built 3 zonelists, mobility grouping on. Total pages: 2002419 >> [13853.712991] RDX: 00007f1d32c00010 RSI: 0000000000000000 RDI: >> 00007f1d32d48000 >> [13853.712991] RBP: 00007ffe083a23b0 R08: 00000000ffffffff R09: >> 0000000000000000 >> [13853.714439] Policy zone: Normal >> [13853.712991] R10: 00007f1d3acd5200 R11: 00007f1d3ae241c0 R12: >> 0000000000000002 >> [13853.712991] R13: 0000000000000000 R14: 00007f1d3aeea000 R15: >> 0000000000403e00 >> [13853.712991] </TASK> >> [13853.716564] Mem-Info: >> [13853.716688] active_anon:17939 inactive_anon:0 isolated_anon:0 >> [13853.716688] active_file:132347 inactive_file:109560 isolated_file:0 >> [13853.716688] unevictable:0 dirty:2021 writeback:0 >> [13853.716688] slab_reclaimable:5876 slab_unreclaimable:18566 >> [13853.716688] mapped:35589 shmem:251 pagetables:1809 >> [13853.716688] sec_pagetables:0 bounce:0 >> [13853.716688] kernel_misc_reclaimable:0 >> [13853.716688] free:1694176 free_pcp:0 free_cma:0 >> [13853.718420] Node 2 hugepages_total=0 hugepages_free=0 >> hugepages_surp=0 hugepages_size=1048576kB >> [13853.718730] Node 2 hugepages_total=0 hugepages_free=0 >> hugepages_surp=0 hugepages_size=2048kB >> [13853.719127] 242158 total pagecache pages >> [13853.719310] 0 pages in swap cache >> [13853.719441] Free swap = 8142844kB >> [13853.719583] Total swap = 8143868kB >> [13853.719731] 2097019 pages RAM >> [13853.719890] 0 pages HighMem/MovableOnly >> [13853.720155] 60814 pages reserved >> [13853.720278] 0 pages cma reserved >> [13853.720393] 0 pages hwpoisoned >> [13853.720494] Tasks state (memory values in pages): >> [13853.720686] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem >> pgtables_bytes swapents oom_score_adj name >> [13853.721214] [ 718] 0 718 40965 29598 256 29342 0 368640 0 -250 >> systemd-journal >> <...snip...> >> [13853.747190] [ 274715] 0 274715 8520 1731 879 852 0 73728 0 0 >> (udev-worker) >> [13853.747561] [ 274743] 0 274743 617 384 0 384 0 45056 0 0 consume_activit >> [13853.748099] [ 274744] 0 274744 617 281 0 281 0 45056 0 0 consume_activit >> [13853.748479] [ 274745] 0 274745 2369 954 128 826 0 61440 0 0 daxctl >> [13853.748885] [ 274746] 0 274746 33386 667 320 347 0 49152 0 0 >> consume_std_pag >> <...snip...> >> [13853.755653] [ 274808] 0 274808 3534 251 32 219 0 61440 0 0 systemctl >> [13853.756151] >> oom-kill:constraint=CONSTRAINT_MEMORY_POLICY,nodemask=2,cpuset=/,mems_allowed=0-2,global_oom,task_memcg=/system.slice/rsyslog.service,task=rsyslogd,pid=274557,uid=0 >> [13853.756791] Out of memory: Killed process 274557 (rsyslogd) >> total-vm:957964kB, anon-rss:640kB, file-rss:46496kB, shmem-rss:0kB, >> UID:0 pgtables:1512kB oom_score_adj:0 > > OK, I guess I can see what is going on now. You are binding your memory > allocating task to the node you are offlining completely. This obviously > triggers OOM on that node. The oom killer is not really great at > handling memory policy OOMs because it is lacking per-node memory > consumption data for processes. Yes, that's the situation > That means that rather than killing the > test program which continues consuming memory - and not much of it - it > keeps killing other tasks with a higher memory consumption. This behavior is not my(administrator) expectation. > > This is really unfortunate but not something that should be handled by > special casing memory offlining but rather handling the mempolicy OOMs > better. There were some attempts in the past but never made it to a > mergable state. Maybe you want to pick up on that. Well, tell me the previous proposals(mail/url) please if you have the them in hand. I want to take a look. > >> [13853.758192] pagefault_out_of_memory: 4055 callbacks suppressed >> [13853.758243] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF > > This shouldn't really happen and it indicates that some memory > allocation in the pagefault path has failed. May I know if this will cause side effect to other processes. Thanks Zhijian > >> [13853.758865] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.759319] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.759564] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.759779] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.760128] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.760361] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.760588] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.760794] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.761187] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF >> [13853.774166] Demotion targets for Node 0: null >> [13853.774478] Demotion targets for Node 1: null >> >
