On Mon, Aug 05, 2024 at 02:40:48PM -0400, Pasha Tatashin wrote: > On Fri, Aug 2, 2024 at 3:02 PM Alison Schofield > <alison.schofield@xxxxxxxxx> wrote: > > > > ++ nvdimm, linux-cxl, Yu Zhang > > > > On Wed, Jun 05, 2024 at 10:27:51PM +0000, Sourav Panda wrote: > > > Today, we do not have any observability of per-page metadata > > > and how much it takes away from the machine capacity. Thus, > > > we want to describe the amount of memory that is going towards > > > per-page metadata, which can vary depending on build > > > configuration, machine architecture, and system use. > > > > > > This patch adds 2 fields to /proc/vmstat that can used as shown > > > below: > > > > > > Accounting per-page metadata allocated by boot-allocator: > > > /proc/vmstat:nr_memmap_boot * PAGE_SIZE > > > > > > Accounting per-page metadata allocated by buddy-allocator: > > > /proc/vmstat:nr_memmap * PAGE_SIZE > > > > > > Accounting total Perpage metadata allocated on the machine: > > > (/proc/vmstat:nr_memmap_boot + > > > /proc/vmstat:nr_memmap) * PAGE_SIZE > > > > > > Utility for userspace: > > > > > > Observability: Describe the amount of memory overhead that is > > > going to per-page metadata on the system at any given time since > > > this overhead is not currently observable. > > > > > > Debugging: Tracking the changes or absolute value in struct pages > > > can help detect anomalies as they can be correlated with other > > > metrics in the machine (e.g., memtotal, number of huge pages, > > > etc). > > > > > > page_ext overheads: Some kernel features such as page_owner > > > page_table_check that use page_ext can be optionally enabled via > > > kernel parameters. Having the total per-page metadata information > > > helps users precisely measure impact. Furthermore, page-metadata > > > metrics will reflect the amount of struct pages reliquished > > > (or overhead reduced) when hugetlbfs pages are reserved which > > > will vary depending on whether hugetlb vmemmap optimization is > > > enabled or not. > > > > > > For background and results see: > > > lore.kernel.org/all/20240220214558.3377482-1-souravpanda@xxxxxxxxxx > > > > > > Acked-by: David Rientjes <rientjes@xxxxxxxxxx> > > > Signed-off-by: Sourav Panda <souravpanda@xxxxxxxxxx> > > > Reviewed-by: Pasha Tatashin <pasha.tatashin@xxxxxxxxxx> > > > > This patch is leading to Oops in 6.11-rc1 when CONFIG_MEMORY_HOTPLUG > > is enabled. Folks hitting it have had success with reverting this patch. > > Disabling CONFIG_MEMORY_HOTPLUG is not a long term solution. > > > > Reported here: > > https://lore.kernel.org/linux-cxl/CAHj4cs9Ax1=CoJkgBGP_+sNu6-6=6v=_L-ZBZY0bVLD3wUWZQg@xxxxxxxxxxxxxx/ > > Thank you for the heads up. Can you please attach a full config file, > also was anyone able to reproduce this problem in qemu with emulated > nvdimm? > > Pasha Hi Pasha, This hits every time when boot with a CXL enabled kernel and the cxl-test module loaded. After boot, modprobe -r cxl-test emits the TRACE appended below. Seems to be the same failing signature as in ndctl case above. Applying the diff below works for the cxl-test unload failure. It moves the state update to before freeing the page. I saw a note in the patch review history about this: "v8: Declined changing placement of metrics after attempting" Hope it's as simple as this :) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 829112b0a914..39c9050f8780 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -188,8 +188,8 @@ static inline void free_vmemmap_page(struct page *page) free_bootmem_page(page); mod_node_page_state(page_pgdat(page), NR_MEMMAP_BOOT, -1); } else { - __free_page(page); mod_node_page_state(page_pgdat(page), NR_MEMMAP, -1); + __free_page(page); } } Failure trace [ 94.158105] BUG: unable to handle page fault for address: 0000000000004200 [ 94.159953] #PF: supervisor read access in kernel mode [ 94.161132] #PF: error_code(0x0000) - not-present page [ 94.162300] PGD 0 P4D 0 [ 94.162915] Oops: Oops: 0000 [#1] PREEMPT SMP PTI [ 94.164006] CPU: 0 UID: 0 PID: 1076 Comm: modprobe Tainted: G O N 6.11.0-rc1 #197 [ 94.165966] Tainted: [O]=OOT_MODULE, [N]=TEST [ 94.166973] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 [ 94.168768] RIP: 0010:mod_node_page_state+0x6/0x90 [ 94.169877] Code: 82 e9 ec fd ff ff 31 c9 e9 de fd ff ff 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 41 89 f2 89 d0 <4c> 8b 8f 00 42 00 00 83 ee 05 c1 f8 0c 83 fe 02 4f 8d 44 11 01 0f [ 94.172849] RSP: 0018:ffffc90002a4b760 EFLAGS: 00010287 [ 94.173645] RAX: 00000000fffffe00 RBX: ffffea03c0800000 RCX: 0000000000000000 [ 94.174762] RDX: fffffffffffffe00 RSI: 000000000000002d RDI: 0000000000000000 [ 94.175862] RBP: ffffc90002a4b7b0 R08: 0000000000000000 R09: 0000000000000000 [ 94.176973] R10: 000000000000002d R11: 0000000000080000 R12: ffff888012688040 [ 94.178078] R13: 0000000000200000 R14: ffffea03c0a00000 R15: ffff88812df6ce40 [ 94.179200] FS: 00007f9d9ea91740(0000) GS:ffff888077200000(0000) knlGS:0000000000000000 [ 94.180257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 94.180888] CR2: 0000000000004200 CR3: 0000000129886000 CR4: 00000000000006f0 [ 94.181687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 94.182487] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 94.183278] Call Trace: [ 94.183572] <TASK> [ 94.183825] ? show_regs+0x5f/0x70 [ 94.184217] ? __die+0x1f/0x70 [ 94.184556] ? page_fault_oops+0x14b/0x450 [ 94.185017] ? mod_node_page_state+0x6/0x90 [ 94.185479] ? search_exception_tables+0x5b/0x60 [ 94.185997] ? fixup_exception+0x22/0x300 [ 94.186449] ? kernelmode_fixup_or_oops.constprop.0+0x5a/0x70 [ 94.187088] ? __bad_area_nosemaphore+0x166/0x230 [ 94.187601] ? up_read+0x1d/0x30 [ 94.187973] ? bad_area_nosemaphore+0x11/0x20 [ 94.188452] ? do_user_addr_fault+0x2cb/0x6b0 [ 94.188940] ? __pfx_do_flush_tlb_all+0x10/0x10 [ 94.189445] ? exc_page_fault+0x6e/0x220 [ 94.189857] ? asm_exc_page_fault+0x27/0x30 [ 94.190212] ? mod_node_page_state+0x6/0x90 [ 94.190566] ? section_deactivate+0x242/0x290 [ 94.190946] sparse_remove_section+0x4d/0x70 [ 94.191312] __remove_pages+0x59/0x90 [ 94.191623] arch_remove_memory+0x1a/0x50 [ 94.192224] try_remove_memory+0xe9/0x150 [ 94.192858] remove_memory+0x1d/0x30 [ 94.193457] dev_dax_kmem_remove+0x9d/0x140 [kmem] [ 94.194140] dax_bus_remove+0x1d/0x30 [ 94.194723] device_remove+0x3e/0x70 [ 94.195283] device_release_driver_internal+0x1ae/0x220 [ 94.195980] device_release_driver+0xd/0x20 [ 94.196590] bus_remove_device+0xd7/0x140 [ 94.197189] device_del+0x15b/0x3a0 [ 94.197746] unregister_dev_dax+0x6c/0xd0 [ 94.198335] devm_action_release+0x10/0x20 [ 94.198945] devres_release_all+0xa8/0xe0 [ 94.199536] device_unbind_cleanup+0xd/0x70 [ 94.200138] device_release_driver_internal+0x1d3/0x220 [ 94.200809] device_release_driver+0xd/0x20 [ 94.201396] bus_remove_device+0xd7/0x140 [ 94.201977] device_del+0x15b/0x3a0 [ 94.202484] device_unregister+0x12/0x60 [ 94.203044] cxlr_dax_unregister+0x9/0x10 [cxl_core] [ 94.203712] devm_action_release+0x10/0x20 [ 94.204249] devres_release_all+0xa8/0xe0 [ 94.204780] device_unbind_cleanup+0xd/0x70 [ 94.205327] device_release_driver_internal+0x1d3/0x220 [ 94.205960] device_release_driver+0xd/0x20 [ 94.206485] bus_remove_device+0xd7/0x140 [ 94.207011] device_del+0x15b/0x3a0 [ 94.207486] unregister_region+0x2b/0x80 [cxl_core] [ 94.208092] devm_action_release+0x10/0x20 [ 94.208624] devres_release_all+0xa8/0xe0 [ 94.209130] device_unbind_cleanup+0xd/0x70 [ 94.209666] device_release_driver_internal+0x1d3/0x220 [ 94.210274] device_release_driver+0xd/0x20 [ 94.210812] bus_remove_device+0xd7/0x140 [ 94.211326] device_del+0x15b/0x3a0 [ 94.211807] ? __this_cpu_preempt_check+0x13/0x20 [ 94.212384] ? lock_release+0x133/0x290 [ 94.212896] ? __x64_sys_delete_module+0x171/0x260 [ 94.213474] platform_device_del.part.0+0x13/0x80 [ 94.214046] platform_device_unregister+0x1b/0x40 [ 94.214602] cxl_test_exit+0x1a/0xcb0 [cxl_test] [ 94.215164] __x64_sys_delete_module+0x182/0x260 [ 94.215720] ? __fput+0x1b5/0x2e0 [ 94.216176] ? debug_smp_processor_id+0x17/0x20 [ 94.216735] x64_sys_call+0xcc/0x1f30 [ 94.217206] do_syscall_64+0x47/0x110 [ 94.217691] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 94.218265] RIP: 0033:0x7f9d9e3128cb [ 94.218745] Code: 73 01 c3 48 8b 0d 55 55 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 55 0e 00 f7 d8 64 89 01 48 [ 94.220590] RSP: 002b:00007ffd8ca73a88 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 94.221403] RAX: ffffffffffffffda RBX: 0000564e9737f9f0 RCX: 00007f9d9e3128cb [ 94.222176] RDX: 0000000000000000 RSI: 0000000000000800 RDI: 0000564e9737fa58 [ 94.222960] RBP: 0000564e9737fa58 R08: 1999999999999999 R09: 0000000000000000 [ 94.223772] R10: 00007f9d9e39dac0 R11: 0000000000000206 R12: 0000000000000001 [ 94.224538] R13: 0000000000000000 R14: 00007ffd8ca75dc8 R15: 0000564e9737f480 [ 94.225314] </TASK> [ 94.225721] Modules linked in: kmem device_dax dax_cxl cxl_pci cxl_mock_mem(ON) cxl_test(ON-) cxl_mem(ON) cxl_pmem(ON) cxl_port(ON) cxl_acpi(ON) cxl_mock(ON) cxl_core(ON) libnvdimm [ 94.227422] CR2: 0000000000004200 [ 94.227977] ---[ end trace 0000000000000000 ]--- [ 94.228585] RIP: 0010:mod_node_page_state+0x6/0x90 [ 94.229523] Code: 82 e9 ec fd ff ff 31 c9 e9 de fd ff ff 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 41 89 f2 89 d0 <4c> 8b 8f 00 42 00 00 83 ee 05 c1 f8 0c 83 fe 02 4f 8d 44 11 01 0f [ 94.232041] RSP: 0018:ffffc90002a4b760 EFLAGS: 00010287 [ 94.232891] RAX: 00000000fffffe00 RBX: ffffea03c0800000 RCX: 0000000000000000 [ 94.233921] RDX: fffffffffffffe00 RSI: 000000000000002d RDI: 0000000000000000 [ 94.234898] RBP: ffffc90002a4b7b0 R08: 0000000000000000 R09: 0000000000000000 [ 94.235908] R10: 000000000000002d R11: 0000000000080000 R12: ffff888012688040 [ 94.237077] R13: 0000000000200000 R14: ffffea03c0a00000 R15: ffff88812df6ce40 [ 94.237922] FS: 00007f9d9ea91740(0000) GS:ffff888077200000(0000) knlGS:0000000000000000 [ 94.238844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 94.239584] CR2: 0000000000004200 CR3: 0000000129886000 CR4: 00000000000006f0 [ 94.240566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 94.241787] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
