Hi Catalin, First, thank you for maintaining kmemleak, a very useful tool! I just added linux-mm ML in Cc, I hope that's OK, I didn't know which list to add. Recently, our CI validating our MPTCP tree reported a UaF linked to kmemleak: > ================================================================== > BUG: KASAN: use-after-free in __lock_acquire (kernel/locking/lockdep.c:4925) > Read of size 8 at addr ffff8880288e8738 by task kmemleak/67 > CPU: 0 PID: 67 Comm: kmemleak Tainted: G N 6.2.0-rc2-g1323854aa099 #1 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 > Call Trace: > <TASK> > dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4)) > print_address_description.constprop.0 (mm/kasan/report.c:307) > print_report (mm/kasan/report.c:418) > ? kasan_addr_to_slab (arch/x86/include/asm/bitops.h:207) > ? __lock_acquire (kernel/locking/lockdep.c:4925) > kasan_report (mm/kasan/report.c:184) > ? __lock_acquire (kernel/locking/lockdep.c:4925) > __lock_acquire (kernel/locking/lockdep.c:4925) > ? mark_lock.part.0 (arch/x86/include/asm/bitops.h:228) > ? debug_object_active_state (lib/debugobjects.c:950) > lock_acquire (kernel/locking/lockdep.c:466) > ? kmemleak_scan (mm/kmemleak.c:1527) > ? mark_held_locks (kernel/locking/lockdep.c:4236) > ? rcu_read_unlock (include/linux/rcupdate.h:793 (discriminator 5)) > ? lockdep_hardirqs_on_prepare.part.0 (kernel/locking/lockdep.c:466) > ? __call_rcu_common.constprop.0 (arch/x86/include/asm/irqflags.h:29) > ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4385) > ? _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:117) > _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:120) > ? kmemleak_scan (mm/kmemleak.c:1527) > kmemleak_scan (mm/kmemleak.c:1527) > ? kmemleak_cond_resched (mm/kmemleak.c:1499) > ? do_ipt_get_ctl (net/ipv4/netfilter/ip_tables.c:850) > ? kmemleak_scan.cold (mm/kmemleak.c:1703) > ? kmemleak_scan.cold (mm/kmemleak.c:1703) > kmemleak_scan_thread (mm/kmemleak.c:1724 (discriminator 2)) > kthread (kernel/kthread.c:376) > ? kthread_complete_and_exit (kernel/kthread.c:331) > ret_from_fork (arch/x86/entry/entry_64.S:314) > </TASK> > Allocated by task 15209: > kasan_save_stack (mm/kasan/common.c:46) > kasan_set_track (mm/kasan/common.c:52) > __kasan_slab_alloc (mm/kasan/common.c:328) > kmem_cache_alloc (include/linux/kasan.h:201) > __create_object (mm/kmemleak.c:451) > kmem_cache_alloc_lru (mm/slub.c:3454) > v9fs_alloc_inode (include/linux/fs.h:3116) > alloc_inode (fs/inode.c:259) > iget5_locked (fs/inode.c:1241) > v9fs_inode_from_fid_dotl (fs/9p/vfs_inode_dotl.c:115) > v9fs_vfs_lookup.part.0 (fs/9p/v9fs.h:227) > __lookup_slow (include/linux/dcache.h:359) > walk_component (include/linux/fs.h:771) > link_path_walk.part.0.constprop.0 (fs/namei.c:2320) > path_openat (fs/namei.c:2245 (discriminator 2)) > do_filp_open (fs/namei.c:3741) > do_sys_openat2 (fs/open.c:1310) > __x64_sys_openat (fs/open.c:1337) > do_syscall_64 (arch/x86/entry/common.c:50) > entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) > Freed by task 15: > kasan_save_stack (mm/kasan/common.c:46) > kasan_set_track (mm/kasan/common.c:52) > kasan_save_free_info (mm/kasan/generic.c:520) > ____kasan_slab_free (mm/kasan/common.c:238) > slab_free_freelist_hook (mm/slub.c:1807) > kmem_cache_free (mm/slub.c:3787) > rcu_do_batch (include/linux/rcupdate.h:330) > rcu_core (kernel/rcu/tree.c:2508) > __do_softirq (arch/x86/include/asm/jump_label.h:27) > Last potentially related work creation: > kasan_save_stack (mm/kasan/common.c:46) > __kasan_record_aux_stack (mm/kasan/generic.c:488) > __call_rcu_common.constprop.0 (arch/x86/include/asm/irqflags.h:29) > slab_free_freelist_hook (include/linux/kmemleak.h:48) > kmem_cache_free (mm/slub.c:3787) > rcu_do_batch (include/linux/rcupdate.h:330) > rcu_core (kernel/rcu/tree.c:2508) > __do_softirq (arch/x86/include/asm/jump_label.h:27) > Second to last potentially related work creation: > kasan_save_stack (mm/kasan/common.c:46) > __kasan_record_aux_stack (mm/kasan/generic.c:488) > __call_rcu_common.constprop.0 (arch/x86/include/asm/irqflags.h:29) > slab_free_freelist_hook (include/linux/kmemleak.h:48) > kmem_cache_free (mm/slub.c:3787) > mas_destroy (lib/maple_tree.c:5770) > mas_store_prealloc (lib/maple_tree.c:5701) > __vma_adjust (mm/mmap.c:783) > shift_arg_pages (include/linux/mm.h:2793) > setup_arg_pages (fs/exec.c:832) > load_elf_binary (fs/binfmt_elf.c:1015 (discriminator 8)) > search_binary_handler (fs/exec.c:1737) > exec_binprm (fs/exec.c:1778) > bprm_execve (fs/exec.c:1851) > do_execveat_common.isra.0 (fs/exec.c:1956) > __x64_sys_execve (fs/exec.c:2101) > do_syscall_64 (arch/x86/entry/common.c:50) > entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) > The buggy address belongs to the object at ffff8880288e8720 > which belongs to the cache kmemleak_object of size 240 > The buggy address is located 24 bytes inside of > 240-byte region [ffff8880288e8720, ffff8880288e8810) > The buggy address belongs to the physical page: > page:0000000033bd1263 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff8880288e8720 pfn:0x288e8 > head:0000000033bd1263 order:1 compound_mapcount:0 subpages_mapcount:0 compound_pincount:0 > flags: 0x100000000010200(slab|head|node=0|zone=1) > raw: 0100000000010200 ffff88800104d400 ffffea0000597910 ffffea00005e3090 > raw: ffff8880288e8720 00000000001a0014 00000001ffffffff 0000000000000000 > page dumped because: kasan: bad access detected > Memory state around the buggy address: > ffff8880288e8600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > ffff8880288e8680: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc >>ffff8880288e8700: fc fc fc fc fb fb fb fb fb fb fb fb fb fb fb fb > ^ > ffff8880288e8780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ffff8880288e8800: fb fb fc fc fc fc fc fc fc fc fa fb fb fb fb fb > ================================================================== Followed by a few soft lockup's, see the attached file. The full logs is available there: https://cirrus-ci.com/task/4706769230888960/ We had this issue in our tree when validating: 1323854aa099 ("DO-NOT-MERGE: mptcp: enabled by default") Which was on top of both: - net-next: a6f536063b69 ("qed: fix a typo in comment") - net: 0aa7d35f5d00 ("Merge branch '100GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/net-queue") Which were on top of Linus tree: - 50011c32f421 ("Merge tag 'net-6.2-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net") Unfortunately, apart from the config file -- also attached to this email -- we don't have more to share: we are unable to reproduce it so far, sorry for that. We wanted to share that with you, just in case it could be useful. Hopefully this would be more helpful than creating noise! :) Cheers, Matt -- Tessares | Belgium | Hybrid Access Solutions www.tessares.net
[ 1998.065199][ T67] ================================================================== [ 1998.078868][ T67] BUG: KASAN: use-after-free in __lock_acquire (kernel/locking/lockdep.c:4925) [ 1998.089733][ T67] Read of size 8 at addr ffff8880288e8738 by task kmemleak/67 [ 1998.102152][ T67] [ 1998.105908][ T67] CPU: 0 PID: 67 Comm: kmemleak Tainted: G N 6.2.0-rc2-g1323854aa099 #1 [ 1998.120592][ T67] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 1998.135164][ T67] Call Trace: [ 1998.140312][ T67] <TASK> [ 1998.144750][ T67] dump_stack_lvl (lib/dump_stack.c:107 (discriminator 4)) [ 1998.151979][ T67] print_address_description.constprop.0 (mm/kasan/report.c:307) [ 1998.161217][ T67] print_report (mm/kasan/report.c:418) [ 1998.168878][ T67] ? kasan_addr_to_slab (arch/x86/include/asm/bitops.h:207) [ 1998.175993][ T67] ? __lock_acquire (kernel/locking/lockdep.c:4925) [ 1998.183059][ T67] kasan_report (mm/kasan/report.c:184) [ 1998.190917][ T67] ? __lock_acquire (kernel/locking/lockdep.c:4925) [ 1998.198553][ T67] __lock_acquire (kernel/locking/lockdep.c:4925) [ 1998.207657][ T67] ? mark_lock.part.0 (arch/x86/include/asm/bitops.h:228) [ 1998.215164][ T67] ? debug_object_active_state (lib/debugobjects.c:950) [ 1998.224086][ T67] lock_acquire (kernel/locking/lockdep.c:466) [ 1998.230806][ T67] ? kmemleak_scan (mm/kmemleak.c:1527) [ 1998.238423][ T67] ? mark_held_locks (kernel/locking/lockdep.c:4236) [ 1998.246811][ T67] ? rcu_read_unlock (include/linux/rcupdate.h:793 (discriminator 5)) [ 1998.257376][ T67] ? lockdep_hardirqs_on_prepare.part.0 (kernel/locking/lockdep.c:466) [ 1998.268283][ T67] ? __call_rcu_common.constprop.0 (arch/x86/include/asm/irqflags.h:29) [ 1998.288342][ T67] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4385) [ 1998.306421][ T67] ? _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:117) [ 1998.313565][ T67] _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:120) [ 1998.333162][ T67] ? kmemleak_scan (mm/kmemleak.c:1527) [ 1998.339925][ T67] kmemleak_scan (mm/kmemleak.c:1527) [ 1998.346720][ T67] ? kmemleak_cond_resched (mm/kmemleak.c:1499) [ 1998.353737][ T67] ? do_ipt_get_ctl (net/ipv4/netfilter/ip_tables.c:850) [ 1998.360371][ T67] ? kmemleak_scan.cold (mm/kmemleak.c:1703) [ 1998.367635][ T67] ? kmemleak_scan.cold (mm/kmemleak.c:1703) [ 1998.374625][ T67] kmemleak_scan_thread (mm/kmemleak.c:1724 (discriminator 2)) [ 1998.382251][ T67] kthread (kernel/kthread.c:376) [ 1998.387822][ T67] ? kthread_complete_and_exit (kernel/kthread.c:331) [ 1998.396022][ T67] ret_from_fork (arch/x86/entry/entry_64.S:314) [ 1998.402514][ T67] </TASK> [ 1998.407226][ T67] [ 1998.410802][ T67] Allocated by task 15209: [ 1998.417005][ T67] kasan_save_stack (mm/kasan/common.c:46) [ 1998.422687][ T67] kasan_set_track (mm/kasan/common.c:52) [ 1998.428588][ T67] __kasan_slab_alloc (mm/kasan/common.c:328) [ 1998.435135][ T67] kmem_cache_alloc (include/linux/kasan.h:201) [ 1998.440904][ T67] __create_object (mm/kmemleak.c:451) [ 1998.447205][ T67] kmem_cache_alloc_lru (mm/slub.c:3454) [ 1998.453644][ T67] v9fs_alloc_inode (include/linux/fs.h:3116) [ 1998.459836][ T67] alloc_inode (fs/inode.c:259) [ 1998.464957][ T67] iget5_locked (fs/inode.c:1241) [ 1998.470900][ T67] v9fs_inode_from_fid_dotl (fs/9p/vfs_inode_dotl.c:115) [ 1998.477649][ T67] v9fs_vfs_lookup.part.0 (fs/9p/v9fs.h:227) [ 1998.485232][ T67] __lookup_slow (include/linux/dcache.h:359) [ 1998.491097][ T67] walk_component (include/linux/fs.h:771) [ 1998.496886][ T67] link_path_walk.part.0.constprop.0 (fs/namei.c:2320) [ 1998.504840][ T67] path_openat (fs/namei.c:2245 (discriminator 2)) [ 1998.510395][ T67] do_filp_open (fs/namei.c:3741) [ 1998.516082][ T67] do_sys_openat2 (fs/open.c:1310) [ 1998.521791][ T67] __x64_sys_openat (fs/open.c:1337) [ 1998.528052][ T67] do_syscall_64 (arch/x86/entry/common.c:50) [ 1998.533580][ T67] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) [ 1998.541052][ T67] [ 1998.544065][ T67] Freed by task 15: [ 1998.548799][ T67] kasan_save_stack (mm/kasan/common.c:46) [ 1998.554948][ T67] kasan_set_track (mm/kasan/common.c:52) [ 1998.560529][ T67] kasan_save_free_info (mm/kasan/generic.c:520) [ 1998.566909][ T67] ____kasan_slab_free (mm/kasan/common.c:238) [ 1998.573244][ T67] slab_free_freelist_hook (mm/slub.c:1807) [ 1998.580056][ T67] kmem_cache_free (mm/slub.c:3787) [ 1998.586253][ T67] rcu_do_batch (include/linux/rcupdate.h:330) [ 1998.592100][ T67] rcu_core (kernel/rcu/tree.c:2508) [ 1998.597240][ T67] __do_softirq (arch/x86/include/asm/jump_label.h:27) [ 1998.602851][ T67] [ 1998.605790][ T67] Last potentially related work creation: [ 1998.612917][ T67] kasan_save_stack (mm/kasan/common.c:46) [ 1998.618073][ T67] __kasan_record_aux_stack (mm/kasan/generic.c:488) [ 1998.624387][ T67] __call_rcu_common.constprop.0 (arch/x86/include/asm/irqflags.h:29) [ 1998.630934][ T67] slab_free_freelist_hook (include/linux/kmemleak.h:48) [ 1998.637023][ T67] kmem_cache_free (mm/slub.c:3787) [ 1998.642266][ T67] rcu_do_batch (include/linux/rcupdate.h:330) [ 1998.647362][ T67] rcu_core (kernel/rcu/tree.c:2508) [ 1998.652218][ T67] __do_softirq (arch/x86/include/asm/jump_label.h:27) [ 1998.656897][ T67] [ 1998.659965][ T67] Second to last potentially related work creation: [ 1998.667036][ T67] kasan_save_stack (mm/kasan/common.c:46) [ 1998.672408][ T67] __kasan_record_aux_stack (mm/kasan/generic.c:488) [ 1998.678461][ T67] __call_rcu_common.constprop.0 (arch/x86/include/asm/irqflags.h:29) [ 1998.684810][ T67] slab_free_freelist_hook (include/linux/kmemleak.h:48) [ 1998.690548][ T67] kmem_cache_free (mm/slub.c:3787) [ 1998.695402][ T67] mas_destroy (lib/maple_tree.c:5770) [ 1998.700269][ T67] mas_store_prealloc (lib/maple_tree.c:5701) [ 1998.706705][ T67] __vma_adjust (mm/mmap.c:783) [ 1998.711800][ T67] shift_arg_pages (include/linux/mm.h:2793) [ 1998.717107][ T67] setup_arg_pages (fs/exec.c:832) [ 1998.722226][ T67] load_elf_binary (fs/binfmt_elf.c:1015 (discriminator 8)) [ 1998.727314][ T67] search_binary_handler (fs/exec.c:1737) [ 1998.732522][ T67] exec_binprm (fs/exec.c:1778) [ 1998.736567][ T67] bprm_execve (fs/exec.c:1851) [ 1998.741218][ T67] do_execveat_common.isra.0 (fs/exec.c:1956) [ 1998.747160][ T67] __x64_sys_execve (fs/exec.c:2101) [ 1998.752277][ T67] do_syscall_64 (arch/x86/entry/common.c:50) [ 1998.757132][ T67] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) [ 1998.763921][ T67] [ 1998.766743][ T67] The buggy address belongs to the object at ffff8880288e8720 [ 1998.766743][ T67] which belongs to the cache kmemleak_object of size 240 [ 1998.782545][ T67] The buggy address is located 24 bytes inside of [ 1998.782545][ T67] 240-byte region [ffff8880288e8720, ffff8880288e8810) [ 1998.797022][ T67] [ 1998.799994][ T67] The buggy address belongs to the physical page: [ 1998.807529][ T67] page:0000000033bd1263 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff8880288e8720 pfn:0x288e8 [ 1998.820214][ T67] head:0000000033bd1263 order:1 compound_mapcount:0 subpages_mapcount:0 compound_pincount:0 [ 1998.829500][ T67] flags: 0x100000000010200(slab|head|node=0|zone=1) [ 1998.836373][ T67] raw: 0100000000010200 ffff88800104d400 ffffea0000597910 ffffea00005e3090 [ 1998.845129][ T67] raw: ffff8880288e8720 00000000001a0014 00000001ffffffff 0000000000000000 [ 1998.854246][ T67] page dumped because: kasan: bad access detected [ 1998.861224][ T67] [ 1998.863664][ T67] Memory state around the buggy address: [ 1998.869527][ T67] ffff8880288e8600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1998.878775][ T67] ffff8880288e8680: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc [ 1998.886851][ T67] >ffff8880288e8700: fc fc fc fc fb fb fb fb fb fb fb fb fb fb fb fb [ 1998.894812][ T67] ^ [ 1998.901224][ T67] ffff8880288e8780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 1998.909474][ T67] ffff8880288e8800: fb fb fc fc fc fc fc fc fc fc fa fb fb fb fb fb [ 1998.918350][ T67] ================================================================== [ 1998.927281][ T67] Disabling lock debugging due to kernel taint [ 2022.247667][ C2] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:3:12417] [ 2022.256051][ C2] Modules linked in: xt_mark nft_compat nft_tproxy nf_tproxy_ipv6 nf_tproxy_ipv4 nft_socket nf_socket_ipv4 nf_socket_ipv6 nf_tables sch_netem mptcp_diag inet_diag mptcp_token_test mptcp_crypto_test kunit [ 2022.275721][ C2] irq event stamp: 199641 [ 2022.279553][ C2] hardirqs last enabled at (199641): asm_sysvec_apic_timer_interrupt (arch/x86/include/asm/idtentry.h:649) [ 2022.289138][ C2] hardirqs last disabled at (199640): sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1107) [ 2022.298925][ C2] softirqs last enabled at (172456): mptcp_worker (include/linux/instrumented.h:102) [ 2022.307436][ C2] softirqs last disabled at (172454): release_sock (net/core/sock.c:3484) [ 2022.315269][ C2] CPU: 2 PID: 12417 Comm: kworker/2:3 Tainted: G B N 6.2.0-rc2-g1323854aa099 #1 [ 2022.323845][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 2022.332028][ C2] Workqueue: rcu_gp wait_rcu_exp_gp [ 2022.336475][ C2] RIP: 0010:smp_call_function_single (kernel/smp.c:442) [ 2022.341916][ C2] Code: 46 08 a8 01 74 38 48 b8 00 00 00 00 00 fc ff df 4c 89 ea 4d 89 ef 48 c1 ea 03 41 83 e7 07 48 01 c2 41 83 c7 03 f3 90 0f b6 02 <41> 38 c7 7c 08 84 c0 0f 85 9b 00 00 00 8b 46 08 a8 01 75 e7 48 b8 All code ======== 0: 46 08 a8 01 74 38 48 rex.RX or %r13b,0x48387401(%rax) 7: b8 00 00 00 00 mov $0x0,%eax c: 00 fc add %bh,%ah e: ff (bad) f: df 4c 89 ea fisttps -0x16(%rcx,%rcx,4) 13: 4d 89 ef mov %r13,%r15 16: 48 c1 ea 03 shr $0x3,%rdx 1a: 41 83 e7 07 and $0x7,%r15d 1e: 48 01 c2 add %rax,%rdx 21: 41 83 c7 03 add $0x3,%r15d 25: f3 90 pause 27: 0f b6 02 movzbl (%rdx),%eax 2a:* 41 38 c7 cmp %al,%r15b <-- trapping instruction 2d: 7c 08 jl 0x37 2f: 84 c0 test %al,%al 31: 0f 85 9b 00 00 00 jne 0xd2 37: 8b 46 08 mov 0x8(%rsi),%eax 3a: a8 01 test $0x1,%al 3c: 75 e7 jne 0x25 3e: 48 rex.W 3f: b8 .byte 0xb8 Code starting with the faulting instruction =========================================== 0: 41 38 c7 cmp %al,%r15b 3: 7c 08 jl 0xd 5: 84 c0 test %al,%al 7: 0f 85 9b 00 00 00 jne 0xa8 d: 8b 46 08 mov 0x8(%rsi),%eax 10: a8 01 test $0x1,%al 12: 75 e7 jne 0xfffffffffffffffb 14: 48 rex.W 15: b8 .byte 0xb8 [ 2022.359265][ C2] RSP: 0018:ffffc9000090fb40 EFLAGS: 00000202 [ 2022.364510][ C2] RAX: 0000000000000000 RBX: 1ffff92000121f6c RCX: 1ffffffff5ce2f1a [ 2022.371760][ C2] RDX: ffffed100d9e7879 RSI: ffff88806cf3c3c0 RDI: ffffffffae7178d0 [ 2022.379221][ C2] RBP: ffffc9000090fc10 R08: 0000000000000000 R09: ffffffffaf837f97 [ 2022.386358][ C2] R10: fffffbfff5f06ff2 R11: 0000000000000001 R12: 0000000000000001 [ 2022.393867][ C2] R13: ffff88806cf3c3c8 R14: 0000000000000000 R15: 0000000000000003 [ 2022.401211][ C2] FS: 0000000000000000(0000) GS:ffff88806cf00000(0000) knlGS:0000000000000000 [ 2022.408728][ C2] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2022.415494][ C2] CR2: 00007fc2565a8c14 CR3: 000000000a814001 CR4: 0000000000170ee0 [ 2022.422855][ C2] Call Trace: [ 2022.425844][ C2] <TASK> [ 2022.429061][ C2] ? rcu_barrier (kernel/rcu/tree_exp.h:735) [ 2022.432970][ C2] ? generic_exec_single (kernel/smp.c:729) [ 2022.437235][ C2] ? lockdep_hardirqs_on_prepare.part.0 (kernel/locking/lockdep.c:466) [ 2022.442838][ C2] __sync_rcu_exp_select_node_cpus (kernel/rcu/tree_exp.h:394) [ 2022.449011][ C2] sync_rcu_exp_select_cpus (kernel/rcu/tree_exp.h:549) [ 2022.453748][ C2] wait_rcu_exp_gp (kernel/rcu/tree_exp.h:513) [ 2022.458041][ C2] process_one_work (kernel/workqueue.c:2294) [ 2022.462536][ C2] ? rcu_read_unlock (include/linux/rcupdate.h:793 (discriminator 5)) [ 2022.467112][ C2] ? pwq_dec_nr_in_flight (kernel/workqueue.c:2184) [ 2022.471553][ C2] ? do_ipt_get_ctl (net/ipv4/netfilter/ip_tables.c:850) [ 2022.475533][ C2] ? rwlock_bug.part.0 (kernel/locking/spinlock_debug.c:113) [ 2022.479862][ C2] ? _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:117) [ 2022.484181][ C2] worker_thread (include/linux/list.h:292) [ 2022.488085][ C2] ? process_one_work (kernel/workqueue.c:2379) [ 2022.492728][ C2] kthread (kernel/kthread.c:376) [ 2022.496232][ C2] ? kthread_complete_and_exit (kernel/kthread.c:331) [ 2022.500995][ C2] ret_from_fork (arch/x86/entry/entry_64.S:314) [ 2022.504975][ C2] </TASK> [ 2024.080661][ C3] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 2024.087741][ C3] rcu: 0-...!: (1 GPs behind) idle=484c/1/0x4000000000000000 softirq=833496/833497 fqs=24 [ 2024.097869][ C3] (detected by 3, t=26017 jiffies, g=1514213, q=53340 ncpus=4) [ 2024.105736][ C3] Sending NMI from CPU 3 to CPUs 0: [ 2024.111097][ C0] NMI backtrace for cpu 0 [ 2024.111108][ C0] CPU: 0 PID: 67 Comm: kmemleak Tainted: G B L N 6.2.0-rc2-g1323854aa099 #1 [ 2024.111114][ C0] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 2024.111118][ C0] RIP: 0010:report_enabled (mm/kasan/report.c:96) [ 2024.111130][ C0] Code: 10 48 89 df e8 08 1c 00 00 e9 39 ff ff ff 0f 1f 00 48 8b 05 49 62 e9 05 48 d1 e8 83 e0 01 75 0d f0 48 0f ba 2d 37 62 e9 05 00 <0f> 93 c0 c3 0f 1f 00 66 90 65 8b 05 63 25 f9 54 3b 05 31 fc 78 04 All code ======== 0: 10 48 89 adc %cl,-0x77(%rax) 3: df e8 fucomip %st(0),%st 5: 08 1c 00 or %bl,(%rax,%rax,1) 8: 00 e9 add %ch,%cl a: 39 ff cmp %edi,%edi c: ff (bad) d: ff 0f decl (%rdi) f: 1f (bad) 10: 00 48 8b add %cl,-0x75(%rax) 13: 05 49 62 e9 05 add $0x5e96249,%eax 18: 48 d1 e8 shr %rax 1b: 83 e0 01 and $0x1,%eax 1e: 75 0d jne 0x2d 20: f0 48 0f ba 2d 37 62 lock btsq $0x0,0x5e96237(%rip) # 0x5e96261 27: e9 05 00 2a:* 0f 93 c0 setae %al <-- trapping instruction 2d: c3 ret 2e: 0f 1f 00 nopl (%rax) 31: 66 90 xchg %ax,%ax 33: 65 8b 05 63 25 f9 54 mov %gs:0x54f92563(%rip),%eax # 0x54f9259d 3a: 3b 05 31 fc 78 04 cmp 0x478fc31(%rip),%eax # 0x478fc71 Code starting with the faulting instruction =========================================== 0: 0f 93 c0 setae %al 3: c3 ret 4: 0f 1f 00 nopl (%rax) 7: 66 90 xchg %ax,%ax 9: 65 8b 05 63 25 f9 54 mov %gs:0x54f92563(%rip),%eax # 0x54f92573 10: 3b 05 31 fc 78 04 cmp 0x478fc31(%rip),%eax # 0x478fc47 [ 2024.111134][ C0] RSP: 0018:ffffc9000048fc58 EFLAGS: 00000047 [ 2024.111141][ C0] RAX: 0000000000000000 RBX: ffff8880288e8720 RCX: ffffffffad9624fd [ 2024.111144][ C0] RDX: ffff888006453600 RSI: 0000000000000004 RDI: ffff8880288e8720 [ 2024.111148][ C0] RBP: 000000002c688000 R08: 0000000000000000 R09: ffff8880288e8723 [ 2024.111151][ C0] R10: ffffed100511d0e4 R11: 0000000000000001 R12: 0000000000000003 [ 2024.111154][ C0] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000030000 [ 2024.111175][ C0] FS: 0000000000000000(0000) GS:ffff88806ce00000(0000) knlGS:0000000000000000 [ 2024.111182][ C0] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.111185][ C0] CR2: 0000556edcd34b08 CR3: 000000002e92c002 CR4: 0000000000170ef0 [ 2024.111189][ C0] Call Trace: [ 2024.111194][ C0] <TASK> [ 2024.111197][ C0] kasan_report (mm/kasan/report.c:501) [ 2024.111208][ C0] ? __lock_acquire (kernel/locking/lockdep.c:5055) [ 2024.111219][ C0] queued_spin_lock_slowpath (arch/x86/include/asm/atomic.h:29) [ 2024.111228][ C0] ? kasan_report (mm/kasan/report.c:501) [ 2024.111233][ C0] ? _raw_write_unlock_irqrestore (kernel/locking/qspinlock.c:317) [ 2024.111239][ C0] ? kasan_report (mm/kasan/report.c:501) [ 2024.111242][ C0] ? lock_acquire (kernel/locking/lockdep.c:466) [ 2024.111249][ C0] ? mark_held_locks (kernel/locking/lockdep.c:4236) [ 2024.111258][ C0] do_raw_spin_lock (include/asm-generic/qspinlock.h:114) [ 2024.111265][ C0] ? rwlock_bug.part.0 (kernel/locking/spinlock_debug.c:113) [ 2024.111270][ C0] ? _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:117) [ 2024.111281][ C0] kmemleak_scan (mm/kmemleak.c:1527) [ 2024.111291][ C0] ? kmemleak_cond_resched (mm/kmemleak.c:1499) [ 2024.111296][ C0] ? do_ipt_get_ctl (net/ipv4/netfilter/ip_tables.c:850) [ 2024.111305][ C0] ? kmemleak_scan.cold (mm/kmemleak.c:1703) [ 2024.111313][ C0] ? kmemleak_scan.cold (mm/kmemleak.c:1703) [ 2024.111321][ C0] kmemleak_scan_thread (mm/kmemleak.c:1724 (discriminator 2)) [ 2024.111327][ C0] kthread (kernel/kthread.c:376) [ 2024.111333][ C0] ? kthread_complete_and_exit (kernel/kthread.c:331) [ 2024.111343][ C0] ret_from_fork (arch/x86/entry/entry_64.S:314) [ 2024.111380][ C0] </TASK> [ 2024.112022][ C3] rcu: rcu_preempt kthread timer wakeup didn't happen for 25701 jiffies! g1514213 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 [ 2024.345723][ C3] rcu: Possible timer handling issue on cpu=2 timer-softirq=170509 [ 2024.353593][ C3] rcu: rcu_preempt kthread starved for 25942 jiffies! g1514213 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2 [ 2024.365018][ C3] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 2024.375010][ C3] rcu: RCU grace-period kthread stack dump: [ 2024.380875][ C3] task:rcu_preempt state:I stack:29512 pid:16 ppid:2 flags:0x00004000 [ 2024.390069][ C3] Call Trace: [ 2024.393454][ C3] <TASK> [ 2024.396462][ C3] __schedule (kernel/sched/core.c:5244) [ 2024.400912][ C3] ? io_schedule_timeout (kernel/sched/core.c:6437) [ 2024.406164][ C3] ? timer_fixup_activate (kernel/time/timer.c:1014) [ 2024.411545][ C3] ? debug_object_deactivate (lib/debugobjects.c:557) [ 2024.417176][ C3] schedule (kernel/sched/core.c:6632 (discriminator 1)) [ 2024.421307][ C3] schedule_timeout (kernel/time/timer.c:1628) [ 2024.426185][ C3] ? usleep_range_state (kernel/time/timer.c:2129) [ 2024.431415][ C3] ? destroy_timer_on_stack (kernel/time/timer.c:2091) [ 2024.436670][ C3] ? do_ipt_get_ctl (net/ipv4/netfilter/ip_tables.c:850) [ 2024.441478][ C3] ? _raw_spin_unlock_irqrestore (arch/x86/include/asm/preempt.h:104) [ 2024.447260][ C3] ? prepare_to_swait_event (kernel/sched/swait.c:122 (discriminator 15)) [ 2024.452953][ C3] rcu_gp_fqs_loop (kernel/rcu/tree.c:1656 (discriminator 13)) [ 2024.457690][ C3] ? rcu_qs (kernel/rcu/tree.c:1626) [ 2024.461825][ C3] rcu_gp_kthread (kernel/rcu/tree.c:1858) [ 2024.466496][ C3] ? rcu_gp_init (kernel/rcu/tree.c:1830) [ 2024.471162][ C3] ? _raw_spin_unlock_irqrestore (include/linux/spinlock_api_smp.h:151) [ 2024.476900][ C3] ? __kthread_parkme (arch/x86/include/asm/bitops.h:207 (discriminator 4)) [ 2024.481700][ C3] ? rcu_gp_init (kernel/rcu/tree.c:1830) [ 2024.486316][ C3] kthread (kernel/kthread.c:376) [ 2024.490500][ C3] ? kthread_complete_and_exit (kernel/kthread.c:331) [ 2024.495859][ C3] ret_from_fork (arch/x86/entry/entry_64.S:314) [ 2024.500224][ C3] </TASK> [ 2024.503340][ C3] rcu: Stack dump where RCU GP kthread last ran: [ 2024.509455][ C3] Sending NMI from CPU 3 to CPUs 2: [ 2024.514745][ C2] NMI backtrace for cpu 2 [ 2024.514754][ C2] CPU: 2 PID: 12417 Comm: kworker/2:3 Tainted: G B L N 6.2.0-rc2-g1323854aa099 #1 [ 2024.514765][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 2024.514771][ C2] Workqueue: rcu_gp wait_rcu_exp_gp [ 2024.514784][ C2] RIP: 0010:smp_call_function_single (kernel/smp.c:442) [ 2024.514793][ C2] Code: 46 08 a8 01 74 38 48 b8 00 00 00 00 00 fc ff df 4c 89 ea 4d 89 ef 48 c1 ea 03 41 83 e7 07 48 01 c2 41 83 c7 03 f3 90 0f b6 02 <41> 38 c7 7c 08 84 c0 0f 85 9b 00 00 00 8b 46 08 a8 01 75 e7 48 b8 All code ======== 0: 46 08 a8 01 74 38 48 rex.RX or %r13b,0x48387401(%rax) 7: b8 00 00 00 00 mov $0x0,%eax c: 00 fc add %bh,%ah e: ff (bad) f: df 4c 89 ea fisttps -0x16(%rcx,%rcx,4) 13: 4d 89 ef mov %r13,%r15 16: 48 c1 ea 03 shr $0x3,%rdx 1a: 41 83 e7 07 and $0x7,%r15d 1e: 48 01 c2 add %rax,%rdx 21: 41 83 c7 03 add $0x3,%r15d 25: f3 90 pause 27: 0f b6 02 movzbl (%rdx),%eax 2a:* 41 38 c7 cmp %al,%r15b <-- trapping instruction 2d: 7c 08 jl 0x37 2f: 84 c0 test %al,%al 31: 0f 85 9b 00 00 00 jne 0xd2 37: 8b 46 08 mov 0x8(%rsi),%eax 3a: a8 01 test $0x1,%al 3c: 75 e7 jne 0x25 3e: 48 rex.W 3f: b8 .byte 0xb8 Code starting with the faulting instruction =========================================== 0: 41 38 c7 cmp %al,%r15b 3: 7c 08 jl 0xd 5: 84 c0 test %al,%al 7: 0f 85 9b 00 00 00 jne 0xa8 d: 8b 46 08 mov 0x8(%rsi),%eax 10: a8 01 test $0x1,%al 12: 75 e7 jne 0xfffffffffffffffb 14: 48 rex.W 15: b8 .byte 0xb8 [ 2024.514800][ C2] RSP: 0018:ffffc9000090fb40 EFLAGS: 00000202 [ 2024.514806][ C2] RAX: 0000000000000000 RBX: 1ffff92000121f6c RCX: 1ffffffff5ce2f1a [ 2024.514811][ C2] RDX: ffffed100d9e7879 RSI: ffff88806cf3c3c0 RDI: ffffffffae7178d0 [ 2024.514819][ C2] RBP: ffffc9000090fc10 R08: 0000000000000000 R09: ffffffffaf837f97 [ 2024.514823][ C2] R10: fffffbfff5f06ff2 R11: 0000000000000001 R12: 0000000000000001 [ 2024.514828][ C2] R13: ffff88806cf3c3c8 R14: 0000000000000000 R15: 0000000000000003 [ 2024.514842][ C2] FS: 0000000000000000(0000) GS:ffff88806cf00000(0000) knlGS:0000000000000000 [ 2024.514851][ C2] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.514855][ C2] CR2: 00007fc2565a8c14 CR3: 000000000a814001 CR4: 0000000000170ee0 [ 2024.514862][ C2] Call Trace: [ 2024.514866][ C2] <TASK> [ 2024.514873][ C2] ? rcu_barrier (kernel/rcu/tree_exp.h:735) [ 2024.514882][ C2] ? generic_exec_single (kernel/smp.c:729) [ 2024.514894][ C2] ? lockdep_hardirqs_on_prepare.part.0 (kernel/locking/lockdep.c:466) [ 2024.514912][ C2] __sync_rcu_exp_select_node_cpus (kernel/rcu/tree_exp.h:394) [ 2024.514941][ C2] sync_rcu_exp_select_cpus (kernel/rcu/tree_exp.h:549) [ 2024.514975][ C2] wait_rcu_exp_gp (kernel/rcu/tree_exp.h:513) [ 2024.514986][ C2] process_one_work (kernel/workqueue.c:2294) [ 2024.515003][ C2] ? rcu_read_unlock (include/linux/rcupdate.h:793 (discriminator 5)) [ 2024.515011][ C2] ? pwq_dec_nr_in_flight (kernel/workqueue.c:2184) [ 2024.515022][ C2] ? do_ipt_get_ctl (net/ipv4/netfilter/ip_tables.c:850) [ 2024.515028][ C2] ? rwlock_bug.part.0 (kernel/locking/spinlock_debug.c:113) [ 2024.515034][ C2] ? _raw_spin_lock_irq (include/linux/spinlock_api_smp.h:117) [ 2024.515047][ C2] worker_thread (include/linux/list.h:292) [ 2024.515072][ C2] ? process_one_work (kernel/workqueue.c:2379) [ 2024.515089][ C2] kthread (kernel/kthread.c:376) [ 2024.515098][ C2] ? kthread_complete_and_exit (kernel/kthread.c:331) [ 2024.515109][ C2] ret_from_fork (arch/x86/entry/entry_64.S:314) [ 2024.515132][ C2] </TASK>
Attachment:
config.xz
Description: application/xz
