Hello maintainers,
Our fuzzer identified one interesting reentrant bug that could cause hang
in the kernel. The crash log is like below:
[ 32.616575][ T2983]
[ 32.617000][ T2983] ============================================
[ 32.617879][ T2983] WARNING: possible recursive locking detected
[ 32.618759][ T2983] 6.1.70 #1 Not tainted
[ 32.619362][ T2983] --------------------------------------------
[ 32.620248][ T2983] hello.elf/2983 is trying to acquire lock:
[ 32.621084][ T2983] ffffffff91978ff8 (&rdma_nl_types[idx].sem){.+.+}-{3:3}, at: rdma_nl_rcv+0x30f/0x990
[ 32.624234][ T2983]
[ 32.624234][ T2983] but task is already holding lock:
[ 32.625237][ T2983] ffffffff91978ff8 (&rdma_nl_types[idx].sem){.+.+}-{3:3}, at: rdma_nl_rcv+0x30f/0x990
[ 32.626562][ T2983]
[ 32.626562][ T2983] other info that might help us debug this:
[ 32.627648][ T2983] Possible unsafe locking scenario:
[ 32.627648][ T2983]
[ 32.633708][ T2983] CPU0
[ 32.634184][ T2983] ----
[ 32.634646][ T2983] lock(&rdma_nl_types[idx].sem);
[ 32.635433][ T2983] lock(&rdma_nl_types[idx].sem);
[ 32.636155][ T2983]
[ 32.636155][ T2983] *** DEADLOCK ***
[ 32.636155][ T2983]
[ 32.637236][ T2983] May be due to missing lock nesting notation
[ 32.637236][ T2983]
[ 32.638408][ T2983] 2 locks held by hello.elf/2983:
[ 32.639135][ T2983] #0: ffffffff91978ff8 (&rdma_nl_types[idx].sem){.+.+}-{3:3}, at: rdma_nl_rcv+0x30f/0x990
[ 32.640605][ T2983] #1: ffff888103f8f690 (nlk_cb_mutex-RDMA){+.+.}-{3:3}, at: netlink_dump+0xd3/0xc60
[ 32.641981][ T2983]
[ 32.641981][ T2983] stack backtrace:
[ 32.642833][ T2983] CPU: 0 PID: 2983 Comm: hello.elf Not tainted 6.1.70 #1
[ 32.643830][ T2983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 32.645243][ T2983] Call Trace:
[ 32.645735][ T2983] <TASK>
[ 32.646197][ T2983] dump_stack_lvl+0x177/0x231
[ 32.646901][ T2983] ? nf_tcp_handle_invalid+0x605/0x605
[ 32.647705][ T2983] ? panic+0x725/0x725
[ 32.648350][ T2983] validate_chain+0x4dd0/0x6010
[ 32.649080][ T2983] ? reacquire_held_locks+0x5a0/0x5a0
[ 32.649864][ T2983] ? mark_lock+0x94/0x320
[ 32.650506][ T2983] ? lockdep_hardirqs_on_prepare+0x3fd/0x760
[ 32.651376][ T2983] ? print_irqtrace_events+0x210/0x210
[ 32.652182][ T2983] ? mark_lock+0x94/0x320
[ 32.652825][ T2983] __lock_acquire+0x12ad/0x2010
[ 32.653541][ T2983] lock_acquire+0x1b4/0x490
[ 32.654211][ T2983] ? rdma_nl_rcv+0x30f/0x990
[ 32.654891][ T2983] ? __might_sleep+0xd0/0xd0
[ 32.655569][ T2983] ? __lock_acquire+0x12ad/0x2010
[ 32.656316][ T2983] ? read_lock_is_recursive+0x10/0x10
[ 32.657109][ T2983] down_read+0x42/0x2d0
[ 32.657723][ T2983] ? rdma_nl_rcv+0x30f/0x990
[ 32.658400][ T2983] rdma_nl_rcv+0x30f/0x990
[ 32.659132][ T2983] ? rdma_nl_net_init+0x160/0x160
[ 32.659847][ T2983] ? netlink_lookup+0x30/0x200
[ 32.660519][ T2983] ? __netlink_lookup+0x2a/0x6d0
[ 32.661214][ T2983] ? netlink_lookup+0x30/0x200
[ 32.661880][ T2983] ? netlink_lookup+0x30/0x200
[ 32.662545][ T2983] netlink_unicast+0x74b/0x8c0
[ 32.663215][ T2983] rdma_nl_unicast+0x4b/0x60
[ 32.663852][ T2983] iwpm_send_hello+0x1d8/0x350
[ 32.664525][ T2983] ? iwpm_mapinfo_available+0x130/0x130
[ 32.665295][ T2983] ? iwpm_parse_nlmsg+0x124/0x260
[ 32.665995][ T2983] iwpm_hello_cb+0x1e1/0x2e0
[ 32.666638][ T2983] ? netlink_dump+0x236/0xc60
[ 32.667294][ T2983] ? iwpm_mapping_error_cb+0x3e0/0x3e0
[ 32.668064][ T2983] netlink_dump+0x592/0xc60
[ 32.668706][ T2983] ? netlink_lookup+0x200/0x200
[ 32.669381][ T2983] ? __netlink_lookup+0x2a/0x6d0
[ 32.670073][ T2983] ? netlink_lookup+0x30/0x200
[ 32.670731][ T2983] ? netlink_lookup+0x30/0x200
[ 32.671411][ T2983] __netlink_dump_start+0x54e/0x710
[ 32.672220][ T2983] rdma_nl_rcv+0x753/0x990
[ 32.672846][ T2983] ? rdma_nl_net_init+0x160/0x160
[ 32.673538][ T2983] ? iwpm_mapping_error_cb+0x3e0/0x3e0
[ 32.674316][ T2983] ? netlink_deliver_tap+0x2e/0x1b0
[ 32.675106][ T2983] ? net_generic+0x1e/0x240
[ 32.675778][ T2983] ? netlink_deliver_tap+0x2e/0x1b0
[ 32.676553][ T2983] netlink_unicast+0x74b/0x8c0
[ 32.677262][ T2983] netlink_sendmsg+0x882/0xb90
[ 32.677969][ T2983] ? netlink_getsockopt+0x550/0x550
[ 32.678732][ T2983] ? aa_sock_msg_perm+0x94/0x150
[ 32.679465][ T2983] ? bpf_lsm_socket_sendmsg+0x5/0x10
[ 32.680243][ T2983] ? security_socket_sendmsg+0x7c/0xa0
[ 32.681048][ T2983] __sys_sendto+0x456/0x5b0
[ 32.681724][ T2983] ? __ia32_sys_getpeername+0x80/0x80
[ 32.682510][ T2983] ? __lock_acquire+0x2010/0x2010
[ 32.683241][ T2983] ? lockdep_hardirqs_on_prepare+0x3fd/0x760
[ 32.684134][ T2983] ? fd_install+0x5c/0x4f0
[ 32.684794][ T2983] ? print_irqtrace_events+0x210/0x210
[ 32.685608][ T2983] __x64_sys_sendto+0xda/0xf0
[ 32.686298][ T2983] do_syscall_64+0x45/0x90
[ 32.686955][ T2983] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 32.687810][ T2983] RIP: 0033:0x440624
[ 32.691944][ T2983] RSP: 002b:00007ffc82f3ee48 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[ 32.693136][ T2983] RAX: ffffffffffffffda RBX: 0000000000400400 RCX: 0000000000440624
[ 32.694264][ T2983] RDX: 0000000000000018 RSI: 00007ffc82f3ee80 RDI: 0000000000000003
[ 32.695387][ T2983] RBP: 00007ffc82f3fe90 R08: 000000000047df08 R09: 000000000000000c
[ 32.696621][ T2983] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000403990
[ 32.697693][ T2983] R13: 0000000000000000 R14: 00000000006a6018 R15: 0000000000000000
[ 32.698774][ T2983] </TASK>
In a nutshell, the callback function for the command RDMA_NL_IWPM_HELLO, iwpm_hello_cb,
can further call rdma_nl_unicast, leading to repeated calls that may cause
a deadlock and potentially harm the kernel.
I am not familiar with the internal workings of the callback mechanism or how
IWPMD utilizes it, so I'm uncertain whether this reentrancy is expected behavior.
If it is, perhaps a reference counter should be used instead of an rw_semaphore.
If not, a proper check should be implemented.
Regards,
Lin
