On 4/15/22 7:31 AM, Luis Chamberlain wrote:
<...>
[ 195.218783] ------------[ cut here ]------------
[ 195.221242] WARNING: CPU: 7 PID: 201 at drivers/infiniband/sw/siw/siw_cm.c:255 siw_cep_put+0x125/0x130 [siw]
[ 195.222838] Modules linked in: ib_srp(E) scsi_transport_srp(E) target_core_pscsi(E) target_core_file(E) ib_srpt(E) target_core_iblock(E) target_core_mod(E) rdma_cm(E) iw_cm(E) ib_cm(E) scsi_debug(E) siw(E) null_blk(E) ib_umad(E) ib_uverbs(E) sd_mod(E) sg(E) dm_service_time(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) dm_multipath(E) ib_core(E) dm_mod(E) nvme_fabrics(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) ghash_clmulni_intel(E) aesni_intel(E) crypto_simd(E) cryptd(E) joydev(E) evdev(E) serio_raw(E) cirrus(E) drm_shmem_helper(E) drm_kms_helper(E) virtio_balloon(E) cec(E) i6300esb(E) button(E) drm(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) btrfs(E) blake2b_generic(E) xor(E) raid6_pq(E) zstd_compress(E) libcrc32c(E) crc32c_generic(E) virtio_net(E) net_failover(E) failover(E) virtio_blk(E) ata_generic(E) uhci_hcd(E) ehci_hcd(E) crc32_pclmul(E) crc32c_intel(E) ata_piix(E) psmouse(E) nvme(E) libata(E) virtio_pci(E)
[ 195.222986] virtio_pci_legacy_dev(E) virtio_pci_modern_dev(E) usbcore(E) virtio(E) usb_common(E) scsi_mod(E) nvme_core(E) i2c_piix4(E) virtio_ring(E) t10_pi(E) scsi_common(E) [last unloaded: null_blk]
[ 195.241036] sd 3:0:0:1: [sdd] Attached SCSI disn
[ 195.241188] CPU: 2 PID: 201 Comm: kworker/u16:22 Kdump: loaded Tainted: G E 5.17.0-rc7 #1
[ 195.246053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[ 195.249123] Workqueue: iw_cm_wq cm_work_handler [iw_cm]
[ 195.251274] RIP: 0010:siw_cep_put+0x125/0x130 [siw]
[ 195.253548] Code: bb c0 e8 ae 74 0f d7 48 89 ef 5d 41 5c 41 5d e9 b1 d6 ef d6 5d be 03 00 00 00 41 5c 41 5d e9 22 b7 0c d7 0f 0b e9 f3 fe ff ff <0f> 0b e9 1c ff ff ff 0f 1f 40 00 0f 1f 44 00 00 55 48 8d 6f 20 53
[ 195.258982] RSP: 0018:ffffbc53404ebc98 EFLAGS: 00010286
[ 195.261018] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
[ 195.263569] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffa03d1102a924
[ 195.266151] RBP: ffffa03d1102a900 R08: ffffa03d1102a920 R09: ffffbc53404ebc50
[ 195.269150] R10: ffffffff98a060e0 R11: 0000000000000000 R12: ffffa03cc4297000
[ 195.272744] R13: ffffa03d2a48aea0 R14: ffffa03d2a48ae78 R15: ffffa03cc427ad58
[ 195.275575] FS: 0000000000000000(0000) GS:ffffa03df7c80000(0000) knlGS:0000000000000000
[ 195.278932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 195.280963] CR2: 00005590bc2e4fe8 CR3: 000000008500a004 CR4: 0000000000770ee0
[ 195.282803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 195.284650] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 195.286522] PKRU: 55555554
[ 195.287998] Call Trace:
[ 195.289210] <TASK>
[ 195.290969] siw_reject+0xac/0x180 [siw]
[ 195.292679] iw_cm_reject+0x68/0xc0 [iw_cm]
[ 195.294136] cm_work_handler+0x59d/0xe20 [iw_cm]
[ 195.295588] process_one_work+0x1e2/0x3b0
[ 195.298338] worker_thread+0x50/0x3a0
[ 195.300330] ? rescuer_thread+0x390/0x390
[ 195.302269] kthread+0xe5/0x110
[ 195.304062] ? kthread_complete_and_exit+0x20/0x20
[ 195.307612] ret_from_fork+0x1f/0x30
[ 195.309585] </TASK>
[ 195.310674] ---[ end trace 0000000000000000 ]---
[ 195.313290] scsi host4: ib_srp: REJ received
[ 195.313293] scsi host4: REJ reason 0xffffff98
[ 195.315433] scsi host4: ib_srp: Connection 0/8 to 172.17.8.113 failed
[ 195.472718] ib_srp:srp_parse_in: ib_srp: 172.17.8.113 -> 172.17.8.113:0
[ 195.472739] ib_srp:srp_parse_in: ib_srp: 172.17.8.113:5555 -> 172.17.8.113:5555
[ 195.472807] ib_srp:srp_parse_in: ib_srp: [fe80::5054:ff:fe5b:90dc%3] -> [fe80::5054:ff:fe5b:90dc]:0/202442865%3
> [0] https://github.com/mcgrof/kdevops
> Luis
Hi, Bernard
I reproduced this issue, and it looks like a condition race between
'cm_work_handler' and 'siw_cm_work_handler'.
----------------------------------------------------------------
Thread0: Thread1:
siw_cm_work_handler cm_work_handler
----------------------------------------------------------------
step0:
siw_cm_upcall with
IW_CM_EVENT_CONNECT_REQUEST
===> cm_conn_req_handler
...
cm_id->cm_handler (failed)
iw_cm_reject
siw_reject
*step1*:
detach cep with listen_cep
----------------------------------------------------------------
When siw_reject is called in cm_work_handler, the related cep may have
not been detached with its listen_cep, through the two steps are very
close.
I think one simple way to fix this issue is keep step1 under
siw_cep_set_inuse's protection, and this will make siw_reject will be
pending util siw_cm_work_handler release the lock:
diff --git a/drivers/infiniband/sw/siw/siw_cm.c
b/drivers/infiniband/sw/siw/siw_cm.c
index 7acdd3c3a599..f033b6da1e9f 100644
--- a/drivers/infiniband/sw/siw/siw_cm.c
+++ b/drivers/infiniband/sw/siw/siw_cm.c
@@ -968,13 +968,15 @@ static void siw_accept_newconn(struct siw_cep *cep)
siw_cep_set_inuse(new_cep);
rv = siw_proc_mpareq(new_cep);
- siw_cep_set_free(new_cep);
if (rv != -EAGAIN) {
siw_cep_put(cep);
new_cep->listen_cep = NULL;
+ siw_cep_set_free(new_cep);
if (rv)
goto error;
+ } else {
+ siw_cep_set_free(new_cep);
}
}
return;
Thanks,
Cheng Xu
