Hello With the concurrent blktests nvme-rdma execution with both rdma_rxe and siw lead kernel BUG on 5.16.0-rc3, pls help check it, thanks. Reproducer: Run blktests nvme-rdma on two terminal at the same time terminal 1: # use_siw=1 nvme_trtype=rdma ./check nvme/ terminal 2: # nvme_trtype=rdma ./check nvme/ [ 1685.584327] run blktests nvme/013 at 2021-12-02 21:08:46 [ 1685.669804] eno2 speed is unknown, defaulting to 1000 [ 1685.674866] eno2 speed is unknown, defaulting to 1000 [ 1685.679941] eno2 speed is unknown, defaulting to 1000 [ 1685.686033] eno2 speed is unknown, defaulting to 1000 [ 1685.691087] eno2 speed is unknown, defaulting to 1000 [ 1685.697677] eno2 speed is unknown, defaulting to 1000 [ 1685.703727] eno3 speed is unknown, defaulting to 1000 [ 1685.708798] eno3 speed is unknown, defaulting to 1000 [ 1685.713863] eno3 speed is unknown, defaulting to 1000 [ 1685.719965] eno3 speed is unknown, defaulting to 1000 [ 1685.725043] eno3 speed is unknown, defaulting to 1000 [ 1685.731688] eno2 speed is unknown, defaulting to 1000 [ 1685.736763] eno3 speed is unknown, defaulting to 1000 [ 1685.742818] eno4 speed is unknown, defaulting to 1000 [ 1685.747881] eno4 speed is unknown, defaulting to 1000 [ 1685.752949] eno4 speed is unknown, defaulting to 1000 [ 1685.759134] eno4 speed is unknown, defaulting to 1000 [ 1685.764195] eno4 speed is unknown, defaulting to 1000 [ 1685.770914] eno2 speed is unknown, defaulting to 1000 [ 1685.775980] eno3 speed is unknown, defaulting to 1000 [ 1685.781047] eno4 speed is unknown, defaulting to 1000 [ 1686.002801] eno2 speed is unknown, defaulting to 1000 [ 1686.007867] eno3 speed is unknown, defaulting to 1000 [ 1686.012934] eno4 speed is unknown, defaulting to 1000 [ 1686.022521] rdma_rxe: rxe-ah pool destroyed with unfree'd elem [ 1686.289384] run blktests nvme/013 at 2021-12-02 21:08:46 [ 1686.356666] eno2 speed is unknown, defaulting to 1000 [ 1686.361735] eno2 speed is unknown, defaulting to 1000 [ 1686.366807] eno2 speed is unknown, defaulting to 1000 [ 1686.371876] eno2 speed is unknown, defaulting to 1000 [ 1686.378400] eno2 speed is unknown, defaulting to 1000 [ 1686.384419] eno3 speed is unknown, defaulting to 1000 [ 1686.389494] eno3 speed is unknown, defaulting to 1000 [ 1686.394583] eno3 speed is unknown, defaulting to 1000 [ 1686.399660] eno3 speed is unknown, defaulting to 1000 [ 1686.406219] eno2 speed is unknown, defaulting to 1000 [ 1686.411291] eno3 speed is unknown, defaulting to 1000 [ 1686.417275] eno4 speed is unknown, defaulting to 1000 [ 1686.422338] eno4 speed is unknown, defaulting to 1000 [ 1686.427401] eno4 speed is unknown, defaulting to 1000 [ 1686.432475] eno4 speed is unknown, defaulting to 1000 [ 1686.439038] eno2 speed is unknown, defaulting to 1000 [ 1686.444109] eno3 speed is unknown, defaulting to 1000 [ 1686.449180] eno4 speed is unknown, defaulting to 1000 [ 1686.873596] xfs filesystem being mounted at /mnt/blktests supports timestamps until 2038 (0x7fffffff) [ 1687.540606] xfs filesystem being mounted at /mnt/blktests supports timestamps until 2038 (0x7fffffff) [ 1693.658327] block nvme0n1: no available path - failing I/O [ 1693.663038] block nvme0n1: no available path - failing I/O [ 1693.663828] XFS (nvme0n1): log I/O error -5 [ 1693.665024] block nvme0n1: no available path - failing I/O [ 1693.665041] XFS (nvme0n1): log I/O error -5 [ 1693.665044] XFS (nvme0n1): Log I/O Error (0x2) detected at xlog_ioend_work+0x71/0x80 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem. [ 1693.665142] XFS (nvme0n1): Please unmount the filesystem and rectify the problem(s) [ 1693.720462] block nvme0n1: no available path - failing I/O [ 1693.728150] nvmet_rdma: post_recv cmd failed [ 1693.732432] nvmet_rdma: sending cmd response failed [ 1693.836083] eno2 speed is unknown, defaulting to 1000 [ 1693.841152] eno3 speed is unknown, defaulting to 1000 [ 1693.846217] eno4 speed is unknown, defaulting to 1000 [ 1693.852280] BUG: unable to handle page fault for address: ffffffffc09d2680 [ 1693.859156] #PF: supervisor instruction fetch in kernel mode [ 1693.864815] #PF: error_code(0x0010) - not-present page [ 1693.869953] PGD 2b5813067 P4D 2b5813067 PUD 2b5815067 PMD 13a157067 PTE 0 [ 1693.876740] Oops: 0010 [#1] PREEMPT SMP NOPTI [ 1693.881098] CPU: 15 PID: 16091 Comm: rdma Tainted: G S I 5.16.0-rc3 #1 [ 1693.888751] Hardware name: Dell Inc. PowerEdge R640/06NR82, BIOS 2.11.2 004/21/2021 [ 1693.896403] RIP: 0010:0xffffffffc09d2680 [ 1693.900329] Code: Unable to access opcode bytes at RIP 0xffffffffc09d2656. [ 1693.907202] RSP: 0018:ffffb3d5456237b0 EFLAGS: 00010286 [ 1693.912428] RAX: ffffffffc09d2680 RBX: ffff9d4adade2000 RCX: 0000000000000001 [ 1693.919559] RDX: 0000000080000001 RSI: ffffb3d5456237e8 RDI: ffff9d4adade2000 [ 1693.926693] RBP: ffffb3d5456237e8 R08: ffffb3d545623850 R09: 0000000000000230 [ 1693.933823] R10: 0000000000000002 R11: ffffb3d545623840 R12: ffff9d4adade2270 [ 1693.940957] R13: ffff9d4adade21e0 R14: 0000000000000005 R15: ffff9d4adade2220 [ 1693.948089] FS: 00007f2f0601c000(0000) GS:ffff9d59ffdc0000(0000) knlGS:0000000000000000 [ 1693.956176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1693.961921] CR2: ffffffffc09d2656 CR3: 0000000180578004 CR4: 00000000007706e0 [ 1693.969052] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1693.976177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1693.983309] PKRU: 55555554 [ 1693.986023] Call Trace: [ 1693.988474] <TASK> [ 1693.990582] ? cma_cm_event_handler+0x1d/0xd0 [rdma_cm] [ 1693.995817] ? cma_process_remove+0x73/0x290 [rdma_cm] [ 1694.000954] ? cma_remove_one+0x5a/0xd0 [rdma_cm] [ 1694.005661] ? remove_client_context+0x88/0xd0 [ib_core] [ 1694.010990] ? disable_device+0x8c/0x130 [ib_core] [ 1694.015790] ? xa_load+0x73/0xa0 [ 1694.019024] ? __ib_unregister_device+0x40/0xa0 [ib_core] [ 1694.024431] ? ib_unregister_device_and_put+0x33/0x50 [ib_core] [ 1694.030360] ? nldev_dellink+0x86/0xe0 [ib_core] [ 1694.035000] ? rdma_nl_rcv_msg+0x109/0x200 [ib_core] [ 1694.039978] ? __alloc_skb+0x8c/0x1b0 [ 1694.043645] ? __kmalloc_node_track_caller+0x184/0x340 [ 1694.048785] ? rdma_nl_rcv+0xc8/0x110 [ib_core] [ 1694.053325] ? netlink_unicast+0x1a2/0x280 [ 1694.057424] ? netlink_sendmsg+0x244/0x480 [ 1694.061524] ? sock_sendmsg+0x58/0x60 [ 1694.065188] ? __sys_sendto+0xee/0x160 [ 1694.068944] ? netlink_setsockopt+0x26e/0x3d0 [ 1694.073300] ? __sys_setsockopt+0xdc/0x1d0 [ 1694.077400] ? __x64_sys_sendto+0x24/0x30 [ 1694.081414] ? do_syscall_64+0x37/0x80 [ 1694.085164] ? entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1694.090391] </TASK> [ 1694.092584] Modules linked in: siw rpcrdma rdma_ucm ib_uverbs ib_srpt ib_isert iscsi_target_mod target_core_mod loop ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm ib_cm ib_core rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc vfat fat dm_multipath intel_rapl_msr intel_rapl_common isst_if_common skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm mgag200 i2c_algo_bit drm_kms_helper iTCO_wdt iTCO_vendor_support syscopyarea irqbypass sysfillrect crct10dif_pclmul sysimgblt crc32_pclmul fb_sys_fops ghash_clmulni_intel acpi_ipmi drm rapl ipmi_si intel_cstate mei_me intel_uncore i2c_i801 mei ipmi_devintf nd_pmem dax_pmem_compat wmi_bmof pcspkr device_dax intel_pch_thermal i2c_smbus lpc_ich ipmi_msghandler nd_btt dax_pmem_core acpi_power_meter xfs libcrc32c sd_mod t10_pi sg ahci libahci libata megaraid_sas nfit tg3 crc32c_intel libnvdimm wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nvmet] [ 1694.178277] CR2: ffffffffc09d2680 [ 1694.181596] ---[ end trace 9c234cd612cbb92a ]--- [ 1694.217410] RIP: 0010:0xffffffffc09d2680 [ 1694.221343] Code: Unable to access opcode bytes at RIP 0xffffffffc09d2656. [ 1694.228212] RSP: 0018:ffffb3d5456237b0 EFLAGS: 00010286 [ 1694.233437] RAX: ffffffffc09d2680 RBX: ffff9d4adade2000 RCX: 0000000000000001 [ 1694.240570] RDX: 0000000080000001 RSI: ffffb3d5456237e8 RDI: ffff9d4adade2000 [ 1694.247702] RBP: ffffb3d5456237e8 R08: ffffb3d545623850 R09: 0000000000000230 [ 1694.254828] R10: 0000000000000002 R11: ffffb3d545623840 R12: ffff9d4adade2270 [ 1694.261958] R13: ffff9d4adade21e0 R14: 0000000000000005 R15: ffff9d4adade2220 [ 1694.269091] FS: 00007f2f0601c000(0000) GS:ffff9d59ffdc0000(0000) knlGS:0000000000000000 [ 1694.277178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1694.282922] CR2: ffffffffc09d2656 CR3: 0000000180578004 CR4: 00000000007706e0 [ 1694.290054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1694.297180] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1694.304312] PKRU: 55555554 [ 1694.307025] Kernel panic - not syncing: Fatal exception [ 1694.772244] Kernel Offset: 0x35c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 1694.794394] ---[ end Kernel panic - not syncing: Fatal exception ]--- -- Best Regards, Yi Zhang
