On 2022/1/20 1:09, Lukasz Maniak wrote:
> On Wed, Jan 19, 2022 at 05:06:55PM +0100, Lukasz Maniak wrote:
>> On Wed, Jan 19, 2022 at 06:22:07PM +0800, Yicong Yang wrote:
>>> Hi Lukasz, Bjorn,
>>>
>>> FYI, I tested with Mellanox CX-5, the VF also exists after FLR. Here's the operation:
>>
>
> Please disregard my previous email. I missed your point.
> I take it that the Mellanox CX-5 also violates the spec.
>
> As for using pci_disable_sriov() I did a test to get a backtrace for
> deadlock:
> [ 846.904248] Call Trace:
> [ 846.904251] <TASK>
> [ 846.904272] __schedule+0x302/0x950
> [ 846.904282] schedule+0x58/0xd0
> [ 846.904286] pci_wait_cfg+0x63/0xb0
> [ 846.904290] ? wait_woken+0x70/0x70
> [ 846.904296] pci_cfg_access_lock+0x48/0x50
> [ 846.904300] sriov_disable+0x4d/0xf0
> [ 846.904306] pci_disable_sriov+0x26/0x30
> [ 846.904310] pcie_flr+0x2b/0x100
> [ 846.904317] pcie_reset_flr+0x25/0x30
> [ 846.904322] __pci_reset_function_locked+0x42/0x60
> [ 846.904327] pci_reset_function+0x40/0x70
> [ 846.904334] reset_store+0x5c/0xa0
> [ 846.904347] dev_attr_store+0x17/0x30
> [ 846.904357] sysfs_kf_write+0x3f/0x50
> [ 846.904365] kernfs_fop_write_iter+0x13b/0x1d0
> [ 846.904371] new_sync_write+0x117/0x1b0
> [ 846.904379] vfs_write+0x219/0x2b0
> [ 846.904384] ksys_write+0x67/0xe0
> [ 846.904390] __x64_sys_write+0x1a/0x20
> [ 846.904395] do_syscall_64+0x5c/0xc0
> [ 846.904401] ? debug_smp_processor_id+0x17/0x20
> [ 846.904406] ? fpregs_assert_state_consistent+0x26/0x50
> [ 846.904413] ? exit_to_user_mode_prepare+0x3f/0x1b0
> [ 846.904418] ? irqentry_exit_to_user_mode+0x9/0x20
> [ 846.904423] ? irqentry_exit+0x33/0x40
> [ 846.904427] ? exc_page_fault+0x89/0x180
> [ 846.904431] ? asm_exc_page_fault+0x8/0x30
> [ 846.904438] entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> As can be noticed during FLR we are already on a locked path for the
> device in __pci_reset_function_locked(). In addition, the device will reset
> the BARs during FLR on its own.
>
> If we still would like to use pci_disable_sriov() for this purpose we
> need to pass a flag to sriov_disable() and use conditionally twice. It
> would look something like this:
>
> static void sriov_disable(struct pci_dev *dev, bool flr)
> {
> struct pci_sriov *iov = dev->sriov;
>
> if (!iov->num_VFs)
> return;
>
> sriov_del_vfs(dev);
>
> if (!flr) {
> iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
> pci_cfg_access_lock(dev);
> pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
> ssleep(1);
> pci_cfg_access_unlock(dev);
> }
>
It still leaves the VFE uncleared. So after reset the hardware IOV state is unsynchronized
with the system as we've removed the VFs already. so you may need:
static void sriov_disable(struct pci_dev *dev, bool locked)
{
struct pci_sriov *iov = dev->sriov;
if (!iov->num_VFs)
return;
sriov_del_vfs(dev);
iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
if (!locked)
pci_cfg_access_lock(dev);
pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
ssleep(1);
if (!locked)
pci_cfg_access_unlock(dev);
pcibios_sriov_disable(dev);
if (iov->link != dev->devfn)
sysfs_remove_link(&dev->dev.kobj, "dep_link");
iov->num_VFs = 0;
if (!flr)
pci_iov_set_numvfs(dev, 0);
}
I'm not sure this is correct as we disable VF not through PF driver
and whether these PF driver involed need to modified after this
change.
(Yes through pdev->driver->sriov_configure() we'll also meet the
deadlock problem but that's the next step question).
With your patch based on 5.16 release when doing FLR reset on VF's PF
of Mellanox CX-5, the log says that there's a resource leakage and
leads to several calltraces. I paste the log below.
Perhaps Mellanox maintainers could help on this.
Thanks.
[ 435.211235] mlx5_core 0000:01:00.0: E-Switch: Enable: mode(LEGACY), nvfs(1), active vports(2)
[ 435.327158] pci 0000:01:00.2: [15b3:101a] type 00 class 0x020000
[ 435.333197] pci 0000:01:00.2: enabling Extended Tags
[ 435.338936] pci 0000:01:00.2: calling mellanox_check_broken_intx_masking+0x0/0x1a0 @ 4328
[ 435.347174] pci 0000:01:00.2: mellanox_check_broken_intx_masking+0x0/0x1a0 took 0 usecs
[ 435.355224] mlx5_core 0000:01:00.2: Adding to iommu group 49
[ 435.361639] mlx5_core 0000:01:00.2: enabling device (0000 -> 0002)
[ 435.367917] mlx5_core 0000:01:00.2: firmware version: 16.27.1016
[ 435.611252] mlx5_core 0000:01:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 435.628931] mlx5_core 0000:01:00.2: Assigned random MAC address 72:51:df:ba:6a:1e
[ 435.636824] mlx5_core 0000:01:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[ 435.744665] mlx5_core 0000:01:00.2: Supported tc offload range - chains: 1, prios: 1
[ 446.080370] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): 2RST_QP(0x50a) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x7ea02d)
[ 446.094054] infiniband mlx5_2: destroy_qp_common:2599:(pid 4328): mlx5_ib: modify QP 0x000504 to RESET failed
[ 446.104036] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DESTROY_QP(0x501) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x25b161)
[ 446.118092] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DESTROY_CQ(0x401) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x1870ad)
[ 446.132028] ------------[ cut here ]------------
[ 446.136629] Destroy of kernel CQ shouldn't fail
[ 446.136648] WARNING: CPU: 37 PID: 4328 at drivers/infiniband/core/cq.c:345 ib_free_cq+0x16c/0x174
[ 446.149991] Modules linked in: bluetooth rfkill xt_addrtype iptable_filter xt_conntrack overlay dm_mod ib_isert iscsi_target_mod rpcrdma ib_umad ib_iser ib_ipoib libiscsi scsi_transport_iscsi mlx5_ib hns_roce_hw_v2 hisi_hpre hisi_sec2 sbsa_gwdt hisi_zip hisi_trng_v2 arm_spe_pmu hisi_qm hisi_uncore_l3c_pmu hisi_uncore_hha_pmu uacce hisi_uncore_ddrc_pmu crct10dif_ce rng_core spi_dw_mmio hisi_uncore_pmu hns3 hclge hisi_sas_v3_hw hnae3 hisi_sas_main libsas
[ 446.189868] CPU: 37 PID: 4328 Comm: bash Kdump: loaded Not tainted 5.16.0-pcie-iov+ #14
[ 446.197833] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
[ 446.206661] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 446.213591] pc : ib_free_cq+0x16c/0x174
[ 446.217410] lr : ib_free_cq+0x16c/0x174
[ 446.221229] sp : ffff80006172b5f0
[ 446.224527] x29: ffff80006172b5f0 x28: ffff008009880000 x27: 0000000000000000
[ 446.231632] x26: ffff00800e8c00d0 x25: 0000000000000000 x24: 0000000000000000
[ 446.238734] x23: ffff0041495b0000 x22: 0000000000000000 x21: ffff00413f3c0800
[ 446.245837] x20: ffff00413f3c0928 x19: ffff00408d4bf800 x18: 0000000000000030
[ 446.252939] x17: 0000000000000001 x16: ffffd9e776fbe8a0 x15: ffff0080098805d0
[ 446.260043] x14: 0000000000000000 x13: 6c6961662074276e x12: 646c756f68732051
[ 446.267145] x11: 43206c656e72656b x10: ffff205f2b8c5a28 x9 : ffffd9e776239854
[ 446.274247] x8 : ffff205f2b5e0000 x7 : ffff205f2b8a0000 x6 : 0000000000025a28
[ 446.281350] x5 : ffff00af3faf39b0 x4 : 0000000000000000 x3 : 0000000000000027
[ 446.288452] x2 : 0000000000000023 x1 : 16e32808cb35d300 x0 : 0000000000000000
[ 446.295554] Call trace:
[ 446.297989] ib_free_cq+0x16c/0x174
[ 446.301463] mlx5_ib_destroy_gsi+0xac/0x110 [mlx5_ib]
[ 446.306507] mlx5_ib_destroy_qp+0x64/0x70 [mlx5_ib]
[ 446.311372] ib_destroy_qp_user+0x7c/0x19c
[ 446.315450] ib_mad_port_close+0xac/0x164
[ 446.319444] ib_mad_remove_device+0x88/0xdc
[ 446.323608] remove_client_context+0xa4/0x100
[ 446.327946] disable_device+0x98/0x170
[ 446.331680] __ib_unregister_device+0x54/0xf0
[ 446.336017] ib_unregister_device+0x34/0x50
[ 446.340182] mlx5_ib_stage_ib_reg_cleanup+0x1c/0x2c [mlx5_ib]
[ 446.345909] mlx5r_remove+0x54/0x80 [mlx5_ib]
[ 446.350254] auxiliary_bus_remove+0x30/0x4c
[ 446.354421] __device_release_driver+0x190/0x23c
[ 446.359017] device_release_driver+0x38/0x50
[ 446.363268] bus_remove_device+0x130/0x140
[ 446.367345] device_del+0x184/0x434
[ 446.370819] delete_drivers+0x54/0xe0
[ 446.374466] mlx5_unregister_device+0x40/0x80
[ 446.378803] mlx5_uninit_one+0x34/0xd4
[ 446.382536] remove_one+0x4c/0xd0
[ 446.385838] pci_device_remove+0x48/0xe0
[ 446.389744] __device_release_driver+0x190/0x23c
[ 446.394340] device_release_driver+0x38/0x50
[ 446.398592] pci_stop_bus_device+0x8c/0xd0
[ 446.402670] pci_stop_and_remove_bus_device+0x24/0x40
[ 446.407699] pci_iov_remove_virtfn+0xb8/0x130
[ 446.412038] pci_reset_iov_state+0x5c/0xb0
[ 446.416114] pcie_flr+0x38/0x130
[ 446.419329] pcie_reset_flr+0x40/0x54
[ 446.422976] __pci_reset_function_locked+0x54/0x80
[ 446.427745] pci_reset_function+0x4c/0x90
[ 446.431738] reset_store+0x70/0xc0
[ 446.435125] dev_attr_store+0x24/0x40
[ 446.438772] sysfs_kf_write+0x50/0x60
[ 446.442421] kernfs_fop_write_iter+0x124/0x1b4
[ 446.446845] new_sync_write+0xf0/0x190
[ 446.450578] vfs_write+0x25c/0x2c0
[ 446.453964] ksys_write+0x78/0x104
[ 446.457350] __arm64_sys_write+0x28/0x3c
[ 446.461254] invoke_syscall+0x50/0x120
[ 446.464988] el0_svc_common.constprop.0+0x188/0x190
[ 446.469843] do_el0_svc+0x30/0x90
[ 446.473145] el0_svc+0x20/0x90
[ 446.476186] el0t_64_sync_handler+0x1a8/0x1ac
[ 446.480524] el0t_64_sync+0x1a0/0x1a4
[ 446.484171] ---[ end trace d334416dff5120e3 ]---
[ 446.488831] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DESTROY_CQ(0x401) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x1870ad)
[ 446.503029] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DEALLOC_PD(0x801) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x31b635)
[ 446.516961] ------------[ cut here ]------------
[ 446.521560] Destroy of kernel PD shouldn't fail
[ 446.521570] WARNING: CPU: 37 PID: 4328 at include/rdma/ib_verbs.h:3498 ib_mad_port_close+0x138/0x164
[ 446.535170] Modules linked in: bluetooth rfkill xt_addrtype iptable_filter xt_conntrack overlay dm_mod ib_isert iscsi_target_mod rpcrdma ib_umad ib_iser ib_ipoib libiscsi scsi_transport_iscsi mlx5_ib hns_roce_hw_v2 hisi_hpre hisi_sec2 sbsa_gwdt hisi_zip hisi_trng_v2 arm_spe_pmu hisi_qm hisi_uncore_l3c_pmu hisi_uncore_hha_pmu uacce hisi_uncore_ddrc_pmu crct10dif_ce rng_core spi_dw_mmio hisi_uncore_pmu hns3 hclge hisi_sas_v3_hw hnae3 hisi_sas_main libsas
[ 446.575038] CPU: 37 PID: 4328 Comm: bash Kdump: loaded Tainted: G W 5.16.0-pcie-iov+ #14
[ 446.584386] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
[ 446.593214] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 446.600144] pc : ib_mad_port_close+0x138/0x164
[ 446.604567] lr : ib_mad_port_close+0x138/0x164
[ 446.608990] sp : ffff80006172b6b0
[ 446.612290] x29: ffff80006172b6b0 x28: ffff008009880000 x27: 0000000000000000
[ 446.619393] x26: ffff00800e8c00d0 x25: 0000000000000000 x24: 0000000000000000
[ 446.626495] x23: ffff0041495b04a8 x22: ffffd9e7794f8920 x21: ffff0040ed5978f8
[ 446.633598] x20: ffff0040ed597870 x19: ffff0040ed597000 x18: 0000000000000030
[ 446.640700] x17: 0000000000000001 x16: ffffd9e776fb5f04 x15: ffff0080098805d0
[ 446.647801] x14: 0000000000000000 x13: 6c6961662074276e x12: 646c756f68732044
[ 446.654903] x11: 50206c656e72656b x10: ffff205f2b8c60b8 x9 : ffffd9e776239854
[ 446.662006] x8 : ffff205f2b5e0000 x7 : ffff205f2b8a0000 x6 : 00000000000260b8
[ 446.669110] x5 : ffff00af3faf39b0 x4 : 0000000000000000 x3 : 0000000000000027
[ 446.676212] x2 : 0000000000000023 x1 : 16e32808cb35d300 x0 : 0000000000000000
[ 446.683314] Call trace:
[ 446.685750] ib_mad_port_close+0x138/0x164
[ 446.689828] ib_mad_remove_device+0x88/0xdc
[ 446.693994] remove_client_context+0xa4/0x100
[ 446.698330] disable_device+0x98/0x170
[ 446.702062] __ib_unregister_device+0x54/0xf0
[ 446.706400] ib_unregister_device+0x34/0x50
[ 446.710565] mlx5_ib_stage_ib_reg_cleanup+0x1c/0x2c [mlx5_ib]
[ 446.716293] mlx5r_remove+0x54/0x80 [mlx5_ib]
[ 446.720639] auxiliary_bus_remove+0x30/0x4c
[ 446.724804] __device_release_driver+0x190/0x23c
[ 446.729402] device_release_driver+0x38/0x50
[ 446.733652] bus_remove_device+0x130/0x140
[ 446.737731] device_del+0x184/0x434
[ 446.741204] delete_drivers+0x54/0xe0
[ 446.744850] mlx5_unregister_device+0x40/0x80
[ 446.749186] mlx5_uninit_one+0x34/0xd4
[ 446.752918] remove_one+0x4c/0xd0
[ 446.756219] pci_device_remove+0x48/0xe0
[ 446.760125] __device_release_driver+0x190/0x23c
[ 446.764720] device_release_driver+0x38/0x50
[ 446.768971] pci_stop_bus_device+0x8c/0xd0
[ 446.773048] pci_stop_and_remove_bus_device+0x24/0x40
[ 446.778075] pci_iov_remove_virtfn+0xb8/0x130
[ 446.782413] pci_reset_iov_state+0x5c/0xb0
[ 446.786490] pcie_flr+0x38/0x130
[ 446.789703] pcie_reset_flr+0x40/0x54
[ 446.793349] __pci_reset_function_locked+0x54/0x80
[ 446.798118] pci_reset_function+0x4c/0x90
[ 446.802110] reset_store+0x70/0xc0
[ 446.805498] dev_attr_store+0x24/0x40
[ 446.809144] sysfs_kf_write+0x50/0x60
[ 446.812791] kernfs_fop_write_iter+0x124/0x1b4
[ 446.817216] new_sync_write+0xf0/0x190
[ 446.820947] vfs_write+0x25c/0x2c0
[ 446.824335] ksys_write+0x78/0x104
[ 446.827723] __arm64_sys_write+0x28/0x3c
[ 446.831628] invoke_syscall+0x50/0x120
[ 446.835359] el0_svc_common.constprop.0+0x188/0x190
[ 446.840215] do_el0_svc+0x30/0x90
[ 446.843516] el0_svc+0x20/0x90
[ 446.846558] el0t_64_sync_handler+0x1a8/0x1ac
[ 446.850894] el0t_64_sync+0x1a0/0x1a4
[ 446.854539] ---[ end trace d334416dff5120e4 ]---
[ 446.903110] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DESTROY_UCTX(0xa06) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x15555)
[ 446.917110] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): 2RST_QP(0x50a) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x7ea02d)
[ 446.930787] infiniband mlx5_2: destroy_qp_common:2599:(pid 4328): mlx5_ib: modify QP 0x000505 to RESET failed
[ 446.940726] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DESTROY_QP(0x501) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x25b161)
[ 446.954736] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DESTROY_CQ(0x401) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x1870ad)
[ 446.968832] mlx5_core 0000:01:00.2: mlx5_cmd_check:782:(pid 4328): DEALLOC_PD(0x801) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0x31b635)
[ 446.982795] ------------[ cut here ]------------
[ 446.987400] Destroy of kernel PD shouldn't fail
[ 446.987417] WARNING: CPU: 37 PID: 4328 at include/rdma/ib_verbs.h:3498 mlx5_ib_stage_pre_ib_reg_umr_cleanup+0x8c/0xc0 [mlx5_ib]
[ 447.003359] Modules linked in: bluetooth rfkill xt_addrtype iptable_filter xt_conntrack overlay dm_mod ib_isert iscsi_target_mod rpcrdma ib_umad ib_iser ib_ipoib libiscsi scsi_transport_iscsi mlx5_ib hns_roce_hw_v2 hisi_hpre hisi_sec2 sbsa_gwdt hisi_zip hisi_trng_v2 arm_spe_pmu hisi_qm hisi_uncore_l3c_pmu hisi_uncore_hha_pmu uacce hisi_uncore_ddrc_pmu crct10dif_ce rng_core spi_dw_mmio hisi_uncore_pmu hns3 hclge hisi_sas_v3_hw hnae3 hisi_sas_main libsas
[ 447.043229] CPU: 37 PID: 4328 Comm: bash Kdump: loaded Tainted: G W 5.16.0-pcie-iov+ #14
[ 447.052577] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
[ 447.061406] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 447.068336] pc : mlx5_ib_stage_pre_ib_reg_umr_cleanup+0x8c/0xc0 [mlx5_ib]
[ 447.075102] lr : mlx5_ib_stage_pre_ib_reg_umr_cleanup+0x8c/0xc0 [mlx5_ib]
[ 447.081866] sp : ffff80006172b7e0
[ 447.085165] x29: ffff80006172b7e0 x28: ffff008009880000 x27: 0000000000000000
[ 447.092269] x26: ffff00800e8c00d0 x25: 0000000000000000 x24: ffff008009880000
[ 447.099372] x23: 0000000000000080 x22: ffffd9e70116cfd0 x21: ffffd9e701163d28
[ 447.106474] x20: ffff0041495b0000 x19: ffff0041495b0000 x18: 0000000000000030
[ 447.113576] x17: 0000000000000001 x16: ffffd9e77771be14 x15: ffff0080098805d0
[ 447.120679] x14: 0000000000000000 x13: 6c6961662074276e x12: 646c756f68732044
[ 447.127781] x11: 50206c656e72656b x10: ffff205f2b8c6748 x9 : ffffd9e776239854
[ 447.134884] x8 : ffff205f2b5e0000 x7 : ffff205f2b8a0000 x6 : 0000000000026748
[ 447.141986] x5 : ffff00af3faf39b0 x4 : 0000000000000000 x3 : 0000000000000027
[ 447.149090] x2 : 0000000000000023 x1 : 16e32808cb35d300 x0 : 0000000000000000
[ 447.156192] Call trace:
[ 447.158627] mlx5_ib_stage_pre_ib_reg_umr_cleanup+0x8c/0xc0 [mlx5_ib]
[ 447.165046] mlx5r_remove+0x54/0x80 [mlx5_ib]
[ 447.169390] auxiliary_bus_remove+0x30/0x4c
[ 447.173556] __device_release_driver+0x190/0x23c
[ 447.178152] device_release_driver+0x38/0x50
[ 447.182402] bus_remove_device+0x130/0x140
[ 447.186480] device_del+0x184/0x434
[ 447.189954] delete_drivers+0x54/0xe0
[ 447.193600] mlx5_unregister_device+0x40/0x80
[ 447.197938] mlx5_uninit_one+0x34/0xd4
[ 447.201671] remove_one+0x4c/0xd0
[ 447.204972] pci_device_remove+0x48/0xe0
[ 447.208876] __device_release_driver+0x190/0x23c
[ 447.213472] device_release_driver+0x38/0x50
[ 447.217723] pci_stop_bus_device+0x8c/0xd0
[ 447.221801] pci_stop_and_remove_bus_device+0x24/0x40
[ 447.226830] pci_iov_remove_virtfn+0xb8/0x130
[ 447.231168] pci_reset_iov_state+0x5c/0xb0
[ 447.235247] pcie_flr+0x38/0x130
[ 447.238460] pcie_reset_flr+0x40/0x54
[ 447.242106] __pci_reset_function_locked+0x54/0x80
[ 447.246874] pci_reset_function+0x4c/0x90
[ 447.250866] reset_store+0x70/0xc0
[ 447.254252] dev_attr_store+0x24/0x40
[ 447.257898] sysfs_kf_write+0x50/0x60
[ 447.261545] kernfs_fop_write_iter+0x124/0x1b4
[ 447.265969] new_sync_write+0xf0/0x190
[ 447.269702] vfs_write+0x25c/0x2c0
[ 447.273089] ksys_write+0x78/0x104
[ 447.276477] __arm64_sys_write+0x28/0x3c
[ 447.280382] invoke_syscall+0x50/0x120
[ 447.284115] el0_svc_common.constprop.0+0x188/0x190
[ 447.288970] do_el0_svc+0x30/0x90
[ 447.292269] el0_svc+0x20/0x90
[ 447.295310] el0t_64_sync_handler+0x1a8/0x1ac
[ 447.299648] el0t_64_sync+0x1a0/0x1a4
[ 447.303294] ---[ end trace d334416dff5120e5 ]---
[ 447.307952] mlx5_core 0000:01:00.2: up_rel_func:90:(pid 4328): failed to free uar index 17
[ 447.351101] ------------[ cut here ]------------
[ 447.355699] Destroy of kernel SRQ shouldn't fail
[ 447.355712] WARNING: CPU: 37 PID: 4328 at include/rdma/ib_verbs.h:3688 mlx5_ib_dev_res_cleanup+0xc4/0x150 [mlx5_ib]
[ 447.370701] Modules linked in: bluetooth rfkill xt_addrtype iptable_filter xt_conntrack overlay dm_mod ib_isert iscsi_target_mod rpcrdma ib_umad ib_iser ib_ipoib libiscsi scsi_transport_iscsi mlx5_ib hns_roce_hw_v2 hisi_hpre hisi_sec2 sbsa_gwdt hisi_zip hisi_trng_v2 arm_spe_pmu hisi_qm hisi_uncore_l3c_pmu hisi_uncore_hha_pmu uacce hisi_uncore_ddrc_pmu crct10dif_ce rng_core spi_dw_mmio hisi_uncore_pmu hns3 hclge hisi_sas_v3_hw hnae3 hisi_sas_main libsas
[ 447.410571] CPU: 37 PID: 4328 Comm: bash Kdump: loaded Tainted: G W 5.16.0-pcie-iov+ #14
[ 447.419919] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
[ 447.428749] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 447.435678] pc : mlx5_ib_dev_res_cleanup+0xc4/0x150 [mlx5_ib]
[ 447.441406] lr : mlx5_ib_dev_res_cleanup+0xc4/0x150 [mlx5_ib]
[ 447.447132] sp : ffff80006172b7e0
[ 447.450431] x29: ffff80006172b7e0 x28: ffff008009880000 x27: 0000000000000000
[ 447.457533] x26: ffff00800e8c00d0 x25: 0000000000000000 x24: ffff008009880000
[ 447.464635] x23: 0000000000000080 x22: ffffd9e70116cfd0 x21: ffffd9e701163d28
[ 447.471737] x20: ffff0041495b0b70 x19: ffff0041495b0000 x18: 0000000000000030
[ 447.478838] x17: 657266206f742064 x16: ffffd9e77771be14 x15: ffff0080098805d0
[ 447.485942] x14: 0000000000000000 x13: 6c6961662074276e x12: 646c756f68732051
[ 447.493045] x11: 5253206c656e7265 x10: ffff205f2b8c6cd0 x9 : ffffd9e776239854
[ 447.500147] x8 : ffff205f2b5e0000 x7 : ffff205f2b8a0000 x6 : 0000000000026cd0
[ 447.507250] x5 : ffff00af3faf39b0 x4 : 0000000000000000 x3 : 0000000000000027
[ 447.514351] x2 : 0000000000000023 x1 : 16e32808cb35d300 x0 : 0000000000000000
[ 447.521453] Call trace:
[ 447.523888] mlx5_ib_dev_res_cleanup+0xc4/0x150 [mlx5_ib]
[ 447.529270] mlx5r_remove+0x54/0x80 [mlx5_ib]
[ 447.533613] auxiliary_bus_remove+0x30/0x4c
[ 447.537778] __device_release_driver+0x190/0x23c
[ 447.542375] device_release_driver+0x38/0x50
[ 447.546627] bus_remove_device+0x130/0x140
[ 447.550703] device_del+0x184/0x434
[ 447.554175] delete_drivers+0x54/0xe0
[ 447.557822] mlx5_unregister_device+0x40/0x80
[ 447.562159] mlx5_uninit_one+0x34/0xd4
[ 447.565892] remove_one+0x4c/0xd0
[ 447.569192] pci_device_remove+0x48/0xe0
[ 447.573098] __device_release_driver+0x190/0x23c
[ 447.577694] device_release_driver+0x38/0x50
[ 447.581944] pci_stop_bus_device+0x8c/0xd0
[ 447.586022] pci_stop_and_remove_bus_device+0x24/0x40
[ 447.591049] pci_iov_remove_virtfn+0xb8/0x130
[ 447.595386] pci_reset_iov_state+0x5c/0xb0
[ 447.599464] pcie_flr+0x38/0x130
[ 447.602678] pcie_reset_flr+0x40/0x54
[ 447.606323] __pci_reset_function_locked+0x54/0x80
[ 447.611092] pci_reset_function+0x4c/0x90
[ 447.615085] reset_store+0x70/0xc0
[ 447.618473] dev_attr_store+0x24/0x40
[ 447.622120] sysfs_kf_write+0x50/0x60
[ 447.625766] kernfs_fop_write_iter+0x124/0x1b4
[ 447.630189] new_sync_write+0xf0/0x190
[ 447.633921] vfs_write+0x25c/0x2c0
[ 447.637308] ksys_write+0x78/0x104
[ 447.640696] __arm64_sys_write+0x28/0x3c
[ 447.644601] invoke_syscall+0x50/0x120
[ 447.648334] el0_svc_common.constprop.0+0x188/0x190
[ 447.653190] do_el0_svc+0x30/0x90
[ 447.656490] el0_svc+0x20/0x90
[ 447.659531] el0t_64_sync_handler+0x1a8/0x1ac
[ 447.663867] el0t_64_sync+0x1a0/0x1a4
[ 447.667512] ---[ end trace d334416dff5120e6 ]---
[ 447.672291] ------------[ cut here ]------------
[ 447.676892] Destroy of kernel CQ shouldn't fail
[ 447.676904] WARNING: CPU: 37 PID: 4328 at include/rdma/ib_verbs.h:3936 mlx5_ib_dev_res_cleanup+0x118/0x150 [mlx5_ib]
[ 447.691895] Modules linked in: bluetooth rfkill xt_addrtype iptable_filter xt_conntrack overlay dm_mod ib_isert iscsi_target_mod rpcrdma ib_umad ib_iser ib_ipoib libiscsi scsi_transport_iscsi mlx5_ib hns_roce_hw_v2 hisi_hpre hisi_sec2 sbsa_gwdt hisi_zip hisi_trng_v2 arm_spe_pmu hisi_qm hisi_uncore_l3c_pmu hisi_uncore_hha_pmu uacce hisi_uncore_ddrc_pmu crct10dif_ce rng_core spi_dw_mmio hisi_uncore_pmu hns3 hclge hisi_sas_v3_hw hnae3 hisi_sas_main libsas
[ 447.731764] CPU: 37 PID: 4328 Comm: bash Kdump: loaded Tainted: G W 5.16.0-pcie-iov+ #14
[ 447.741112] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
[ 447.749942] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 447.756872] pc : mlx5_ib_dev_res_cleanup+0x118/0x150 [mlx5_ib]
[ 447.762685] lr : mlx5_ib_dev_res_cleanup+0x118/0x150 [mlx5_ib]
[ 447.768497] sp : ffff80006172b7e0
[ 447.771795] x29: ffff80006172b7e0 x28: ffff008009880000 x27: 0000000000000000
[ 447.778897] x26: ffff00800e8c00d0 x25: 0000000000000000 x24: ffff008009880000
[ 447.786000] x23: 0000000000000080 x22: ffffd9e70116cfd0 x21: ffffd9e701163d28
[ 447.793103] x20: ffff0041495b0b70 x19: ffff0041495b0000 x18: 0000000000000030
[ 447.800207] x17: 657266206f742064 x16: ffffd9e77771be14 x15: ffff0080098805d0
[ 447.807309] x14: 0000000000000000 x13: 6c6961662074276e x12: 646c756f68732051
[ 447.814410] x11: 43206c656e72656b x10: ffff205f2b8c7240 x9 : ffffd9e776239854
[ 447.821514] x8 : ffff205f2b5e0000 x7 : ffff205f2b8a0000 x6 : 0000000000027240
[ 447.828615] x5 : ffff00af3faf39b0 x4 : 0000000000000000 x3 : 0000000000000027
[ 447.835717] x2 : 0000000000000023 x1 : 16e32808cb35d300 x0 : 0000000000000000
[ 447.842818] Call trace:
[ 447.845252] mlx5_ib_dev_res_cleanup+0x118/0x150 [mlx5_ib]
[ 447.850719] mlx5r_remove+0x54/0x80 [mlx5_ib]
[ 447.855063] auxiliary_bus_remove+0x30/0x4c
[ 447.859227] __device_release_driver+0x190/0x23c
[ 447.863825] device_release_driver+0x38/0x50
[ 447.868076] bus_remove_device+0x130/0x140
[ 447.872154] device_del+0x184/0x434
[ 447.875628] delete_drivers+0x54/0xe0
[ 447.879274] mlx5_unregister_device+0x40/0x80
[ 447.883611] mlx5_uninit_one+0x34/0xd4
[ 447.887343] remove_one+0x4c/0xd0
[ 447.890644] pci_device_remove+0x48/0xe0
[ 447.894549] __device_release_driver+0x190/0x23c
[ 447.899146] device_release_driver+0x38/0x50
[ 447.903396] pci_stop_bus_device+0x8c/0xd0
[ 447.907473] pci_stop_and_remove_bus_device+0x24/0x40
[ 447.912501] pci_iov_remove_virtfn+0xb8/0x130
[ 447.916838] pci_reset_iov_state+0x5c/0xb0
[ 447.920916] pcie_flr+0x38/0x130
[ 447.924131] pcie_reset_flr+0x40/0x54
[ 447.927777] __pci_reset_function_locked+0x54/0x80
[ 447.932547] pci_reset_function+0x4c/0x90
[ 447.936538] reset_store+0x70/0xc0
[ 447.939924] dev_attr_store+0x24/0x40
[ 447.943570] sysfs_kf_write+0x50/0x60
[ 447.947216] kernfs_fop_write_iter+0x124/0x1b4
[ 447.951640] new_sync_write+0xf0/0x190
[ 447.955373] vfs_write+0x25c/0x2c0
[ 447.958760] ksys_write+0x78/0x104
[ 447.962147] __arm64_sys_write+0x28/0x3c
[ 447.966052] invoke_syscall+0x50/0x120
[ 447.969784] el0_svc_common.constprop.0+0x188/0x190
[ 447.974639] do_el0_svc+0x30/0x90
[ 447.977939] el0_svc+0x20/0x90
[ 447.980980] el0t_64_sync_handler+0x1a8/0x1ac
[ 447.985316] el0t_64_sync+0x1a0/0x1a4
[ 447.988961] ---[ end trace d334416dff5120e7 ]---
[ 447.993669] ------------[ cut here ]------------
[ 447.998264] WARNING: CPU: 37 PID: 4328 at drivers/infiniband/core/verbs.c:347 ib_dealloc_pd_user+0x94/0x9c
[ 448.007872] Modules linked in: bluetooth rfkill xt_addrtype iptable_filter xt_conntrack overlay dm_mod ib_isert iscsi_target_mod rpcrdma ib_umad ib_iser ib_ipoib libiscsi scsi_transport_iscsi mlx5_ib hns_roce_hw_v2 hisi_hpre hisi_sec2 sbsa_gwdt hisi_zip hisi_trng_v2 arm_spe_pmu hisi_qm hisi_uncore_l3c_pmu hisi_uncore_hha_pmu uacce hisi_uncore_ddrc_pmu crct10dif_ce rng_core spi_dw_mmio hisi_uncore_pmu hns3 hclge hisi_sas_v3_hw hnae3 hisi_sas_main libsas
[ 448.047740] CPU: 37 PID: 4328 Comm: bash Kdump: loaded Tainted: G W 5.16.0-pcie-iov+ #14
[ 448.057088] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
[ 448.065917] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 448.072845] pc : ib_dealloc_pd_user+0x94/0x9c
[ 448.077183] lr : ib_dealloc_pd_user+0x3c/0x9c
[ 448.081519] sp : ffff80006172b7c0
[ 448.084819] x29: ffff80006172b7c0 x28: ffff008009880000 x27: 0000000000000000
[ 448.091922] x26: ffff00800e8c00d0 x25: 0000000000000000 x24: ffff008009880000
[ 448.099025] x23: 0000000000000080 x22: ffffd9e70116cfd0 x21: ffffd9e701163d28
[ 448.106128] x20: 0000000000000000 x19: ffff00414927d080 x18: 0000000000000030
[ 448.113230] x17: 657266206f742064 x16: ffffd9e77641d550 x15: 0000000000000000
[ 448.120333] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 448.127435] x11: 0000000000000000 x10: 0000000000000bb0 x9 : ffffd9e77641d820
[ 448.134537] x8 : ffff008009880c10 x7 : 0000000000000000 x6 : 0000000000000001
[ 448.141640] x5 : ffffd9e701141da8 x4 : 0000000000000000 x3 : ffff0041396fe400
[ 448.148743] x2 : 16e32808cb35d300 x1 : 0000000000000000 x0 : 0000000000000002
[ 448.155846] Call trace:
[ 448.158280] ib_dealloc_pd_user+0x94/0x9c
[ 448.162271] mlx5_ib_dev_res_cleanup+0x90/0x150 [mlx5_ib]
[ 448.167654] mlx5r_remove+0x54/0x80 [mlx5_ib]
[ 448.171997] auxiliary_bus_remove+0x30/0x4c
[ 448.176161] __device_release_driver+0x190/0x23c
[ 448.180756] device_release_driver+0x38/0x50
[ 448.185007] bus_remove_device+0x130/0x140
[ 448.189085] device_del+0x184/0x434
[ 448.192558] delete_drivers+0x54/0xe0
[ 448.196204] mlx5_unregister_device+0x40/0x80
[ 448.200541] mlx5_uninit_one+0x34/0xd4
[ 448.204273] remove_one+0x4c/0xd0
[ 448.207574] pci_device_remove+0x48/0xe0
[ 448.211480] __device_release_driver+0x190/0x23c
[ 448.216075] device_release_driver+0x38/0x50
[ 448.220326] pci_stop_bus_device+0x8c/0xd0
[ 448.224404] pci_stop_and_remove_bus_device+0x24/0x40
[ 448.229431] pci_iov_remove_virtfn+0xb8/0x130
[ 448.233769] pci_reset_iov_state+0x5c/0xb0
[ 448.237846] pcie_flr+0x38/0x130
[ 448.241060] pcie_reset_flr+0x40/0x54
[ 448.244706] __pci_reset_function_locked+0x54/0x80
[ 448.249475] pci_reset_function+0x4c/0x90
[ 448.253467] reset_store+0x70/0xc0
[ 448.256853] dev_attr_store+0x24/0x40
[ 448.260499] sysfs_kf_write+0x50/0x60
[ 448.264146] kernfs_fop_write_iter+0x124/0x1b4
[ 448.268569] new_sync_write+0xf0/0x190
[ 448.272301] vfs_write+0x25c/0x2c0
[ 448.275688] ksys_write+0x78/0x104
[ 448.279075] __arm64_sys_write+0x28/0x3c
[ 448.282980] invoke_syscall+0x50/0x120
[ 448.286714] el0_svc_common.constprop.0+0x188/0x190
[ 448.291571] do_el0_svc+0x30/0x90
[ 448.294871] el0_svc+0x20/0x90
[ 448.297911] el0t_64_sync_handler+0x1a8/0x1ac
[ 448.302249] el0t_64_sync+0x1a0/0x1a4
[ 448.305894] ---[ end trace d334416dff5120e8 ]---
[ 448.363136] restrack: ------------[ cut here ]------------
[ 448.368601] infiniband mlx5_2: BUG: RESTRACK detected leak of resources
[ 448.375187] restrack: Kernel PD object allocated by mlx5_ib is not freed
[ 448.381861] restrack: Kernel PD object allocated by ib_core is not freed
[ 448.388534] restrack: Kernel PD object allocated by mlx5_ib is not freed
[ 448.395207] restrack: Kernel CQ object allocated by mlx5_ib is not freed
[ 448.401879] restrack: Kernel SRQ object allocated by mlx5_ib is not freed
[ 448.408638] restrack: Kernel SRQ object allocated by mlx5_ib is not freed
[ 448.415401] restrack: ------------[ cut here ]------------
[ 448.455025] mlx5_core 0000:01:00.0: poll_health:795:(pid 0): Fatal error 1 detected
[ 448.455107] pcieport 0000:00:00.0: AER: Corrected error received: 0000:01:00.0
[ 448.469914] mlx5_core 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[ 448.479792] mlx5_core 0000:01:00.0: device [15b3:1019] error status/mask=00002000/00000000
[ 448.488196] mlx5_core 0000:01:00.0: [13] NonFatalErr
[ 448.494415] pcieport 0000:00:00.0: AER: Multiple Corrected error received: 0000:01:00.0
[ 448.502452] mlx5_core 0000:01:00.1: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[ 448.512324] mlx5_core 0000:01:00.1: device [15b3:1019] error status/mask=00002000/00000000
[ 448.520726] mlx5_core 0000:01:00.1: [13] NonFatalErr
[ 448.526951] pcieport 0000:00:00.0: AER: Corrected error received: 0000:01:00.0
[ 448.534176] pcieport 0000:00:00.0: AER: Multiple Corrected error received: 0000:01:00.0
[ 448.619235] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 8192 of flow group id 19
[ 448.630750] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 21 of ft 262149
[ 448.641277] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 20 of ft 262149
[ 448.651794] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 19 of ft 262149
[ 448.662309] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 18 of ft 262149
[ 448.672830] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 17 of ft 262149
[ 448.683417] mlx5_core 0000:01:00.2: update_root_ft_destroy:2127:(pid 4328): Update root flow table of id(262149) qpn(0) failed
[ 448.694843] mlx5_core 0000:01:00.2: del_hw_flow_table:507:(pid 4328): flow steering can't destroy ft
[ 448.703993] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 16 of ft 262148
[ 448.714516] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 15 of ft 262148
[ 448.725033] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 14 of ft 262148
[ 448.735564] mlx5_core 0000:01:00.2: del_hw_flow_table:507:(pid 4328): flow steering can't destroy ft
[ 448.744714] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 0 of flow group id 11
[ 448.755936] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 1 of flow group id 11
[ 448.767145] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 2 of flow group id 11
[ 448.778352] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 3 of flow group id 11
[ 448.789558] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 4 of flow group id 11
[ 448.800770] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 5 of flow group id 11
[ 448.812049] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 6 of flow group id 11
[ 448.823261] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 7 of flow group id 11
[ 448.834471] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 14 of flow group id 12
[ 448.845774] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 15 of flow group id 12
[ 448.857072] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 16 of flow group id 13
[ 448.868370] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 8 of flow group id 11
[ 448.879579] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 9 of flow group id 11
[ 448.890788] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 10 of flow group id 11
[ 448.902087] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 11 of flow group id 11
[ 448.913382] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 12 of flow group id 11
[ 448.924675] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 13 of flow group id 11
[ 448.935980] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 13 of ft 2
[ 448.946072] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 12 of ft 2
[ 448.956160] mlx5_core 0000:01:00.2: del_hw_flow_group:644:(pid 4328): flow steering can't destroy fg 11 of ft 2
[ 448.966249] mlx5_core 0000:01:00.2: del_hw_flow_table:507:(pid 4328): flow steering can't destroy ft
[ 448.975395] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 0 of flow group id 8
[ 448.986526] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 1 of flow group id 8
[ 448.997647] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 2 of flow group id 8
[ 449.008768] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 3 of flow group id 8
[ 449.019890] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 4 of flow group id 8
[ 449.031013] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 5 of flow group id 8
[ 449.042135] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 6 of flow group id 8
[ 449.053257] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 7 of flow group id 8
[ 449.064380] mlx5_core 0000:01:00.2: del_hw_fte:605:(pid 4328): flow steering can't delete fte in index 8 of flow group id 9
[...]
> pcibios_sriov_disable(dev);
>
> if (iov->link != dev->devfn)
> sysfs_remove_link(&dev->dev.kobj, "dep_link");
>
> iov->num_VFs = 0;
>
> if (!flr)
> pci_iov_set_numvfs(dev, 0);
> }
>
> Whether this is better, I leave to your evaluation.
>
> Thanks,
> Lukasz
>
>> Did you test with or without my patch?
>>
>> Here is the result with my patch for the NVMe device in QEMU:
>>
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# lspci -s 01:
>> 01:00.0 Non-Volatile memory controller: Red Hat, Inc. Device 0010 (rev 02)
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# lspci -vvv -s 01:00.0 | egrep "IOV|VF"
>> Capabilities: [120 v1] Single Root I/O Virtualization (SR-IOV)
>> IOVCap: Migration-, Interrupt Message Number: 000
>> IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
>> IOVSta: Migration-
>> Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
>> VF offset: 1, stride: 1, Device ID: 0010
>> VF Migration: offset: 00000000, BIR: 0
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# echo 1 > sriov_numvfs
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# lspci -vvv -s 01:00.0 | egrep "IOV|VF"
>> Capabilities: [120 v1] Single Root I/O Virtualization (SR-IOV)
>> IOVCap: Migration-, Interrupt Message Number: 000
>> IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
>> IOVSta: Migration-
>> Initial VFs: 8, Total VFs: 8, Number of VFs: 1, Function Dependency Link: 00
>> VF offset: 1, stride: 1, Device ID: 0010
>> VF Migration: offset: 00000000, BIR: 0
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# echo 1 > reset
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# lspci -vvv -s 01:00.0 | egrep "IOV|VF"
>> Capabilities: [120 v1] Single Root I/O Virtualization (SR-IOV)
>> IOVCap: Migration-, Interrupt Message Number: 000
>> IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
>> IOVSta: Migration-
>> Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
>> VF offset: 1, stride: 1, Device ID: 0010
>> VF Migration: offset: 00000000, BIR: 0
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# lspci -xxx -s 01:00.0
>> 01:00.0 Non-Volatile memory controller: Red Hat, Inc. Device 0010 (rev 02)
>> 00: 36 1b 10 00 07 05 10 00 02 02 08 01 00 00 00 00
>> 10: 04 00 80 fe 00 00 00 00 00 00 00 00 00 00 00 00
>> 20: 00 00 00 00 00 00 00 00 00 00 00 00 f4 1a 00 11
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00
>> 40: 11 80 40 80 00 20 00 00 00 30 00 00 00 00 00 00
>> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> 60: 01 00 03 00 08 00 00 00 00 00 00 00 00 00 00 00
>> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> 80: 10 60 02 00 00 80 00 10 00 00 00 00 11 04 00 00
>> 90: 00 00 11 00 00 00 00 00 00 00 00 00 00 00 00 00
>> a0: 00 00 00 00 00 00 30 00 00 00 00 00 00 00 00 00
>> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>
>> root@qemu-sriov:/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0# cat reset_method
>> flr bus
>>
>>>
>>> [root@localhost ~]# lspci -s 01:
>>> 01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
>>> 01:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
>>> [root@localhost ~]# lspci -vvv -s 01:00.0 | egrep "IOV|VF"
>>> Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
>>> IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
>>> IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
>>> IOVSta: Migration-
>>> Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
>>> VF offset: 2, stride: 1, Device ID: 101a
>>> VF Migration: offset: 00000000, BIR: 0
>>> [root@localhost 0000:01:00.0]# echo 1 > sriov_numvfs
>>> [root@localhost ~]# lspci -vvv -s 01:00.0 | egrep "IOV|VF"
>>> Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
>>> IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
>>> IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+ 10BitTagReq-
>>> IOVSta: Migration-
>>> Initial VFs: 16, Total VFs: 16, Number of VFs: 1, Function Dependency Link: 00
>>> VF offset: 2, stride: 1, Device ID: 101a
>>> VF Migration: offset: 00000000, BIR: 0
>>> [root@localhost 0000:01:00.0]# echo 1 > reset
>>> [root@localhost ~]# lspci -vvv -s 01:00.0 | egrep "IOV|VF"
>>> Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
>>> IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
>>> IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+ 10BitTagReq-
>>> IOVSta: Migration-
>>> Initial VFs: 16, Total VFs: 16, Number of VFs: 1, Function Dependency Link: 00
>>> VF offset: 2, stride: 1, Device ID: 101a
>>> VF Migration: offset: 00000000, BIR: 0
>>> [root@localhost ~]# lspci -xxx -s 01:00.0
>>> 01:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
>>> 00: b3 15 19 10 46 05 10 00 00 00 00 02 08 00 80 00
>>> 10: 0c 00 00 00 00 08 00 00 00 00 00 00 00 00 00 00
>>> 20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 08 00
>>> 30: 00 00 70 e6 60 00 00 00 00 00 00 00 ff 01 00 00
>>> 40: 01 00 c3 81 08 00 00 00 03 9c cc 80 00 78 00 00
>>> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 20 00 01
>>> 60: 10 48 02 00 e2 8f e0 11 5f 29 00 00 04 71 41 00
>>> 70: 08 00 04 11 00 00 00 00 00 00 00 00 00 00 00 00
>>> 80: 00 00 00 00 17 00 01 00 40 00 00 00 1e 00 80 01
>>> 90: 04 00 1e 00 00 00 00 00 00 00 00 00 11 c0 3f 80
>>> a0: 00 20 00 00 00 30 00 00 00 00 00 00 00 00 00 00
>>> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> c0: 09 40 18 00 0a 00 00 20 f0 1a 00 00 00 00 00 00
>>> d0: 20 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00
>>> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> [root@localhost 0000:01:00.0]# cat reset_method
>>> flr bus
>>>
>>> On 2022/1/19 10:47, Yicong Yang wrote:
>>>> On 2022/1/19 0:30, Lukasz Maniak wrote:
>>>>> On Tue, Jan 18, 2022 at 07:07:23PM +0800, Yicong Yang wrote:
>>>>>> On 2022/1/18 6:55, Bjorn Helgaas wrote:
>>>>>>> [+cc Alex in case he has comments on how FLR should work on
>>>>>>> non-conforming hns3 devices]
>>>>>>>
>>>>>>> On Sat, Jan 15, 2022 at 05:22:19PM +0800, Yicong Yang wrote:
>>>>>>>> On 2022/1/15 0:37, Bjorn Helgaas wrote:
>>>>>>>>> On Fri, Jan 14, 2022 at 05:42:48PM +0800, Yicong Yang wrote:
>>>>>>>>>> On 2022/1/14 0:45, Lukasz Maniak wrote:
>>>>>>>>>>> On Wed, Jan 12, 2022 at 08:49:03AM -0600, Bjorn Helgaas wrote:
>>>>>>>>>>>> On Wed, Dec 22, 2021 at 08:19:57PM +0100, Lukasz Maniak wrote:
>>>>>>>>>>>>> As per PCI Express specification, FLR to a PF resets the PF state as
>>>>>>>>>>>>> well as the SR-IOV extended capability including VF Enable which means
>>>>>>>>>>>>> that VFs no longer exist.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you add a specific reference to the spec, please?
>>>>>>>>>>>>
>>>>>>>>>>> Following the Single Root I/O Virtualization and Sharing Specification:
>>>>>>>>>>> 2.2.3. FLR That Targets a PF
>>>>>>>>>>> PFs must support FLR.
>>>>>>>>>>> FLR to a PF resets the PF state as well as the SR-IOV extended
>>>>>>>>>>> capability including VF Enable which means that VFs no longer exist.
>>>>>>>>>>>
>>>>>>>>>>> For PCI Express Base Specification Revision 5.0 and later, this is
>>>>>>>>>>> section 9.2.2.3.
>>>>>>>>>
>>>>>>>>> This is also the section in the new PCIe r6.0. Let's use that.
>>>>>>>>>
>>>>>>>>>>>>> Currently, the IOV state is not updated during FLR, resulting in
>>>>>>>>>>>>> non-compliant PCI driver behavior.
>>>>>>>>>>>>
>>>>>>>>>>>> And include a little detail about what problem is observed? How would
>>>>>>>>>>>> a user know this problem is occurring?
>>>>>>>>>>>>
>>>>>>>>>>> The problem is that the state of the kernel and HW as to the number of
>>>>>>>>>>> VFs gets out of sync after FLR.
>>>>>>>>>>>
>>>>>>>>>>> This results in further listing, after the FLR is performed by the HW,
>>>>>>>>>>> of VFs that actually no longer exist and should no longer be reported on
>>>>>>>>>>> the PCI bus. lspci return FFs for these VFs.
>>>>>>>>>>
>>>>>>>>>> There're some exceptions. Take HiSilicon's hns3 and sec device as an
>>>>>>>>>> example, the VF won't be destroyed after the FLR reset.
>>>>>>>>>
>>>>>>>>> If FLR on an hns3 PF does *not* clear VF Enable, and the VFs still
>>>>>>>>> exist after FLR, isn't that a violation of sec 9.2.2.3?
>>>>>>>>
>>>>>>>> yes I think it's a violation to the spec.
>>>>>>>
>>>>>>> Thanks for confirming that.
>>>>>>>
>>>>>>>>> If hns3 and sec don't conform to the spec, we should have some sort of
>>>>>>>>> quirk that serves to document and work around this.
>>>>>>>>
>>>>>>>> ok I think it'll help. Do you mean something like this based on this patch:
>>>>>>>>
>>>>>>>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>>>>>>>> index 69ee321027b4..0e4976c669b2 100644
>>>>>>>> --- a/drivers/pci/iov.c
>>>>>>>> +++ b/drivers/pci/iov.c
>>>>>>>> @@ -1025,6 +1025,8 @@ void pci_reset_iov_state(struct pci_dev *dev)
>>>>>>>> return;
>>>>>>>> if (!iov->num_VFs)
>>>>>>>> return;
>>>>>>>> + if (dev->flr_no_vf_reset)
>>>>>>>> + return;
>>>>>>>>
>>>>>>>> sriov_del_vfs(dev);
>>>>>>>>
>>>>>>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
>>>>>>>> index 003950c738d2..c8ffcb0ac612 100644
>>>>>>>> --- a/drivers/pci/quirks.c
>>>>>>>> +++ b/drivers/pci/quirks.c
>>>>>>>> @@ -1860,6 +1860,17 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_HUAWEI, 0xa256, quirk_huawei_pcie_sva);
>>>>>>>> DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_HUAWEI, 0xa258, quirk_huawei_pcie_sva);
>>>>>>>> DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_HUAWEI, 0xa259, quirk_huawei_pcie_sva);
>>>>>>>>
>>>>>>>> +/*
>>>>>>>> + * Some HiSilicon PCIe devices' VF won't be destroyed after a FLR reset.
>>>>>>>> + * Don't reset these devices' IOV state when doing FLR.
>>>>>>>> + */
>>>>>>>> +static void quirk_huawei_pcie_flr(struct pci_dev *pdev)
>>>>>>>> +{
>>>>>>>> + pdev->flr_no_vf_reset = 1;
>>>>>>>> +}
>>>>>>>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_HUAWEI, 0xa255, quirk_huawei_pcie_flr);
>>>>>>>> +/* ...some other devices have this quirk */
>>>>>>>
>>>>>>> Yes, I think something along this line will help.
>>>>>>>
>>>>>>>> diff --git a/include/linux/pci.h b/include/linux/pci.h
>>>>>>>> index 18a75c8e615c..e62f9fa4d48f 100644
>>>>>>>> --- a/include/linux/pci.h
>>>>>>>> +++ b/include/linux/pci.h
>>>>>>>> @@ -454,6 +454,7 @@ struct pci_dev {
>>>>>>>> unsigned int is_probed:1; /* Device probing in progress */
>>>>>>>> unsigned int link_active_reporting:1;/* Device capable of reporting link active */
>>>>>>>> unsigned int no_vf_scan:1; /* Don't scan for VFs after IOV enablement */
>>>>>>>> + unsigned int flr_no_vf_reset:1; /* VF won't be destroyed after PF's FLR */
>>>>>>>>
>>>>>>>>>> Currently the transactions with the VF will be restored after the
>>>>>>>>>> FLR. But this patch will break that, the VF is fully disabled and
>>>>>>>>>> the transaction cannot be restored. User needs to reconfigure it,
>>>>>>>>>> which is unnecessary before this patch.
>>>>>>>>>
>>>>>>>>> What does it mean for a "transaction to be restored"? Maybe you mean
>>>>>>>>> this patch removes the *VFs* via sriov_del_vfs(), and whoever
>>>>>>>>> initiated the FLR would need to re-enable VFs via pci_enable_sriov()
>>>>>>>>> or something similar?
>>>>>>>>
>>>>>>>> Partly. It'll also terminate the VF users.
>>>>>>>> Think that I attach the VF of hns to a VM by vfio and ping the network
>>>>>>>> in the VM, when doing FLR the 'ping' will pause and after FLR it'll
>>>>>>>> resume. Currenlty The driver handle this in the ->reset_{prepare, done}()
>>>>>>>> methods. The user of VM may not realize there is a FLR of the PF as the
>>>>>>>> VF always exists and the 'ping' is never terminated.
>>>>>>>>
>>>>>>>> If we remove the VF when doing FLR, then 1) we'll block in the VF->remove()
>>>>>>>> until no one is using the device, for example the 'ping' is finished.
>>>>>>>> 2) the VF in the VM no longer exists and we have to re-enable VF and hotplug
>>>>>>>> it into the VM and restart the ping. That's a big difference.
>>>>>>>>
>>>>>>>>> If FLR disables VFs, it seems like we should expect to have to
>>>>>>>>> re-enable them if we want them.
>>>>>>>>
>>>>>>>> It involves a remove()/probe() process of the VF driver and the user
>>>>>>>> of the VF will be terminated, just like the situation illustrated
>>>>>>>> above.
>>>>>>>
>>>>>>> I think users of FLR should be able to rely on it working per spec,
>>>>>>> i.e., that VFs will be destroyed. If hardware like hns3 doesn't do
>>>>>>> that, the quirk should work around that in software by doing it
>>>>>>> explicitly.
>>>>>>>
>>>>>>> I don't think the non-standard behavior should be exposed to the
>>>>>>> users. The user should not have to know about this hns3 issue.
>>>>>>>
>>>>>>> If FLR on a standard NIC terminates a ping on a VF, FLR on an hns3 NIC
>>>>>>> should also terminate a ping on a VF.
>>>>>>>
>>>>>>
>>>>>> ok thanks for the discussion, agree on that. According to the spec, after
>>>>>> the FLR to the PF the VF does not exist anymore, so the ping will be terminated.
>>>>>> Our hns3 and sec team are still evaluating it before coming to a solution of
>>>>>> whether using a quirk or comform to the spec.
>>>>>>
>>>>>> For this patch it looks reasonable to me, but some questions about the code below.
>>>>>>
>>>>>>>>>> Can we handle this problem in another way? Maybe test the VF's
>>>>>>>>>> vendor device ID after the FLR reset to see whether it has really
>>>>>>>>>> gone or not?
>>>>>>>>>>
>>>>>>>>>>> sriov_numvfs in sysfs returns old invalid value and does not allow
>>>>>>>>>>> setting a new value before explicitly setting 0 in the first place.
>>>>>>>>>>>
>>>>>>>>>>>>> This patch introduces a simple function, called on the FLR path, that
>>>>>>>>>>>>> removes the virtual function devices from the PCI bus and their
>>>>>>>>>>>>> corresponding sysfs links with a final clear of the num_vfs value in IOV
>>>>>>>>>>>>> state.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Lukasz Maniak <lukasz.maniak@xxxxxxxxxxxxxxx>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> drivers/pci/iov.c | 21 +++++++++++++++++++++
>>>>>>>>>>>>> drivers/pci/pci.c | 2 ++
>>>>>>>>>>>>> drivers/pci/pci.h | 4 ++++
>>>>>>>>>>>>> 3 files changed, 27 insertions(+)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>>>>>>>>>>>>> index 0267977c9f17..69ee321027b4 100644
>>>>>>>>>>>>> --- a/drivers/pci/iov.c
>>>>>>>>>>>>> +++ b/drivers/pci/iov.c
>>>>>>>>>>>>> @@ -1013,6 +1013,27 @@ int pci_iov_bus_range(struct pci_bus *bus)
>>>>>>>>>>>>> return max ? max - bus->number : 0;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * pci_reset_iov_state - reset the state of the IOV capability
>>>>>>>>>>>>> + * @dev: the PCI device
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +void pci_reset_iov_state(struct pci_dev *dev)
>>>>>>>>>>>>> +{
>>>>>>>>>>>>> + struct pci_sriov *iov = dev->sriov;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> + if (!dev->is_physfn)
>>>>>>>>>>>>> + return;
>>>>>>>>>>>>> + if (!iov->num_VFs)
>>>>>>>>>>>>> + return;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> + sriov_del_vfs(dev);
>>>>>>>>>>>>> +
>>>>>>>>>>>>> + if (iov->link != dev->devfn)
>>>>>>>>>>>>> + sysfs_remove_link(&dev->dev.kobj, "dep_link");
>>>>>>>>>>>>> +
>>>>>>>>>>>>> + iov->num_VFs = 0;
>>>>>>>>>>>>> +}
>>>>>>>>>>>>> +
>>>>>>
>>>>>> Any reason for not using pci_disable_sriov()?
>>>>>
>>>>> The issue with pci_disable_sriov() is that it calls sriov_disable(),
>>>>> which directly uses pci_cfg_access_lock(), leading to deadlock on the
>>>>> FLR path.
>>>>>
>>>>
>>>> That'll be a problem. Well my main concern is whether the VFs will be reset
>>>> correctly through pci_reset_iov_state() as it lacks the participant of
>>>> PF driver and bios (seems may needed only on powerpc, not sure), which is
>>>> necessary in the enable/disable routine through $pci_dev/sriov_numvfs.
>>>>
>>>>>>
>>>>>> With the spec the related registers in the SRIOV cap will be reset so
>>>>>> it's ok in general. But for some devices not following the spec like hns3,
>>>>>> some fields like VF enable won't be reset and keep enabled after the FLR.
>>>>>> In this case after the FLR the VF devices in the system has gone but
>>>>>> the state of the PF SRIOV cap leaves uncleared. pci_disable_sriov()
>>>>>> will reset the whole SRIOV cap. It'll also call pcibios_sriov_disable()
>>>>>> to correct handle the VF disabling on some platforms, IIUC.
>>>>>>
>>>>>> Or is it better to use pdev->driver->sriov_configure(pdev,0)?
>>>>>> PF drivers must implement ->sriov_configure() for enabling/disabling
>>>>>> the VF but we totally skip the PF driver here.
>>>>>>
>>>>>> Thanks,
>>>>>> Yicong
>>>>>>
>>>>>>>>>>>>> /**
>>>>>>>>>>>>> * pci_enable_sriov - enable the SR-IOV capability
>>>>>>>>>>>>> * @dev: the PCI device
>>>>>>>>>>>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>>>>>>>>>>>> index 3d2fb394986a..535f19d37e8d 100644
>>>>>>>>>>>>> --- a/drivers/pci/pci.c
>>>>>>>>>>>>> +++ b/drivers/pci/pci.c
>>>>>>>>>>>>> @@ -4694,6 +4694,8 @@ EXPORT_SYMBOL(pci_wait_for_pending_transaction);
>>>>>>>>>>>>> */
>>>>>>>>>>>>> int pcie_flr(struct pci_dev *dev)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> + pci_reset_iov_state(dev);
>>>>>>>>>>>>> +
>>>>>>>>>>>>> if (!pci_wait_for_pending_transaction(dev))
>>>>>>>>>>>>> pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
>>>>>>>>>>>>> index 3d60cabde1a1..7bb144fbec76 100644
>>>>>>>>>>>>> --- a/drivers/pci/pci.h
>>>>>>>>>>>>> +++ b/drivers/pci/pci.h
>>>>>>>>>>>>> @@ -480,6 +480,7 @@ void pci_iov_update_resource(struct pci_dev *dev, int resno);
>>>>>>>>>>>>> resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno);
>>>>>>>>>>>>> void pci_restore_iov_state(struct pci_dev *dev);
>>>>>>>>>>>>> int pci_iov_bus_range(struct pci_bus *bus);
>>>>>>>>>>>>> +void pci_reset_iov_state(struct pci_dev *dev);
>>>>>>>>>>>>> extern const struct attribute_group sriov_pf_dev_attr_group;
>>>>>>>>>>>>> extern const struct attribute_group sriov_vf_dev_attr_group;
>>>>>>>>>>>>> #else
>>>>>>>>>>>>> @@ -501,6 +502,9 @@ static inline int pci_iov_bus_range(struct pci_bus *bus)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> return 0;
>>>>>>>>>>>>> }
>>>>>>>>>>>>> +static inline void pci_reset_iov_state(struct pci_dev *dev)
>>>>>>>>>>>>> +{
>>>>>>>>>>>>> +}
>>>>>>>>>>>>>
>>>>>>>>>>>>> #endif /* CONFIG_PCI_IOV */
>>>>>>> .
>>>>>>>
>>>>> .
>>>>>
>>>> .
>>>>
> .
>
