I hit the same problem and have worked with the developers on a patch. You might give this a try if you are building your own kernel/driver. https://www.spinics.net/lists/linux-pci/msg92395.html -----Original Message----- From: Haeuptle, Michael <michael.haeuptle@xxxxxxx> Sent: Tuesday, March 24, 2020 10:37 AM To: Hoyer, David <David.Hoyer@xxxxxxxxxx>; linux-pci@xxxxxxxxxxxxxxx Cc: michaelhaeuptle@xxxxxxxxx Subject: RE: Deadlock during PCIe hot remove NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi David, yes, it does. -- Michael -----Original Message----- From: Hoyer, David [mailto:David.Hoyer@xxxxxxxxxx] Sent: Tuesday, March 24, 2020 9:35 AM To: Haeuptle, Michael <michael.haeuptle@xxxxxxx>; linux-pci@xxxxxxxxxxxxxxx Cc: michaelhaeuptle@xxxxxxxxx Subject: RE: Deadlock during PCIe hot remove You mentioned that you are using the latest pciehp code. Does this code contain the recently introduced ist_running flag? -----Original Message----- From: linux-pci-owner@xxxxxxxxxxxxxxx <linux-pci-owner@xxxxxxxxxxxxxxx> On Behalf Of Haeuptle, Michael Sent: Tuesday, March 24, 2020 10:22 AM To: linux-pci@xxxxxxxxxxxxxxx Cc: michaelhaeuptle@xxxxxxxxx Subject: RE: Deadlock during PCIe hot remove NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. From: Haeuptle, Michael Sent: Tuesday, March 24, 2020 8:46 AM To: 'linux-pci@xxxxxxxxxxxxxxx' <linux-pci@xxxxxxxxxxxxxxx> Cc: 'michaelhaeuptle@xxxxxxxxx' <michaelhaeuptle@xxxxxxxxx> Subject: Deadlock during PCIe hot remove Dear PCI maintainers, I'm running into a deadlock scenario between the hotplug, pcie and vfio_pci driver when removing multiple devices in parallel. This is happening on CentOS8 (4.18) with SPDK (spdk.io). I'm using the latest pciehp code, the rest is all 4.18. The sequence that leads to the deadlock is as follows: The pciehp_ist() takes the reset_lock early in its processing. While the pciehp_ist processing is progressing, vfio_pci calls pci_try_reset_function() as part of vfio_pci_release or open. The pci_try_reset_function() takes the device lock. Eventually, pci_try_reset_function() calls pci_reset_hotplug_slot() which calls pciehp_reset_slot(). The pciehp_reset_slot() tries to take the reset_lock but has to wait since it is already taken by pciehp_ist(). Eventually pciehp_ist calls pcie_stop_device() which calls device_release_driver_internal(). This function also tries to take device_lock causing the dead lock. Here's the kernel stack trace when the deadlock occurs: [root@localhost ~]# cat /proc/8594/task/8598/stack [<0>] pciehp_reset_slot+0xa5/0x220 [<0>] pci_reset_hotplug_slot.cold.72+0x20/0x36 [<0>] pci_dev_reset_slot_function+0x72/0x9b [<0>] __pci_reset_function_locked+0x15b/0x190 [<0>] pci_try_reset_function.cold.77+0x9b/0x108 [<0>] vfio_pci_disable+0x261/0x280 [<0>] vfio_pci_release+0xcb/0xf0 [<0>] vfio_device_fops_release+0x1e/0x40 [<0>] __fput+0xa5/0x1d0 [<0>] task_work_run+0x8a/0xb0 [<0>] exit_to_usermode_loop+0xd3/0xe0 [<0>] do_syscall_64+0xe1/0x100 [<0>] entry_SYSCALL_64_after_hwframe+0x65/0xca [<0>] 0xffffffffffffffff I was wondering if there's a quick workaround. I think the pci_try_reset_function would need to take the reset_lock before the device lock but there doesn't seem to be a good way of doing that. I'm also trying to see if we can delay calling the vfio functions that are initiated by SPDK but I think this inherent race should be addressed. I'm also happy to submit a defect report if this emailing list is not appropriate. * Michael
