On Mon, 3 Jun 2019 14:17:36 -0700 JD Zheng <jiandong.zheng@xxxxxxxxxxxx> wrote: > On 6/2/19 6:40 PM, Alex Williamson wrote: > > On Sun, 2 Jun 2019 19:44:14 -0500 > > Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > > >> [+cc Alex, Lukas] > >> > >> On Fri, May 31, 2019 at 09:55:20AM -0700, JD Zheng wrote: > >>> Hello, > >>> > >>> I am running DPDK 18.11+SPDK 19.04 with v5.1 kernel. DPDK/SPDK uses SSD vfio > >>> devices and after running SPDK's nvmf_tgt, unplugging a SSD cause kernel to > >>> print out following: > >>> [ 105.426952] vfio-pci 0000:04:00.0: not ready 2047ms after FLR; waiting > >>> [ 107.698953] vfio-pci 0000:04:00.0: not ready 4095ms after FLR; waiting > >>> [ 112.050960] vfio-pci 0000:04:00.0: not ready 8191ms after FLR; waiting > >>> [ 120.498953] vfio-pci 0000:04:00.0: not ready 16383ms after FLR; waiting > >>> [ 138.418957] vfio-pci 0000:04:00.0: not ready 32767ms after FLR; waiting > >>> [ 173.234953] vfio-pci 0000:04:00.0: not ready 65535ms after FLR; giving up > >>> > >>> Looks like it is a PCI hotplug racing condition between DPDK's > >>> eal-intr-thread thread and kernel's pciehp thread. And it causes lockup in > >>> pci_dev_wait() at kernel side. > >>> > >>> When SSD is removed, eal-intr-thread immediately receives > >>> RTE_INTR_HANDLE_ALARM and handler calls rte_pci_detach_dev() and at kernel > >>> side vfio_pci_release() is triggered to release this vfio device, which > >>> calls pci_try_reset_function(), then _pci_reset_function_locked(). > >>> pci_try_reset_function acquires the device lock but > >>> _pci_reset_function_locked() doesn't return, therefore lock is NOT released. > > > > To what extent does vfio-pci need to learn about surprise hotplug? My > > expectation is that the current state of the code would only support > > cooperative hotplug. When a device is surprise removed, what backs a > > user's mmaps? AIUI, we don't have a revoke interface to invalidate > > these. We should probably start with an RFE or some development effort > > to harden vfio-pci for surprise hotplug, it's not surprising it doesn't > > just work TBH. Thanks, > > > > Alex > > > > > > I did see other issues that DPDK unmap vifo device memory was stuck due > to surprise removal. > > Are you saying that vfio-pci surprise removal is not fully implemented yet? Has not even been evaluated for that use case afaik. You are in unknown and expected broken territory. If this is an area you'd like to contribute, great, but expect more than a bug fix here and there. Thanks, Alex
