On Wed, Aug 09, 2023 at 06:05:18PM -0500, Bjorn Helgaas wrote:
> On Mon, Apr 10, 2023 at 08:34:11PM +0800, Wu Zongyong wrote:
> > NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4
> > card is direct attached to a Root Port only. So avoid bus reset by
> > marking T4 GPUs PCI_DEV_FLAGS_NO_BUS_RESET.
> >
> > Fixes: 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")
> > Signed-off-by: Wu Zongyong <wuzongyong@xxxxxxxxxxxxxxxxx>
>
> Applied to pci/virtualization for v6.6, thanks!
I talk about the issue with NVIDIA, and they think the issue is probably related
the pci link instead of the T4 GPU card.
I will try to describe the issue I met in detail.
The T4 card which is direct attached to a Root Port and I rebind it to
vfio-pci driver. Then I try to use to call some vfio-related api and the
ioctl VFIO_GROUP_GET_DEVICE_FD failed.
The stack is (base on kernel v5.10):
vfio_group_fops_unl_ioctl
vfio_group_get_device_fd
vfio_pci_open
vfio_pci_enable // return value is -19
pci_try_reset_function
__pci_reset_function_locked
After the __pci_reset_function_locked(), the dmesg shows:
[12207494.508467] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Down
[12207494.508535] vfio-pci 0000:40:00.0: No device request channel registered, blocked until released by user
[12207494.518426] pci 0000:40:00.0: Removing from iommu group 84
[12207495.532365] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Card present
[12207495.532367] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Up
NVIDIA people thinks this root port is not going through this reset logic and getting the
link down/hot plug interrupts[1].
Can you revert the patch I sent and maybe we should dig it deeply.
>
> > ---
> > drivers/pci/quirks.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 44cab813bf95..be86b93b9e38 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
> > */
> > static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
> > {
> > - if ((dev->device & 0xffc0) == 0x2340)
> > + if ((dev->device & 0xffc0) == 0x2340 || dev->device == 0x1eb8)
> > quirk_no_bus_reset(dev);
> > }
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
> > --
> > 2.34.3
> >
