Hi Alex,
On May 24, 2019, at 18:49, Alex Williamson <alex.williamson@xxxxxxxxxx> wrote:
> On Fri, 24 May 2019 17:31:18 +0200
> Maik Broemme <mbroemme@xxxxxxxxxx> wrote:
>
> > The Intel PCI bridge on SuperMicro Atom C3xxx motherboards do not
> > successfully complete a bus reset when used with certain child devices.
>
> What are these 'certain child devices'? We can't really regression
> test to know if/when the problem might be resolved if we don't know
> what to test.
I tried the following devices:
- Digital Devices GmbH Octopus DVB Adapter
- Digital Devices GmbH Cine S2 V6.5
- Digital Devices GmbH Cine S2 V7
- RealTek RTL8111D
- RealTek RTL8168B
- Intel I210-T1
> Do these devices reset properly in other systems?
All these cards survive VFIO reset and VM reboot cycles in another
motherboard (SuperMicro X11SPM-F). They only fail in the SuperMicro
A2SDi-*C-HLN4F series. I have a 8 core and 16 core version of this
motherboard.
I've tried a passive Mini PCI-E adapter (MikroTik RB11E) in the slot
with several wireless adapters (don't remember them all) and the
'Digital Devices GmbH Octopus DVB Adapter' which is Mini PCI-E. They all
produced the same error 'Failed to return from FLR' and '!!! Unknown
header type 7f'
Also I've tried a PCI-E switch from PLX technology, sold by MikroTik, the
RouterBoard RB14eU. It is exports 4 Mini PCI ports in one PCI-E port and
I tried it with one card and multiple cards.
All these devices start to work once I enabled the bus reset quirk. The
RB14eU even allows to assign the individual Mini PCI-E ports to
different VMs and survive independent resets behind the PLX bridge.
> Are there any devices that can do a bus reset properly on this system?
I'm pretty sure not, of course I can test only devices I own and at
least the Intel I210-T1 supports all functionality to do a proper reset.
I own these motherboards since ~2 years and tried almost any device I
had during this time.
> We'd really only want to blacklist bus reset on this root port(?) if this is
> a systemic problem with the root port, which is not clearly proven
> here. Thanks,
I can rework the patch and apply the quirk only to my tested devices but
I strongly believe that it is an issue on the root port, independent
from the device.
>
> Alex
>
> > After the reset, config accesses to the child may fail. If assigning
> > such device via VFIO it will immediately fail with:
> >
> > vfio-pci 0000:01:00.0: Failed to return from FLR
> > vfio-pci 0000:01:00.0: timed out waiting for pending transaction;
> > performing function level reset anyway
> >
> > Device will disappear from PCI device list:
> >
> > !!! Unknown header type 7f
> > Kernel driver in use: vfio-pci
> > Kernel modules: ddbridge
> >
> > The attached patch will mark the root port as incapable of doing a
> > bus level reset. After that all my tested devices survive a VFIO
> > assignment and several VM reboot cycles.
> >
> > Signed-off-by: Maik Broemme <mbroemme@xxxxxxxxxx>
> > ---
> > drivers/pci/quirks.c | 7 +++++++
> > 1 file changed, 7 insertions(+)
> >
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 0f16acc323c6..86cd42872708 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -3433,6 +3433,13 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034, quirk_no_bus_reset);
> > */
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_CAVIUM, 0xa100, quirk_no_bus_reset);
> >
> > +/*
> > + * Root port on some SuperMicro Atom C3xxx motherboards do not successfully
> > + * complete a bus reset when used with certain child devices. After the
> > + * reset, config accesses to the child may fail.
> > + */
> > +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x19a4, quirk_no_bus_reset);
> > +
> > static void quirk_no_pm_reset(struct pci_dev *dev)
> > {
> > /*
>
--Maik
