On Mon, 2021-09-13 at 11:07 -0500, Bjorn Helgaas wrote:
>
> 1) It sounds like this *might* be a workaround for a device defect?
> Should we infer that there's a device where:
>
> - We reset the device
> - We read PCI_COMMAND until it is not ~0, for up to 60 seconds
> - The device returns CRS status for each read, until ...
> - The platform hardware times out before 60 seconds and fails the
> transaction, causing a system crash?
Yes. As detailed in my other reply (which raced with this mail), I
implemented this patch because I encountered a device (and a platform)
that behaved exactly as described.
> But reading PCI_VENDOR_ID instead of PCI_COMMAND somehow avoids the
> platform timeout?
Correct. More specifically, that "somehow" is the CRS SV mechanism
standardized by the PCIe specification. This mechanism relies
specifically on the target offset being PCI_VENDOR_ID.
> 2) I think this should somehow be integrated with pci_bus_wait_crs(),
> which also loops looking for CRS status.
Good point. I will see if that can be incorporated in next version, or
explain why not if that turns out to be the case.
> 3) pci_bus_wait_crs() is used in the enumeration path, and we do a
> 32-bit read there, which reads both the Vendor ID and the Device
> ID. Maybe that's some sort of micro-optimization, but apparently
> there are devices that don't implement CRS SV correctly (see
> 89665a6a7140 ("PCI: Check only the Vendor ID to identify
> Configuration Request Retry"), and doing a 16-bit read would avoid
> that issue.
>
> For pci_dev_wait(), I don't think there's any point in doing a
> 32-bit read, so maybe we should just do a 16-bit read.
I agree, and will change it to a 16-bit read in next version.
> > Signed-off-by: Stanislav Spassov <stanspas@xxxxxxxxx>
> > ---
> > drivers/pci/pci.c | 55 ++++++++++++++++++++++++++++++++++++++++-------
> > 1 file changed, 47 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index 44f5d4907db6..a028147f4471 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c
> > @@ -1073,17 +1073,56 @@ static inline int pci_dev_poll_until_not_equal(struct pci_dev *dev, int where,
> >
> > static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
> > {
> > + int waited = 0;
> > + int rc = 0;
> > +
> > +
> > /*
> > * After reset, the device should not silently discard config
> > * requests, but it may still indicate that it needs more time by
> > - * responding to them with CRS completions. The Root Port will
> > - * generally synthesize ~0 data to complete the read (except when
> > - * CRS SV is enabled and the read was for the Vendor ID; in that
> > - * case it synthesizes 0x0001 data).
> > - *
> > - * Wait for the device to return a non-CRS completion. Read the
> > - * Command register instead of Vendor ID so we don't have to
> > - * contend with the CRS SV value.
> > + * responding to them with CRS completions. For such completions:
> > + * - If CRS SV is enabled on the Root Port, and the read request
> > + * covers both bytes of the Vendor ID register, the Root Port
> > + * will synthesize the value 0x0001 (and set any extra requested
> > + * bytes to 0xff)
> > + * - If CRS SV is not enabled on the Root Port, the Root Port must
> > + * re-issue the Configuration Request as a new Request.
> > + * Depending on platform-specific Root Complex configurations,
> > + * the Root Port may stop retrying after a set number of attempts,
> > + * or a configured timeout is hit, or continue indefinitely
> > + * (ultimately resulting in non-PCI-specific platform errors, such as
> > + * a TOR timeout).
> > + */
> > + if (dev->crssv_enabled) {
> > + u32 id;
> > +
> > + rc = pci_dev_poll_until_not_equal(dev, PCI_VENDOR_ID, 0xffff,
> > + 0x0001, reset_type, timeout,
> > + &waited, &id);
> > + if (rc)
> > + return rc;
> > +
> > + timeout -= waited;
> > +
> > + /*
> > + * If Vendor/Device ID is valid, the device must be ready.
> > + * Note: SR-IOV VFs return ~0 for reads to Vendor/Device
> > + * ID and will not be recognized as ready by this check.
> > + */
> > + if (id != 0x0000ffff && id != 0xffff0000 &&
> > + id != 0x00000000 && id != 0xffffffff)
> > + return 0;
> > + }
> > +
> > + /*
> > + * Root Ports will generally indicate error scenarios (e.g.
> > + * internal timeouts, or received Completion with CA/UR) by
> > + * synthesizing an 'all bits set' value (~0).
> > + * In case CRS is not supported/enabled, as well as for SR-IOV VFs,
> > + * fall back to polling a different register that cannot validly
> > + * contain ~0. As of PCIe 5.0, bits 11-15 of COMMAND are still RsvdP
> > + * and must return 0 when read.
> > + * XXX: These bits might become meaningful in the future
> > */
> > return pci_dev_poll_until_not_equal(dev, PCI_COMMAND, ~0, ~0,
> > reset_type, timeout, NULL,
> > --
> > 2.25.1
> >
> >
> >
> >
> > Amazon Development Center Germany GmbH
> > Krausenstr. 38
> > 10117 Berlin
> > Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
> > Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
> > Sitz: Berlin
> > Ust-ID: DE 289 237 879
> >
> >
> >
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
