On 8/10/2023 2:50 PM, Nirmal Patel wrote: > During domain reset process vmd_domain_reset() clears PCI > configuration space of VMD root ports. But certain platform > has observed following errors and failed to boot. > ... > DMAR: VT-d detected Invalidation Queue Error: Reason f > DMAR: VT-d detected Invalidation Time-out Error: SID ffff > DMAR: VT-d detected Invalidation Completion Error: SID ffff > DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0 > DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0 > DMAR: Invalidation Time-out Error (ITE) cleared > > The root cause is that memset_io() clears prefetchable memory base/limit > registers and prefetchable base/limit 32 bits registers sequentially. > This seems to be enabling prefetchable memory if the device disabled > prefetchable memory originally. > > Here is an example (before memset_io()): > > PCI configuration space for 10000:00:00.0: > 86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00 > 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20 > 00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00 > ... > > So, prefetchable memory is ffffffff00000000-575000fffff, which is > disabled. When memset_io() clears prefetchable base 32 bits register, > the prefetchable memory becomes 0000000000000000-575000fffff, which is > enabled and incorrect. > > Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec: > > The Prefetchable Memory Limit register must be programmed to a smaller > value than the Prefetchable Memory Base register if there is no > prefetchable memory on the secondary side of the bridge. > > This is believed to be the reason for the failure and in addition the > sequence of operation in vmd_domain_reset() is not following the PCIe > specs. > > Disable the bridge window by executing a sequence of operations > borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(), > that comply with the PCI specifications. > > Signed-off-by: Nirmal Patel <nirmal.patel@xxxxxxxxxxxxxxx> > --- > v4->v5: Revome PCI_CAPABILITY_LIST from the reset. > v3->v4: Following same operation as pci_setup_bridge_io. > v2->v3: Add more information to commit description. > v1->v2: Follow same chain of operation as pci_disable_bridge_window > and update commit log. > --- > drivers/pci/controller/vmd.c | 19 +++++++++++++++++-- > 1 file changed, 17 insertions(+), 2 deletions(-) > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > index 769eedeb8802..c5b1295ab208 100644 > --- a/drivers/pci/controller/vmd.c > +++ b/drivers/pci/controller/vmd.c > @@ -526,8 +526,23 @@ static void vmd_domain_reset(struct vmd_dev *vmd) > PCI_CLASS_BRIDGE_PCI)) > continue; > > - memset_io(base + PCI_IO_BASE, 0, > - PCI_ROM_ADDRESS1 - PCI_IO_BASE); > + /* > + * Temporarily disable the I/O range before updating > + * PCI_IO_BASE. > + */ > + writel(0x0000ffff, base + PCI_IO_BASE_UPPER16); > + /* Update lower 16 bits of I/O base/limit */ > + writew(0x00f0, base + PCI_IO_BASE); > + /* Update upper 16 bits of I/O base/limit */ > + writel(0, base + PCI_IO_BASE_UPPER16); > + > + /* MMIO Base/Limit */ > + writel(0x0000fff0, base + PCI_MEMORY_BASE); > + > + /* Prefetchable MMIO Base/Limit */ > + writel(0, base + PCI_PREF_LIMIT_UPPER32); > + writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE); > + writel(0xffffffff, base + PCI_PREF_BASE_UPPER32); > } > } > } Gentle Ping Thanks
