On Fri, 14 Oct 2022 16:30:34 +0300 Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx> wrote: > Hi, > > On Fri, Oct 14, 2022 at 12:45:53PM +0100, Jonathan Cameron wrote: > > On Mon, 5 Sep 2022 11:02:30 +0300 > > Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx> wrote: > > > > > Currently we distribute the spare resources only upon hot-add so if > > > there are PCI devices connected already when the initial root bus scan > > > is done, and they have not been fully configured by the BIOS, we may end > > > up allocating resources just enough to cover only what is currently > > > there. If some of those devices are hotplug bridges themselves we do not > > > leave any additional resource space for future expansion. > > > > > > For this reason distribute the available resources for root buses too to > > > make this work the same way we do in the normal hotplug case. > > > > > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=216000 > > > Reported-by: Chris Chiu <chris.chiu@xxxxxxxxxxxxx> > > > Tested-by: Chris Chiu <chris.chiu@xxxxxxxxxxxxx> > > > Reviewed-by: Andy Shevchenko <andriy.shevchenko@xxxxxxxxxxxxxxx> > > > Signed-off-by: Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx> > > > --- > > > > Early days, but I have a regression that's bisecting to this patch > > after I did an optimistic checkout of mainline mid merge window. > > > > Initial platform was QEMU emulation of CXL on arm64 with EDK2 / ACPI etc. > > That uses an extended version of the PCI eXpander Bus (PXB) with extended > > version of the emulated pci root ports. Particular test was > > > > 1 PXB, 2 RP, below first RP switch USP, with two functions (USP and > > a stand along switch CCI), below that 4 DSP each with a CXL type 3 device. > > > > Clearly something wrong with the error handling in CXL code as it goes boom > > spectacularly. I'll address that separately. Anyhow, setup way to complex > > to debug easily. > > > > Anyhow to make this easier for others to poke I set up equivalent with pxb-pcie > > etc and it happens there. Everything works with the exception of additional > > functions on USP. > > > > Next up in minimal test case (could be more minimal but I'm lazy) is > > > > ../qemu/bin/native/aarch64-softmmu/qemu-system-aarch64 -M virt,nvdimm=on,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max -smp 4 \ > > -kernel Image \ > > -drive if=none,file=full.qcow2,format=qcow2,id=hd \ > > -device pcie-root-port,id=root_port1 -device virtio-blk-pci,drive=hd \ > > -netdev type=user,id=mynet,hostfwd=tcp::5555-:22 \ > > -device virtio-net-pci,netdev=mynet,id=bob \ > > -nographic -no-reboot -append 'earlycon root=/dev/vda2 fsck.mode=skip tp_printk maxcpus=4' \ > > -monitor telnet:127.0.0.1:1235,server,nowait -bios QEMU_EFI.fd \ > > -object memory-backend-ram,size=4G,id=mem0 \ > > -numa node,nodeid=0,cpus=0-3,memdev=mem0 \ > > # Interesting part follows: > > -device pcie-root-port,port=0,id=root_port13,chassis=0,slot=2 \ > > -device x3130-upstream,id=sw1,bus=root_port13,multifunction=on \ > > -device e1000,bus=root_port13,addr=0.1 \ > > -device xio3130-downstream,id=fun1,bus=sw1,chassis=0,slot=3\ > > -device e1000,bus=fun1 > > > > > > Error message on failure to probe my switch integrated e1000 (who doesn't have one of > > those - lets pretend it's a more common switch PMU function or something like > > that) is : > > e1000 0000:02:00.1: can't ioremap BAR 0: [??? 0x00000000 flags 0x0] > > > > With this patch reverted it works as normal and the e1000 driver comes up. > > > > Possibly useful stuff follows: > > > > # lspci -tv > > -[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge > > +-01.0-[01]-- > > +-02.0 Red Hat, Inc. Virtio block device > > +-03.0 Red Hat, Inc. Virtio network device > > \-04.0-[02-04]--+-00.0-[03-04]----00.0-[04]----00.0 Intel Corporation 82540EM Gigabit Ethernet Controller > > \-00.1 Intel Corporation 82540EM Gigabit Ethernet Controller > > > > with patch in place. > > //without patch > > Note I aligned by hand so may be errorr. > > > > # dmesg | grep pci > > pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff window] > > pci_bus 0000:00: root bus resource [io 0x0000-0xffff window] > > pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff window] > > pci_bus 0000:00: root bus resource [bus 00-ff] > > pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 > > pci 0000:00:01.0: [1b36:000c] type 01 class 0x060400 > > pci 0000:00:01.0: reg 0x10: [mem 0x10643000-0x10643fff] > > pci 0000:00:02.0: [1af4:1001] type 00 class 0x010000 > > pci 0000:00:02.0: reg 0x10: [io 0x2200-0x227f] > > pci 0000:00:02.0: reg 0x14: [mem 0x10642000-0x10642fff] > > pci 0000:00:02.0: reg 0x20: [mem 0x8000000000-0x8000003fff 64bit pref] > > pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 > > pci 0000:00:03.0: reg 0x10: [io 0x2280-0x229f] > > pci 0000:00:03.0: reg 0x14: [mem 0x10641000-0x10641fff] > > pci 0000:00:03.0: reg 0x20: [mem 0x8000004000-0x8000007fff 64bit pref] > > pci 0000:00:03.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] > > pci 0000:00:04.0: [1b36:000c] type 01 class 0x060400 > > pci 0000:00:04.0: reg 0x10: [mem 0x10640000-0x10640fff] > > pci 0000:02:00.0: [104c:8232] type 01 class 0x060400 > > pci 0000:02:00.1: [8086:100e] type 00 class 0x020000 > > pci 0000:02:00.1: reg 0x10: [mem 0x10200000-0x1021ffff] > > pci 0000:02:00.1: reg 0x14: [io 0x1000-0x103f] > > pci 0000:02:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] > > pci 0000:03:00.0: [104c:8233] type 01 class 0x060400 > > pci 0000:04:00.0: [8086:100e] type 00 class 0x020000 > > pci 0000:04:00.0: reg 0x10: [mem 0x10000000-0x1001ffff] > > pci 0000:04:00.0: reg 0x14: [io 0x0000-0x003f] > > pci 0000:04:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] > > pci 0000:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 > > pci 0000:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 > > pci 0000:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000 > > pci 0000:03:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000 > > pci 0000:03:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 04] add_size 100000 add_align 100000 > > pci 0000:02:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 03-04] add_size 200000 add_align 100000 > > pci 0000:02:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 03-04] add_size 100000 add_align 100000 > > pci 0000:00:04.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 02-04] add_size 300000 add_align 100000 > > pci 0000:00:04.0: bridge window [mem 0x00100000-0x002fffff] to [bus 02-04] add_size 100000 add_align 100000 > > pci 0000:00:01.0: BAR 14: assigned [mem 0x10000000-0x101fffff] > > pci 0000:00:01.0: BAR 15: assigned [mem 0x8000000000-0x80001fffff 64bit pref] > > pci 0000:00:04.0: BAR 14: assigned [mem 0x10200000-0x103fffff] > > // pci 0000:00:04.0: BAR 14: assigned [mem 0x10200000-0x104fffff] > > pci 0000:00:04.0: BAR 15: assigned [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci 0000:00:04.0: BAR 15: assigned [mem 0x8000200000-0x80005fffff 64bit pref] > > pci 0000:00:03.0: BAR 6: assigned [mem 0x10400000-0x1043ffff pref] > > // pci 0000:00:03.0: BAR 6: assigned [mem 0x10500000-0x1053ffff pref] > > pci 0000:00:02.0: BAR 4: assigned [mem 0x8000300000-0x8000303fff 64bit pref] > > // pci 0000:00:02.0: BAR 4: assigned [mem 0x8000600000-0x8000603fff 64bit pref] > > pci 0000:00:03.0: BAR 4: assigned [mem 0x8000304000-0x8000307fff 64bit pref] > > // pci 0000:00:03.0: BAR 4: assigned [mem 0x8000604000-0x8000607fff 64bit pref] > > pci 0000:00:01.0: BAR 0: assigned [mem 0x10440000-0x10440fff] > > // pci 0000:00:01.0: BAR 0: assigned [mem 0x10540000-0x10540fff] > > pci 0000:00:01.0: BAR 13: assigned [io 0x1000-0x1fff] > > pci 0000:00:02.0: BAR 1: assigned [mem 0x10441000-0x10441fff] > > // pci 0000:00:02.0: BAR 1: assigned [mem 0x10541000-0x10541fff] > > pci 0000:00:03.0: BAR 1: assigned [mem 0x10442000-0x10442fff] > > // pci 0000:00:03.0: BAR 1: assigned [mem 0x10542000-0x10542fff] > > pci 0000:00:04.0: BAR 0: assigned [mem 0x10443000-0x10443fff] > > // pci 0000:00:01.0: BAR 0: assigned [mem 0x10540000-0x10540fff] > > pci 0000:00:04.0: BAR 13: assigned [io 0x2000-0x3fff] > > pci 0000:00:02.0: BAR 0: assigned [io 0x4000-0x407f] > > pci 0000:00:03.0: BAR 0: assigned [io 0x4080-0x409f] > > pci 0000:00:01.0: PCI bridge to [bus 01] > > pci 0000:00:01.0: bridge window [io 0x1000-0x1fff] > > pci 0000:00:01.0: bridge window [mem 0x10000000-0x101fffff] > > pci 0000:00:01.0: bridge window [mem 0x8000000000-0x80001fffff 64bit pref] > > pci 0000:02:00.0: BAR 14: assigned [mem 0x10200000-0x103fffff] > > pci 0000:02:00.0: BAR 15: assigned [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci 0000:02:00.0: BAR 15: assigned [mem 0x8000200000-0x80004fffff 64bit pref] > > pci 0000:02:00.1: BAR 6: no space for [mem size 0x00040000 pref] > > pci 0000:02:00.1: BAR 6: failed to assign [mem size 0x00040000 pref] > > // pci 0000:02:00.1: BAR 6: assigned [mem 0x10400000-0x1043ffff pref] > > pci 0000:02:00.1: BAR 0: failed to assign [mem size 0x00020000] > > // pci 0000:02:00.1: BAR 0: assigned [mem 0x10440000-0x1045ffff] > > pci 0000:02:00.0: BAR 13: assigned [io 0x2000-0x3fff] > > pci 0000:02:00.1: BAR 1: no space for [io size 0x0040] > > // pci 0000:02:00.1: BAR 1: assigned [io 0x3000-0x303f] > > pci 0000:02:00.1: BAR 1: failed to assign [io size 0x0040] > > pci 0000:03:00.0: BAR 14: assigned [mem 0x10200000-0x103fffff] > > pci 0000:03:00.0: BAR 15: assigned [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci 0000:03:00.0: BAR 15: assigned [mem 0x8000200000-0x80003fffff 64bit pref] > > pci 0000:03:00.0: BAR 13: assigned [io 0x2000-0x3fff] > > // pci 0000:03:00.0: BAR 13: assigned [io 0x2000-0x2fff] > > pci 0000:04:00.0: BAR 6: assigned [mem 0x10200000-0x1023ffff pref] > > pci 0000:04:00.0: BAR 0: assigned [mem 0x10240000-0x1025ffff] > > pci 0000:04:00.0: BAR 1: assigned [io 0x2000-0x203f] > > pci 0000:03:00.0: PCI bridge to [bus 04] > > pci 0000:03:00.0: bridge window [io 0x2000-0x3fff] > > // pci 0000:03:00.0: bridge window [io 0x2000-0x2fff] > > pci 0000:03:00.0: bridge window [mem 0x10200000-0x103fffff] > > pci 0000:03:00.0: bridge window [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci 0000:03:00.0: bridge window [mem 0x8000200000-0x80003fffff 64bit pref] > > pci 0000:02:00.0: PCI bridge to [bus 03-04] > > pci 0000:02:00.0: bridge window [io 0x2000-0x3fff] > > // pci 0000:02:00.0: bridge window [io 0x2000-0x2fff] > > pci 0000:02:00.0: bridge window [mem 0x10200000-0x103fffff] > > pci 0000:02:00.0: bridge window [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci 0000:02:00.0: bridge window [mem 0x8000200000-0x80004fffff 64bit pref] > > pci 0000:00:04.0: PCI bridge to [bus 02-04] > > pci 0000:00:04.0: bridge window [io 0x2000-0x3fff] > > pci 0000:00:04.0: bridge window [mem 0x10200000-0x103fffff] > > // pci 0000:00:04.0: bridge window [mem 0x10200000-0x104fffff] > > pci 0000:00:04.0: bridge window [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci 0000:00:04.0: bridge window [mem 0x8000200000-0x80005fffff 64bit pref] > > pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc > > // LINE NOT PRESENT > > pci_bus 0000:00: resource 4 [mem 0x10000000-0x3efeffff window] > > pci_bus 0000:00: resource 5 [io 0x0000-0xffff window] > > pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff window] > > pci_bus 0000:01: resource 0 [io 0x1000-0x1fff] > > pci_bus 0000:01: resource 1 [mem 0x10000000-0x101fffff] > > pci_bus 0000:01: resource 2 [mem 0x8000000000-0x80001fffff 64bit pref] > > pci_bus 0000:02: resource 0 [io 0x2000-0x3fff] > > pci_bus 0000:02: resource 1 [mem 0x10200000-0x103fffff] > > // pci_bus 0000:02: resource 1 [mem 0x10200000-0x104fffff] > > pci_bus 0000:02: resource 2 [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci_bus 0000:02: resource 2 [mem 0x8000200000-0x80005fffff 64bit pref] > > pci_bus 0000:03: resource 0 [io 0x2000-0x3fff] > > // pci_bus 0000:03: resource 0 [io 0x2000-0x2fff] > > pci_bus 0000:03: resource 1 [mem 0x10200000-0x103fffff] > > pci_bus 0000:03: resource 2 [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci_bus 0000:03: resource 2 [mem 0x8000200000-0x80004fffff 64bit pref] > > pci_bus 0000:04: resource 0 [io 0x2000-0x3fff] > > // pci_bus 0000:04: resource 0 [io 0x2000-0x2fff] > > pci_bus 0000:04: resource 1 [mem 0x10200000-0x103fffff] > > // pci_bus 0000:04: resource 1 [mem 0x10200000-0x103fffff] > > pci_bus 0000:04: resource 2 [mem 0x8000200000-0x80002fffff 64bit pref] > > // pci_bus 0000:04: resource 2 [mem 0x8000200000-0x80003fffff 64bit pref] > > pcieport 0000:00:01.0: PME: Signaling with IRQ 51 > > pcieport 0000:00:01.0: AER: enabled with IRQ 51 > > pcieport 0000:00:01.0: pciehp: Slot #0 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock+ NoCompl- IbPresDis- LLActRep+ > > pcieport 0000:00:04.0: PME: Signaling with IRQ 52 > > pcieport 0000:00:04.0: AER: enabled with IRQ 52 > > pcieport 0000:00:04.0: pciehp: Slot #2 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock+ NoCompl- IbPresDis- LLActRep+ > > pcieport 0000:03:00.0: pciehp: Slot #3 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock+ NoCompl- IbPresDis- LLActRep- > > virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) > > virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) > > > > All suggestions of things to test and other useful info to provide welcome. > > Thanks for the detailed report! I wonder if you could try the below > patch and see if it changes anything? Thanks for the quick response. Doesn't fix it unfortunately. For reference with the extra prints and a bit of context (trivial fix below) pci 0000:0e:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 0f] add_size 200000 add_align 100000 pci 0000:0e:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 0f] add_size 100000 add_align 100000 pci 0000:0d:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 0e-0f] add_size 200000 add_align 100000 pci 0000:0d:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 0e-0f] add_size 100000 add_align 100000 pci 0000:0c:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 0d-0f] add_size 300000 add_align 100000 pci 0000:0c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 0d-0f] add_size 100000 add_align 100000 pci 0000:0d:00.0: check [io 0x1000-0x1fff pci 0000:0d:00.0: check [mem 0x00100000-0x001fffff] pci 0000:0d:00.0: check [mem 0x00100000-0x001fffff 64bit pref] pci 0000:0c:00.0: BAR 14: assigned [mem 0x10400000-0x105fffff] pci 0000:0c:00.0: BAR 15: assigned [mem 0x10600000-0x106fffff 64bit pref] pci 0000:0c:00.0: BAR 0: assigned [mem 0x10740000-0x10740fff] pci 0000:0c:00.0: BAR 13: no space for [io size 0x2000] pci 0000:0c:00.0: BAR 13: failed to assign [io size 0x2000] pci 0000:0d:00.0: BAR 14: assigned [mem 0x10400000-0x105fffff] pci 0000:0d:00.0: BAR 15: assigned [mem 0x10600000-0x106fffff 64bit pref] > I will be on vacation starting > tomorrow for the whole next week so if nothing else helps then I suppose > we should revert that commit for now. > > I can look at this further after I come back. Enjoy your vacation. I might get time to dig a bit more whilst you are away though probably not :( Jonathan > > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c > index dc6a30ee6edf..637f1a1df409 100644 > --- a/drivers/pci/setup-bus.c > +++ b/drivers/pci/setup-bus.c > @@ -1939,7 +1939,7 @@ static void pci_bridge_distribute_available_resources(struct pci_dev *bridge, > available_mmio_pref); > } > > -static bool pci_bridge_resources_not_assigned(struct pci_dev *dev) > +static bool pci_bridge_resources_assigned(struct pci_dev *dev) > { > const struct resource *r; > > @@ -1951,16 +1951,19 @@ static bool pci_bridge_resources_not_assigned(struct pci_dev *dev) > * in the same way we do with the normal hotplug case. > */ > r = &dev->resource[PCI_BRIDGE_IO_WINDOW]; > - if (!r->flags || !(r->flags & IORESOURCE_STARTALIGN)) > - return false; > + dev_info(dev->dev, "check %pR\n", r); &dev->dev > + if (r->flags && !(r->flags & IORESOURCE_STARTALIGN)) > + return true; > r = &dev->resource[PCI_BRIDGE_MEM_WINDOW]; > - if (!r->flags || !(r->flags & IORESOURCE_STARTALIGN)) > - return false; > + dev_info(dev->dev, "check %pR\n", r); > + if (r->flags && !(r->flags & IORESOURCE_STARTALIGN)) > + return true; > r = &dev->resource[PCI_BRIDGE_PREF_MEM_WINDOW]; > - if (!r->flags || !(r->flags & IORESOURCE_STARTALIGN)) > - return false; > + dev_info(dev->dev, "check %pR\n", r); > + if (r->flags && !(r->flags & IORESOURCE_STARTALIGN)) > + return true; > > - return true; > + return false; > } > > static void pci_root_bus_distribute_available_resources(struct pci_bus *bus, > @@ -1979,7 +1982,7 @@ static void pci_root_bus_distribute_available_resources(struct pci_bus *bus, > * Need to check "bridge" here too because it is NULL > * in case of root bus. > */ > - if (bridge && pci_bridge_resources_not_assigned(dev)) { > + if (bridge && !pci_bridge_resources_assigned(dev)) { > pci_bridge_distribute_available_resources(bridge, add_list); > /* > * There is only PCIe upstream port on the bus > >
