On Tue, Nov 17, 2009 at 12:47 AM, Thierry Reding <thierry.reding@xxxxxxxxxxxxxxxxx> wrote: > * Yinghai Lu wrote: >> On Mon, Nov 16, 2009 at 7:40 AM, Thierry Reding >> <thierry.reding@xxxxxxxxxxxxxxxxx> wrote: >> > Hi, >> > >> > I have a setup where an FPGA is connected to a PCIe port. The FPGA itself >> > acts as a bridge and forwards PCI requests to some user logic that in turn >> > simulates multiple PCIe endpoints. The communication between the root complex >> > and the user logic works fine and all endpoints are visible to the BIOS and >> > Linux later on. >> > >> > One problem that we're facing now is that during PCIe enumeration, the BIOS >> > seems to correctly allocate and assign resources for all endpoints, but only >> > if all BARs are designated prefetchable. >> > >> > If the BARs are configured as non-prefetchable, however, the BIOS still seems >> > to allocate and assign resources correctly, but after the first assignment of >> > base addresses, it decides to overwrite these with 0, except for the very >> > first endpoint. Now when Linux comes up it reads those BARs and decodes them >> > again, then writes the original 0 back. Therefore when the BARs are marked >> > non-prefetchable, all endpoints except the first end up unusable in the >> > system. >> > >> > Does anyone have an idea what might be going on here? Is there some >> > restriction in PCIe that may be causing this behaviour? I've been going >> > through the specification looking for clues but so far haven't come up with >> > anything. >> > >> > Presuming I can't make the BIOS "do the right thing", is it possible to have >> > the Linux kernel redo the PCIe enumeration and ignore the BIOS completely? >> > Again I've been going through the code but all I've been able to find is those >> > places where Linux checks the resources as allocated by the BIOS. There seems >> > to be no code to override the BIOS. >> >> please send out bootlog and lspci -tv... > > dmesg (prefetchable and non-prefetchable cases) and lspci -tv output is > attached. Perhaps I should give some more details about the setup: as can be > seen from the lspci setup, there are three endpoints implemented in the FPGA, > each behind its own bridge. All endpoints have all 6 BARs implemented, each > BAR being 1MB non-prefetchable. For what it's worth, decreasing the BAR size > to 64KB makes no difference. > > The odd thing, judging by the dmesg output, is that the bridges before the > endpoints seem to get the correct configuration. Only for the endpoints does > the BIOS seem to get things wrong. Dumping accesses to the BARs from within > the FPGA shows that the BIOS indeed writes the correct bases to the endpoint > registers (the bases nicely fitting into the window that the bridge is > configured with) but then goes and overwrites those values with 0. > > Again, switching from non-prefetchable to prefetchable mode no longer exposes > the issue as demonstrated by the second dmesg output. your BIOS doesn't assign some mmio [ 0.176009] pci 0000:03:00.0: reg 10 32bit mmio: [0xbda00000-0xbdafffff] [ 0.192008] pci 0000:03:00.0: reg 14 32bit mmio: [0xbd900000-0xbd9fffff] [ 0.208008] pci 0000:03:00.0: reg 18 32bit mmio: [0xbd800000-0xbd8fffff] [ 0.224008] pci 0000:03:00.0: reg 1c 32bit mmio: [0xbd700000-0xbd7fffff] [ 0.240008] pci 0000:03:00.0: reg 20 32bit mmio: [0xbd600000-0xbd6fffff] [ 0.256008] pci 0000:03:00.0: reg 24 32bit mmio: [0xbd500000-0xbd5fffff] [ 0.256220] pci 0000:03:00.0: reg 30 32bit mmio pref: [0xfc4f0000-0xfc4f07ff] [ 0.268008] pci 0000:02:00.0: bridge 32bit mmio: [0xbce00000-0xbdafffff] [ 0.292009] pci 0000:04:00.0: reg 10 32bit mmio: [0x000000-0x0fffff] [ 0.308008] pci 0000:04:00.0: reg 14 32bit mmio: [0x000000-0x0fffff] [ 0.324008] pci 0000:04:00.0: reg 18 32bit mmio: [0x000000-0x0fffff] [ 0.340008] pci 0000:04:00.0: reg 1c 32bit mmio: [0x000000-0x0fffff] [ 0.356008] pci 0000:04:00.0: reg 20 32bit mmio: [0x000000-0x0fffff] [ 0.372008] pci 0000:04:00.0: reg 24 32bit mmio: [0x000000-0x0fffff] [ 0.372217] pci 0000:04:00.0: reg 30 32bit mmio pref: [0x000000-0x0007ff] [ 0.380008] pci 0000:02:01.0: bridge 32bit mmio: [0xbdb00000-0xbe7fffff] [ 0.408009] pci 0000:05:00.0: reg 10 32bit mmio: [0x000000-0x0fffff] [ 0.424008] pci 0000:05:00.0: reg 14 32bit mmio: [0x000000-0x0fffff] [ 0.440008] pci 0000:05:00.0: reg 18 32bit mmio: [0x000000-0x0fffff] [ 0.456008] pci 0000:05:00.0: reg 1c 32bit mmio: [0x000000-0x0fffff] [ 0.472009] pci 0000:05:00.0: reg 20 32bit mmio: [0x000000-0x0fffff] [ 0.488008] pci 0000:05:00.0: reg 24 32bit mmio: [0x000000-0x0fffff] [ 0.488217] pci 0000:05:00.0: reg 30 32bit mmio pref: [0x000000-0x0007ff] [ 0.500008] pci 0000:02:02.0: bridge 32bit mmio: [0xbe800000-0xbf4fffff] kernel should allocate them correctly, please enable CONFIG_PCI_DEBUG to get more dump or lspci -vvxxxx could tell right result. YH -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
