Bjorn, Jason, On Mon, 28 Jan 2013 22:55:08 -0700, Jason Gunthorpe wrote: > > There's no Linux requirement that multiple PCIe interfaces appear to > > be in the same hierarchy. You can just use pci_scan_root_bus() > > separately on each interface. Each interface can be in its own domain > > if necessary. > > What you suggest is basically what the Marvell driver did originally, > the probelm is that Linux requires a pre-assigned aperture for each > PCI domain/root bus, and these new chips have so many PCI-E ports that > they can exhaust the physical address space, and also a limited > internal HW resource for setting address routing. > > Thus they require resource allocation that is sensitive to the devices > present downstream. > > By far the simplest solution is to merge all the physical links into a > single domain and rely on existing PCI resource allocation code to > drive allocation of scarce physical address space and demand allocate > the HW routing resource (specifically there are enough resources to > accomidate MMIO only devices on every bus, but not enough to > accomidate MMIO and IO on every bus). > > > > +/* > > > + * For a given PCIe interface (represented by a mvebu_pcie_port > > > + * structure), we read the PCI configuration space of the > > > + * corresponding PCI-to-PCI bridge in order to find out which range of > > > + * I/O addresses and memory addresses have been assigned to this PCIe > > > + * interface. Using these informations, we set up the appropriate > > > + * address decoding windows so that the physical address are actually > > > + * resolved to the right PCIe interface. > > > + */ > > > > Are you inferring the host bridge apertures by using the resources > > assigned to devices under the bridge, i.e., taking the union of all > > The flow is different, a portion of physical address space is set > aside for use by PCI-E (via DT) and that portion is specified in the > struct resource's ultimately attached to the PCI domain for the bus > scan. You could call that the 'host bridge aperture' though it doesn't > reflect any HW configuration at all. The values come from the device > tree. > > During the bus scan the Linux core code splits up that contiguous > space and assigns to the PCI-PCI bridges and devices under that domain. > > Each physical PCI-E link on the chip is seen by Linux through the SW > emulated PCI-PCI bridge attached to bus 0. When Linux configures the > bridge windows it triggers this code here to copy that window > information from the PCI config space into non-standard internal HW > registers. > > The purpose of the SW PCI-PCI bridge and this code here is to give > the Linux PCI core control over the window (MMIO,IO,busnr) assigned > to the PCI-E link. > > This arrangement with PCI-PCI bridges controlling address routing is > part of the PCI-E standard, in this instance Marvell did not implement > the required config space in HW so the driver is working around that > deficiency. > > Other drivers, like tegra have a similar design, but their hardware > does implement PCI-PCI bridge configuration space and does drive > address decoding through the HW PCI-PCI window registers. > > Having PCI-E links be bridges, not domains/root_bus's is in-line with > the standard and works better with the Linux PCI resource allocator. Thanks a lot Jason for this explanation, I couldn't have explained it as clearly as you did. Bjorn, does Jason's reply answers your questions? Or do you need other details? Thanks! Thomas -- Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
