Hi Folks ! It's been a while ... (please extend the CC list as needed) I'd like to re-open an ancient issue because it's still biting us (AWS). A few years back, I updated the PCIe resource allocation code to be a bit more in line with what other architectures do. That said, once thing we couldn't agree on was to do like x86 and default to preserving the firmware provided resources by default. On x86, the kernel "allocates" (claims) the resources (unless it finds something obviously wrong), then allocates anything left unallocated. On arm64, we use to just re-allocate everything. I changed this to first use some more common code for doing all that, but also to have the option to claim existing resources if _DSM tells us to preserve them for a given host brigde. I still think this is the wrong way to go and that we should preserve the UEFI resources by default unless told not to :-) The case back then was that there existed some (how many ? there was one real example if I remember correctly) bogus firwmares that came out of UEFI with too small windows. We could just quirk those .... The reason I'm bringing this back is that re-allocating resources for system devices cause problems. The most obvious one today that is affecting EC2 instances is that the UART address specified in SPCR is no longer valid, causing issues ranging from the console not working to MMIO to what becomes "random addresses". Typically today this is "worked around" by using console=ttyS0 to force selection of the first detected PCI UART, because the match against SPCR is based on address and it won't match, but there's always the underlying risk that things like earlycon starts poking at now-incorrect addresses until 8250 takes over. This is the most obvious problem. Any other "system" device that happens to be PCIe based (anything detected early, via device-tree, ACPI or otherwise) is at risk of a similar issue. On x86 that could be catastrophic because near everything looks like a PCI device, on arm64 we seem to have been getting away with it a bit more easily ... so far. The alternative here would be to use ad-hock kludges for such system devices, to "register" the addresses early, and have some kind of hook in the PCI code that keeps track of them as they get remapped. If we want this, I would propose (happy to provide the implementation but let's discuss the design first) something along the line of a generic mechanism to "register" such a system device, which would add it to a list. That list would be scanned on PCI device discovery for BAR address matches, and the pci_dev/BAR# added to the entry (that or put a pointer to the entry into pci_dev for speed/efficiency). The difficulty is how is that update propagated: This is of course fiddly. For example, the serial info is passed via two different ways, one being earlycon (and probably the easiest to track), the other one an ASCII string passed to add_preferred_console(), which would require more significant hackery (the code dealing with console mathing is a gothic horror). Also if such a system device is in continuous use during the boot process (UART ?) it needs to be "updated" as soon as possible after the BARs (and parent BARs) have been also updated (in fact this is generally why PCI debug dies horribly when using PCI based UARTs). Maybe an (optional) callback that earlycon can add ? Additionally, this would only work if such system devices are "registered" before they get remapped... Another approach would be to have pci_dev keep a copy of the original resources (at least for the primary BARs) and provide an accessor for use by things like earlycon or 8250 to compare against these, though that doesn't solve the problem of promptly "updating" drivers for system devices. Opinions ? Cheers, Ben.
