On Thu, Jul 15, 2021 at 06:12:25PM +0000, Billie Alsup (balsup) wrote: > It took me a while to figure out that the "New Outlook" option > doesn't actually allow sending plain text, so I have to switch to > "Old Outlook" mode. Since you've gone to that much trouble already, also note http://vger.kernel.org/majordomo-info.html and https://people.kernel.org/tglx/notes-about-netiquette BTW, the attribution in the email you quoted below got corrupted in such a way that it appears that I wrote the whole thing, instead of what actually happened, which is that I wrote a half dozen lines of response to your email. Linux uses old skool email conventions ;) > It is not clear as to what parameters Linux would use to consider a > window broken. By "broken," I just mean things that are prohibited by the PCI spec, like "region doesn't fit in a window of an upstream device" or "non-prefetchable region placed in a prefetchable window." > But if the kernel preserves some bridge window > assignment, then it seems feasible for our BIOS to run this same > algorithm (reading PLX persistent scratch registers to determine > window sizes). I will raise this possibility with our own kernel > team to discuss with the bios team. We can also look more closely > at the resource_alignment options to see if that might suffice. > Thanks for the information! > From: Bjorn Helgaas <helgaas@xxxxxxxxxx> > Date: Thursday, July 15, 2021 at 10:14 AM > To: "Billie Alsup (balsup)" <balsup@xxxxxxxxx> > Cc: Paul Menzel <pmenzel@xxxxxxxxxxxxx>, Bjorn Helgaas <bhelgaas@xxxxxxxxxx>, Guohan Lu <lguohan@xxxxxxxxx>, "Madhava Reddy Siddareddygari (msiddare)" <msiddare@xxxxxxxxx>, "linux-pci@xxxxxxxxxxxxxxx" <linux-pci@xxxxxxxxxxxxxxx>, "linux-kernel@xxxxxxxxxxxxxxx" <linux-kernel@xxxxxxxxxxxxxxx>, "David S. Miller" <davem@xxxxxxxxxxxxx>, Jakub Kicinski <kuba@xxxxxxxxxx>, "netdev@xxxxxxxxxxxxxxx" <netdev@xxxxxxxxxxxxxxx>, Sergey Miroshnichenko <s.miroshnichenko@xxxxxxxxx> > Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges > > On Thu, Jul 15, 2021 at 04:52:26PM +0000, Billie Alsup (balsup) wrote: > We are aware of how Cisco device specific this code is, and hadn't > intended to upstream it. This code was originally written for an > older kernel version (4.8.28-WR9.0.0.26_cgl). I am not the original > author; I just ported it into various SONiC linux kernels. We use > ACPI with SONiC (although not on our non-SONiC products), so I > thought I might be able to define such windows within the ACPI tree > and have some generic code to read such configuration information > from the ACPI tables,. However, initial attempts failed so I went > with the existing approach. I believe we did look at the hpmmiosize > parameter, but iirc it applied to each bridge, rather than being a > pool of address space to dynamically parcel out as necessary. > > Right. I mentioned "pci=resource_alignment=" because it claims to be > able to specify window sizes for specific bridges. But I haven't > exercised that myself. > > There are multiple bridges involved in the hardware (there are 8 > hot-plug fabric cards, each with multiple PCI devices). Devices on > the card are in multiple power zones, so all devices are not > immediately visible to the pci scanning code. The top level bridge > reserves close to 5G. The 2nd level (towards the fabric cards) > reserve 4.5G. The 3rd level has 9 bridges each reserving 512M. The > 4th level reserves 384M (with a 512M alignment restriction iirc). > The 5th level reserves 384M (again with an alignment restriction). > That defines the bridge hierarchy visible at boot. Things behind > that 5th level are hot-plugged where there are two more bridge > levels and 5 devices (1 requiring 2x64M blocks and 4 requiring > 1x64M). > > I'm not sure if the Cisco kernel team has revisited the hpmmiosize > and resource_alignment parameters since this initial implementation. > Reading the description of Sergey's patches, he seems to be > approaching the same problem from a different direction. It is > unclear if such an approach is practical for our environment. It > would require updates to all of our SONiC drivers to support > stopping/remapping/restarting, and it is unclear if that is > acceptable. It is certainly less preferable to pre-reserving the > required space. For our embedded product, we know exactly what > devices will be plugged in, and allowing that to be pre-programmed > into the PLX eeprom gives us the flexibility we need. > > If you know up front what devices are possible and how much space they > need, possibly your firmware could assign the bridge windows you need. > Linux generally does not change window assignments unless they are > broken. > > Bjorn > >
