[+cc Frank, original patch at https://lore.kernel.org/r/20241011140043.1250030-2-daire.mcnamara@xxxxxxxxxxxxx] On Thu, Jan 16, 2025 at 04:46:19PM +0000, Conor Dooley wrote: > On Thu, Jan 16, 2025 at 09:42:53AM -0600, Bjorn Helgaas wrote: > > On Tue, Jan 14, 2025 at 06:13:10PM -0600, Bjorn Helgaas wrote: > > > On Fri, Oct 11, 2024 at 03:00:41PM +0100, daire.mcnamara@xxxxxxxxxxxxx wrote: > > > > From: Daire McNamara <daire.mcnamara@xxxxxxxxxxxxx> > > > > > > > > On Microchip PolarFire SoC (MPFS) the PCIe Root Port can be behind one of > > > > three general-purpose Fabric Interface Controller (FIC) buses that > > > > encapsulate an AXI-M interface. That FIC is responsible for managing > > > > the translations of the upper 32-bits of the AXI-M address. On MPFS, > > > > the Root Port driver needs to take account of that outbound address > > > > translation done by the parent FIC bus before setting up its own > > > > outbound address translation tables. In all cases on MPFS, > > > > the remaining outbound address translation tables are 32-bit only. > > > > > > > > Limit the outbound address translation tables to 32-bit only. > > > > > > I don't quite understand what this is saying. It seems like the code > > > keeps only the low 32 bits of a PCI address and throws away any > > > address bits above the low 32. > > > > > > If that's what the FIC does, I wouldn't describe the FIC as > > > "translating the upper 32 bits" since it sounds like the translation > > > is just truncation. > > > > > > I guess it must be more complicated than that? I assume you can still > > > reach BARs that have PCI addresses above 4GB using CPU loads/stores? > > > > > > The apertures through the host bridge for MMIO access are described by > > > DT ranges properties, so this must be something that can't be > > > described that way? > > > > Ping? I'd really like to understand this before the v6.14 merge > > window opens on Sunday. > > Daire's been having some issues getting onto the corporate VPN to send > his reply, I've pasted it below on his behalf: > > There are 3 Fabric Inter Connect (FIC) buses on PolarFire SoC - each of > these FIC buses contain an AXI master bus and are 64-bits wide. These > AXI-Masters (each with an individual 64-bit AXI base address – for example > FIC1’s AXI Master has a base address of 0x2000000000) are connected to > general purpose FPGA logic. This FPGA logic is, in turn, connected to a > 2nd 32-bit AXI master which is attached to the PCIe block in RootPort mode. > Conceptually, on the other side of this configurable logic, there is a > 32-bit bus to a hard PCIe rootport. So, again conceptually, outbound address > translation looks like this: > > Processor Complex à FIC (64-bit AXI-M) à Configurable Logic à 32-bit AXI-M à PCIe Rootport > (This how it came to me from Daire, I think the á is meant to > be an arrow) > > This allows a designer two broad choices: > > Choice of FIC (effectively choice of AXI bus) > Ability to offset the AXI address of any peripherals they add in the > Fabric. > > So, for the case of an outbound AXI address, from the processors’ point > of view (or Linux’ point of view if you prefer), the processor uses a > 64-bit AXI address, then – in a very general way of viewing the process > and thinking only about accessing the PCIe device – the FPGA logic can > be configured to adjust that AXI-M address to any arbitrary “address” > before it passes that new “address” to the Root Port over a second 32-bit > AXI bus (the main constraint is that the FPGA logic can only use a 32-bit > address on that AXI-M interface to the Root Port). > > To manage this complexity, Microchip have design rules for customers > building their FPGA logic where we strongly recommend that they only > interact with the upper 32 bits of the 64-bit address in the FPGA logic > and pass the lower 32 bits through (unmodified) to the AXI-M side of the > PCIe Root Port. This allows them to “move” a 64-bit AXI-M window for their > PCIe Root Port (as viewed by the processor) for their particular design – > if they need to - so that they can also access any other AXI-M windows > associated with any other peripherals they might add to their design. > > In practise, so far, all customers, and our own internal boards have all > started by using one of two major reference designs from us (one using FIC1 > where the AXI-M window destined for the PCIe Root Port starts at 0x2000000000 > and one using FIC2 where its AXI-M window, again destined for the PCIe Root > Port starts at 0x3000000000). Is there something special about this that cannot be described by a DT 'ranges' property? This sounds conceptually similar to Frank's nice picture at https://lore.kernel.org/r/20241119-pci_fixup_addr-v8-2-c4bfa5193288@xxxxxxx Bjorn
