Thanks Rob for taking the time to get back to me. Yes, I thought 2
levels of translation might be the way to go. If you could point me to
a gadget that has 2-level translation working, that'd be great
On Fri, 2022-09-02 at 11:28 -0500, Rob Herring wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you
> know the content is safe
>
> On Fri, Sep 2, 2022 at 9:22 AM <daire.mcnamara@xxxxxxxxxxxxx> wrote:
> > From: Conor Dooley <conor.dooley@xxxxxxxxxxxxx>
> >
> > On PolarFire SoC both in- & out-bound address translations occur in
> > two
> > stages. The specific translations are tightly coupled to the FPGA
> > designs and supplement the {dma-,}ranges properties. The first
> > stage of
> > the translation is done by the FPGA fabric & the second by the root
> > port.
> > Add two properties so that the translation tables in the root
> > port's
> > bridge layer can be configured to account for the translation done
> > by
> > the FPGA fabric.
>
> I'm skeptical that ranges/dma-ranges can't handle what you need.
> Anything in this area is going to need justification 'ranges doesn't
> work because x, y, z...'.
Cool.
>
> > Signed-off-by: Conor Dooley <conor.dooley@xxxxxxxxxxxxx>
> > Signed-off-by: Daire McNamara <daire.mcnamara@xxxxxxxxxxxxx>
> > ---
> > .../bindings/pci/microchip,pcie-host.yaml | 107
> > ++++++++++++++++++
> > 1 file changed, 107 insertions(+)
> >
> > diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > index 23d95c65acff..29bb1fe99a2e 100644
> > --- a/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > +++ b/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > @@ -71,6 +71,113 @@ properties:
> > minItems: 1
> > maxItems: 6
> >
> > + microchip,outbound-fabric-translation-ranges:
> > + $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > + minItems: 1
> > + maxItems: 32
> > + description: |
> > + The CPU-to-PCIe (outbound) address translation takes place
> > in two stages.
> > + Depending on the FPGA bitstream, the outbound address
> > translation tables
> > + in the PCIe root port's bridge layer will need to be
> > configured to account
> > + for only its part of the overall outbound address
> > translation.
> > +
> > + The first stage of outbound address translation occurs
> > between the CPU address
> > + and an intermediate "FPGA address". The second stage of
> > outbound address
> > + translation occurs between this FPGA address and the PCIe
> > address. Use this
> > + property, in conjunction with the ranges property, to divide
> > the overall
> > + address translation into these two stages so that the PCIe
> > address
> > + translation tables can be correctly configured.
>
> Sounds like you need 2 levels of ranges/dma-ranges.
>
> / {
> fpga-bus {
> ranges = ...
> dma-ranges = ...
> pcie@... {
> ranges = ...
> dma-ranges = ...
> };
> };
> };
>
Very possibly! I'd prefer that approach, tbh, if I can get it working.
I'll go back and try 2 levels again.
> > + If this property is present, one entry is required per
> > range. This is so
> > + FPGA designers can choose to route different address ranges
> > through different
> > + Fabric Interface Controllers and other logic as they see
> > fit.
> > +
> > + If this property is not present, the entire address
> > translation
> > + in any ranges property is attempted by the root port driver
> > via its outbound
> > + address translation tables.
> > +
> > + Each element in this property has three components. The
> > first is a
> > + PCIe address, the second is an FPGA address, and the third
> > is a size.
> > + These properties may be 32 or 64 bit values.
> > +
> > + In operation, the driver will expect a one-to-one
> > correspondance between
> > + range properties and this property. For each pair of range
> > and
> > + outbound-fabric-translation-range properties, the root port
> > driver will
> > + subtract the FPGA address in this property from the CPU
> > address in the
> > + corresponding range property and use the remainder to
> > program its
> > + outbound address translation tables.
> > +
> > + For each range, take its PCIe address and size - these are
> > the PCIe
> > + address & size for the element. The FPGA address is derived
> > from a given
> > + FPGA fabric design and is the address delivered by that FPGA
> > fabric
> > + design to the Core Complex. For a trivial configuration, it
> > is likely to be the
> > + lower 32 bits of the PCIe address in the range property and
> > the upper
> > + bits of the base address of the Fabric Interface Controller
> > the design uses.
> > + Otherwise, it is tightly coupled with the data path
> > configured in the
> > + FPGA fabric between the root port and the Core Complex.
> > +
> > + For more information on the tables, see Section 1.3.3,
> > + "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe
> > User Guide:
> > +
> > https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > + items:
> > + minItems: 3
> > + maxItems: 6
> > +
> > + microchip,inbound-fabric-translation-ranges:
> > + $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > + minItems: 1
> > + maxItems: 32
> > + description: |
> > + The PCIe-to-CPU (inbound) address translation takes place in
> > two stages.
> > + Depending on the FPGA bitstream, the inbound address
> > translation tables
> > + in the PCIe root port's bridge layer will need to be
> > configured to account
> > + for only its part of the overall inbound address
> > translation.
> > +
> > + The first stage of address translation occurs between the
> > PCIe address and
> > + an intermediate FPGA address. The second stage of address
> > translation
> > + occurs between the FPGA address and the CPU address. Use
> > this property
> > + in conjunction with the dma-ranges property to divide the
> > address
> > + translation into these two stages.
> > +
> > + If this property is present, one entry is required per dma-
> > range. This is so
> > + FPGA designers can choose to route different address ranges
> > through different
> > + Fabric Interface Controllers and other logic as they see
> > fit.
> > +
> > + If this property is not present, the entire address
> > translation
> > + in any dma-ranges property is attempted by the root port
> > driver via its
> > + inbound address translation tables.
> > +
> > + Each element in this property has three components. The
> > first is a
> > + PCIe address, the second is an FPGA address, and the third
> > is a size.
> > + These properties may be 32 or 64 bit values.
> > +
> > + In operation, the driver will expect a one-to-one
> > correspondance between
> > + dma-range properties and this property. For each pair of
> > dma-range and
> > + inbound-fabric-translation-range properties, the root port
> > driver will
> > + subtract the FPGA address in this property from the CPU
> > address in the
> > + corresponding dma-range property and use the remainder to
> > program its
> > + inbound address translation tables.
> > +
> > + From each dma-range, take its PCIe address and size - these
> > are the PCIe
> > + address & size for the element. The FPGA address is derived
> > from a given
> > + FPGA fabric design and is the address delivered by that FPGA
> > fabric
> > + design to the Core Complex. For a trivial configuration,
> > this property
> > + is unlikely to be required (i.e. no fabric translation on
> > the inbound
> > + interface). Otherwise, it is tightly coupled with the
> > inbound data path
> > + configured in the FPGA fabric between the root port and the
> > Core Complex.
> > + It is expected that more than one translation range may be
> > added to
> > + an FPGA fabric design, e.g. to deliver data to cached or
> > non-cached
> > + DDR.
> > +
> > + For more information on the tables, see Section 1.3.3,
> > + "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe
> > User Guide:
> > +
> > https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > + items:
> > + minItems: 4
> > + maxItems: 7
> > +
> > msi-controller:
> > description: Identifies the node as an MSI controller.
> >
> > --
> > 2.25.1
> >
