On Tue, Oct 22, 2024 at 10:51:53AM +0900, Damien Le Moal wrote:
> On 10/22/24 07:19, Bjorn Helgaas wrote:
> > On Sat, Oct 12, 2024 at 08:32:40PM +0900, Damien Le Moal wrote:
> >> This series introduces the new functions pci_epc_mem_map() and
> >> pci_epc_mem_unmap() to improve handling of the PCI address mapping
> >> alignment constraints of endpoint controllers in a controller
> >> independent manner.
> >
> > Hi Damien, I know this is obvious to everybody except me, but who
> > uses this mapping? A driver running on the endpoint that does
> > MMIO? DMA (Memory Reads or Writes that target an endpoint BAR)?
> > I'm still trying to wrap my head around the whole endpoint driver
> > model.
>
> The mapping API is for mmio or DMA using endpoint controller memory
> mapped to a host PCI address range. It is not for BARs. BARs setup
> does not use the same API and has not changed with these patches.
>
> BARs can still be accessed on the endpoint (within the EPF driver)
> with regular READ_ONCE()/WRITE_ONCE() once they are set. But any
> data transfer between the PCI RC host and the EPF driver on the EP
> host that use mmio or DMA generic channel (memory copy offload
> channel) needs the new mapping API. DMA transfers that can be done
> using dedicated DMA rx/tx channels associated with the endpoint
> controller do not need to use this API as the mapping to the host
> PCI address space is automatically handled by the DMA driver.
Sorry I'm dense. I'm really not used to thinking in the endpoint
point of view. Correct me where I go off the rails:
- This code (pci_epc_mem_map()) runs on an endpoint, basically as
part of some endpoint firmware, right?
- On the endpoint's PCIe side, it receives memory read/write TLPs?
- These TLPs would be generated elsewhere in the PCIe fabric, e.g.,
by a driver on the host doing MMIO to the endpoint, or possibly
another endpoint doing peer-to-peer DMA.
- Mem read/write TLPs are routed by address, and the endpoint
accepts them if the address matches one of its BARs.
- This is a little different from a Root Complex, which would
basically treat reads/writes to anything outside the Root Port
windows as incoming DMA headed to physical memory.
- A Root Complex would use the TLP address (the PCI bus address)
directly as a CPU physical memory address unless the TLP address
is translated by an IOMMU.
- For the endpoint, you want the BAR to be an aperture to physical
memory in the address space of the SoC driving the endpoint.
- The SoC physical memory address may be different from the PCI but
address in the TLP, and pci_epc_mem_map() is the way to account
for this?
- IOMMU translations from PCI to CPU physical address space are
pretty arbitrary and needn't be contiguous on the CPU side.
- pci_epc_mem_map() sets up a conceptually similar PCI to CPU
address space translation, but it's much simpler because it
basically applies just a constant offset?
