On Tue, Oct 22, 2024 at 10:51:53AM +0900, Damien Le Moal wrote: > On 10/22/24 07:19, Bjorn Helgaas wrote: > > On Sat, Oct 12, 2024 at 08:32:40PM +0900, Damien Le Moal wrote: > >> This series introduces the new functions pci_epc_mem_map() and > >> pci_epc_mem_unmap() to improve handling of the PCI address mapping > >> alignment constraints of endpoint controllers in a controller > >> independent manner. > > > > Hi Damien, I know this is obvious to everybody except me, but who > > uses this mapping? A driver running on the endpoint that does > > MMIO? DMA (Memory Reads or Writes that target an endpoint BAR)? > > I'm still trying to wrap my head around the whole endpoint driver > > model. > > The mapping API is for mmio or DMA using endpoint controller memory > mapped to a host PCI address range. It is not for BARs. BARs setup > does not use the same API and has not changed with these patches. > > BARs can still be accessed on the endpoint (within the EPF driver) > with regular READ_ONCE()/WRITE_ONCE() once they are set. But any > data transfer between the PCI RC host and the EPF driver on the EP > host that use mmio or DMA generic channel (memory copy offload > channel) needs the new mapping API. DMA transfers that can be done > using dedicated DMA rx/tx channels associated with the endpoint > controller do not need to use this API as the mapping to the host > PCI address space is automatically handled by the DMA driver. Sorry I'm dense. I'm really not used to thinking in the endpoint point of view. Correct me where I go off the rails: - This code (pci_epc_mem_map()) runs on an endpoint, basically as part of some endpoint firmware, right? - On the endpoint's PCIe side, it receives memory read/write TLPs? - These TLPs would be generated elsewhere in the PCIe fabric, e.g., by a driver on the host doing MMIO to the endpoint, or possibly another endpoint doing peer-to-peer DMA. - Mem read/write TLPs are routed by address, and the endpoint accepts them if the address matches one of its BARs. - This is a little different from a Root Complex, which would basically treat reads/writes to anything outside the Root Port windows as incoming DMA headed to physical memory. - A Root Complex would use the TLP address (the PCI bus address) directly as a CPU physical memory address unless the TLP address is translated by an IOMMU. - For the endpoint, you want the BAR to be an aperture to physical memory in the address space of the SoC driving the endpoint. - The SoC physical memory address may be different from the PCI but address in the TLP, and pci_epc_mem_map() is the way to account for this? - IOMMU translations from PCI to CPU physical address space are pretty arbitrary and needn't be contiguous on the CPU side. - pci_epc_mem_map() sets up a conceptually similar PCI to CPU address space translation, but it's much simpler because it basically applies just a constant offset?