Hello, I have been exploring the possibility to add FPGA based PCI endpoint controller support within the current PCI Endpoint Framework [1]. The FPGA IP based controllers are mostly similar to the ones found in SoCs with one major difference. As they reside in the FPGA fabric and do not have direct access to the SoC main memory space they usually come with pre-assigned BAR mappings within their own memory space (memory space that can be accessed by the SoC CPU or softcore within the FPGA running Linux, however the physical addresses may be different). Therefore it is not possible to allocate memory from the CPU running linux with 'pci_epf_alloc_space' and set the endpoint controller to reroute the PCI read/write TLPs to that address with 'pci_epc_set_bar'. Therefore, I suggest to introduce 'pci_epc_get_bar' and the 'fixed_addr' endpoint controller bar feature to indicate that the BAR resides at a given address (usually set during the FPGA IP configuration before synthesis). As for providing this information from the FPGA IDE to Linux, the device tree can be used. This choice is at the discretion of the specific endpoint controller driver. The endpoint functions can query the endpoint controller driver on their BAR features, as is already the case with the features 'fixed_size' or 'only_64bit'. If the BAR is 'fixed_addr' (new feature) they call 'pci_epc_get_bar' instead of 'pci_epf_alloc_space' followed by 'pci_epc_set_bar'. 'pci_epc_get_bar' will fill the 'pci_epf_bar' structure with physical address and do an ioremap() to provide the 'pci_epf_bar.addr' virtual address. This change it would allow writing PCI endpoint controller drivers for Intel/Altera and AMD/Xilinx PCI endpoint IPs. The patches are for linux-next tag : next-20240719 I have successfully demonstrated that it is possible to write such a driver thanks to the new 'pci_epc_get_bar' function for the AMD/Xilinx PCI endpoint IP for ZynqUS+ devices [2,3]. And have successfully run the PCI Test Function [4] as well as NVMe endpoint functions I am working on. (Note: the function is called 'pci_epc_get_fixed_bar' in that example code and drivers). The new 'pci_epc_get_bar' function would also allow to write virtual / emulated PCI endpoint controllers that run on the host machine where both the PCI endpoint function and the driver for the PCI device on the virtual bus run on the same machine. For now this is not possible because the BARs backing memory allocated with 'pci_epf_alloc_space' are in RAM (non reserved part of RAM) and when a PCI driver will need to be loaded e.g., '/drivers/misc/pci_endpoint_test' for the PCI Test Function [4]. It will fail to request the regions for the BARs in '__pci_request_region' (drivers/pci/pci.c) because the memory space allocated is in RAM and will cause a conflict. Normally this problem does not appear because the endpoint function driver and the PCI driver are not running on the same machine (the function driver is running in the endpoint, e.g., an SoC connected to the host PC via PCIe and the PCI driver is running on the host PC). With the new 'pci_epc_get_bar' function one could mark part of the RAM as reserved memory and use it for the BARs, and then because it is reserved memory when loading the PCI driver, there will be no conflict in '__pci_request_region'. This would allow to run the PCI endpoint functions on the host. This is useful for research and development of new functions and very similar to the NVMeVirt project [5], where they emulate an NVMe PCI device on the host PC, in RAM. With the proposed changes I could introduce a "virtual PCI endpoint controller" driver that would allow to run the endpoint functions available in the Linux PCI Endpoint Framework [1]. This would allow for a much lighter setup to develop new functions without the need of an SoC board for most of the development. The suggested change does not break any compatibility with the existing endpoint controllers drivers or endpoint function drivers and could be introduced without further changes. I provided the changes in the 'pci-epf-test' driver to show how endpoint function drivers would take advantage of this new function to be able to become compatible with endpoint controller drivers with fixed address BARs. Cheers. Rick [1] https://www.kernel.org/doc/html/latest/PCI/endpoint/index.html [2] https://github.com/rick-heig/nvme_csd/tree/main/platforms/zcu106 [3] driver (Linux 6.5) https://github.com/rick-heig/linux-xlnx/blob/csd_20231212/drivers/pci/controller/pcie-xilinx-ep.c [4] https://www.kernel.org/doc/html/latest/PCI/endpoint/pci-test-function.html [5] https://github.com/snu-csl/nvmevirt Rick Wertenbroek (2): PCI: endpoint: Introduce 'get_bar' to map fixed address BARs in EPC PCI: endpoint: pci-epf-test: Add support for controller with fixed addr BARs drivers/pci/endpoint/functions/pci-epf-test.c | 38 +++++++++++++++++-- drivers/pci/endpoint/pci-epc-core.c | 37 ++++++++++++++++++ include/linux/pci-epc.h | 7 ++++ 3 files changed, 78 insertions(+), 4 deletions(-) -- 2.25.1
