Hi Frank, On 21/04/22 1:52 am, Zhi Li wrote: > On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@xxxxxxxxx> wrote: >> >> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@xxxxxx> wrote: >>> >>> Hi Frank Li, >>> >>> On 22/02/22 9:53 pm, Frank Li wrote: >>>> This implement NTB function for PCIe EP to RC connections. >>>> The existed ntb epf need two PCI EPs and two PCI Host. >>> >>> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio >>> layer instead of trying to build on NTB layer (which is specific to RC<->RC >>> communication). >>> >>> Are there any specific reasons for not taking that path? >> >> 1. EP side work as vHOST mode. vHost suppose access all memory of virtual io. >> But there are only map windows on the EP side to access RC side >> memory. You have to move >> map windows for each access. It is quite low efficiency. I'm not sure I quite get this. EP HW has limited outbound memory to access RC memory irrespective of how we implement it. This is not a SW framework limitation AFAICS. >> >> 2. So far as I know, virtio is still not DMA yet. CPU access PCI >> can't generate longer PCI TLP, >> So the speed is quite slow. NTB already has DMA support. If you use >> system level DMA, >> no change is needed at NTB level. If we want to use a PCI controller >> embedded DMA, some small >> changes need if based on my other Designware PCI eDMA patches, which >> are under review. Adding dmaengine API to do memcopy should be simple to add in vhost/virtio interface. >> >> 3. All the major data transfer of NTB is using write. Because TLP >> write needn't wait for complete, write >> performance is better than reading. On our platform, write >> performance is about 10% better than read. >> >> Frank > > Any Comments or rejection? @Kishon Vijay Abraham I I'd strongly recommend going with virtio/vhost based approach and standardizing it IMO. Thanks, Kishon > > best regards > Frank Li > >> >>> >>> Thanks, >>> Kishon >>> >>> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@xxxxxx >>>> >>>> This just need EP to RC connections. >>>> >>>> ┌────────────┐ ┌─────────────────────────────────────┐ >>>> │ │ │ │ >>>> ├────────────┤ │ ┌──────────────┤ >>>> │ NTB │ │ │ NTB │ >>>> │ NetDev │ │ │ NetDev │ >>>> ├────────────┤ │ ├──────────────┤ >>>> │ NTB │ │ │ NTB │ >>>> │ Transfer │ │ │ Transfer │ >>>> ├────────────┤ │ ├──────────────┤ >>>> │ │ │ │ │ >>>> │ PCI NTB │ │ │ │ >>>> │ EPF │ │ │ │ >>>> │ Driver │ │ │ PCI Virtual │ >>>> │ │ ├───────────────┐ │ NTB Driver │ >>>> │ │ │ PCI EP NTB │◄────►│ │ >>>> │ │ │ FN Driver │ │ │ >>>> ├────────────┤ ├───────────────┤ ├──────────────┤ >>>> │ │ │ │ │ │ >>>> │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ >>>> │ │ PCI │ │ │ BUS │ >>>> └────────────┘ └───────────────┴──────┴──────────────┘ >>>> PCI RC PCI EP >>>> >>>> >>>> >>>> Frank Li (4): >>>> PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address >>>> NTB: epf: Allow more flexibility in the memory BAR map method >>>> PCI: endpoint: Support NTB transfer between RC and EP >>>> Documentation: PCI: Add specification for the PCI vNTB function device >>>> >>>> Documentation/PCI/endpoint/index.rst | 2 + >>>> .../PCI/endpoint/pci-vntb-function.rst | 126 ++ >>>> Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ >>>> drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- >>>> .../pci/controller/dwc/pcie-designware-ep.c | 10 +- >>>> drivers/pci/endpoint/functions/Kconfig | 11 + >>>> drivers/pci/endpoint/functions/Makefile | 1 + >>>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ >>>> 8 files changed, 1775 insertions(+), 14 deletions(-) >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst >>>> create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c >>>>
