On Fri, Apr 22, 2022 at 10:15 AM Kishon Vijay Abraham I <kishon@xxxxxx> wrote: > > Hi Frank, > > On 21/04/22 1:52 am, Zhi Li wrote: > > On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@xxxxxxxxx> wrote: > >> > >> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@xxxxxx> wrote: > >>> > >>> Hi Frank Li, > >>> > >>> On 22/02/22 9:53 pm, Frank Li wrote: > >>>> This implement NTB function for PCIe EP to RC connections. > >>>> The existed ntb epf need two PCI EPs and two PCI Host. > >>> > >>> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio > >>> layer instead of trying to build on NTB layer (which is specific to RC<->RC > >>> communication). > >>> > >>> Are there any specific reasons for not taking that path? > >> > >> 1. EP side work as vHOST mode. vHost suppose access all memory of virtual io. > >> But there are only map windows on the EP side to access RC side > >> memory. You have to move > >> map windows for each access. It is quite low efficiency. > > I'm not sure I quite get this. EP HW has limited outbound memory to access RC > memory irrespective of how we implement it. This is not a SW framework > limitation AFAICS. Almost all EP HW have limited outbound memory windows to access RC. We face transfer efficiency problems if we stick into vhost. > >> > >> 2. So far as I know, virtio is still not DMA yet. CPU access PCI > >> can't generate longer PCI TLP, > >> So the speed is quite slow. NTB already has DMA support. If you use > >> system level DMA, > >> no change is needed at NTB level. If we want to use a PCI controller > >> embedded DMA, some small > >> changes need if based on my other Designware PCI eDMA patches, which > >> are under review. > > Adding dmaengine API to do memcopy should be simple to add in vhost/virtio > interface. > >> > >> 3. All the major data transfer of NTB is using write. Because TLP > >> write needn't wait for complete, write > >> performance is better than reading. On our platform, write > >> performance is about 10% better than read. > >> > >> Frank > > > > Any Comments or rejection? @Kishon Vijay Abraham I > > I'd strongly recommend going with virtio/vhost based approach and standardizing > it IMO. But No progress in recent years on this path. At least, my patches make PCIe EP work as enet with minimized change. And NTB don't conflict with virtio/vhost solution. Frank > > Thanks, > Kishon > > > > > best regards > > Frank Li > > > >> > >>> > >>> Thanks, > >>> Kishon > >>> > >>> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@xxxxxx > >>>> > >>>> This just need EP to RC connections. > >>>> > >>>> ┌────────────┐ ┌─────────────────────────────────────┐ > >>>> │ │ │ │ > >>>> ├────────────┤ │ ┌──────────────┤ > >>>> │ NTB │ │ │ NTB │ > >>>> │ NetDev │ │ │ NetDev │ > >>>> ├────────────┤ │ ├──────────────┤ > >>>> │ NTB │ │ │ NTB │ > >>>> │ Transfer │ │ │ Transfer │ > >>>> ├────────────┤ │ ├──────────────┤ > >>>> │ │ │ │ │ > >>>> │ PCI NTB │ │ │ │ > >>>> │ EPF │ │ │ │ > >>>> │ Driver │ │ │ PCI Virtual │ > >>>> │ │ ├───────────────┐ │ NTB Driver │ > >>>> │ │ │ PCI EP NTB │◄────►│ │ > >>>> │ │ │ FN Driver │ │ │ > >>>> ├────────────┤ ├───────────────┤ ├──────────────┤ > >>>> │ │ │ │ │ │ > >>>> │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > >>>> │ │ PCI │ │ │ BUS │ > >>>> └────────────┘ └───────────────┴──────┴──────────────┘ > >>>> PCI RC PCI EP > >>>> > >>>> > >>>> > >>>> Frank Li (4): > >>>> PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > >>>> NTB: epf: Allow more flexibility in the memory BAR map method > >>>> PCI: endpoint: Support NTB transfer between RC and EP > >>>> Documentation: PCI: Add specification for the PCI vNTB function device > >>>> > >>>> Documentation/PCI/endpoint/index.rst | 2 + > >>>> .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > >>>> Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > >>>> drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > >>>> .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > >>>> drivers/pci/endpoint/functions/Kconfig | 11 + > >>>> drivers/pci/endpoint/functions/Makefile | 1 + > >>>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > >>>> 8 files changed, 1775 insertions(+), 14 deletions(-) > >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > >>>> create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > >>>>
