> -----Original Message-----
> From: Cheng Xu <chengyou.xc@xxxxxxxxxxxxxxx>
> Sent: Thursday, 17 February 2022 04:01
> To: jgg@xxxxxxxx; dledford@xxxxxxxxxx
> Cc: leon@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx;
> KaiShen@xxxxxxxxxxxxxxxxx; chengyou@xxxxxxxxxxxxxxxxx;
> tonylu@xxxxxxxxxxxxxxxxx
> Subject: [EXTERNAL] [PATCH for-next v3 00/12] Elastic RDMA Adapter (ERDMA)
> driver
>
> From: Cheng Xu <chengyou@xxxxxxxxxxxxxxxxx>
>
> Hello all,
>
> This v3 patch set introduces the Elastic RDMA Adapter (ERDMA) driver,
> which released in Apsara Conference 2021 by Alibaba. The PR of ERDMA
> userspace provider has already been created [1].
>
Is this only at my side? I did not get [09/12] of that v3 set.
If yes, I'll find a way to comment on that as well if
needed. I see it at patchwork.
I am traveling these days, therefore cannot comment before mid
next week.
Thanks,
Bernard.
> ERDMA enables large-scale RDMA acceleration capability in Alibaba ECS
> environment, initially offered in g7re instance. It can improve the
> efficiency of large-scale distributed computing and communication
> significantly and expand dynamically with the cluster scale of Alibaba
> Cloud.
>
> ERDMA is a RDMA networking adapter based on the Alibaba MOC hardware. It
> works in the VPC network environment (overlay network), and uses iWarp
> tranport protocol. ERDMA supports reliable connection (RC). ERDMA also
> supports both kernel space and user space verbs. Now we have already
> supported HPC/AI applications with libfabric, NoF and some other internal
> verbs libraries, such as xrdma, epsl, etc,.
>
> For the ECS instance with RDMA enabled, our MOC hardware generates two
> kinds of PCI devices: one for ERDMA, and one for the original net device
> (virtio-net). They are separated PCI devices, using "rdma link" command
> with a filter inside our rdma_link_ops.newlink implementation can bind
> them together properly.
>
> Besides, this patchset contains a change in iw_query_port to fix this
> issue [2]. This change lets the device drivers decide the return value of
> iw_query_port when attached netdev is NULL. After this change, erdma can
> register device successfully in pci probe function, and keep port state
> invalid until a netdev is binded to it.
>
> Fixed issues or changes in v3:
> - Change char limit of column from 100 to 80.
> - Remove unnecessary field or structure definitions in erdma.h.
> - Use exactly type (bool, unsigned int) instead of "int" in erdma_dev.
> - Make ibdev and pci device having the same lifecycle. ERDMA will remain
> an invalid port state until binded to the corresponding netdev.
> - ib_core: allow query_port when netdev is NULL for iWarp device.
> - Move large inline function in erdma.h to .c files.
> - Use dev_{info, warn, err} or ibdev_{info, warn, err} instead of
> pr_{info, warn, err} function calls.
> - Remove print function calls in userspace-triggered paths.
> - Add necessary comments in CM part.
> - Remove unused entries in map_cqe_opcode[] table.
> - Use rdma_is_kernel_res instead of self-definitions.
> - Remove unsed resources counter in erdma_dev.
> - Use pgprot_device instead of pgprot_noncached in erdma_mmap.
> - Remove disassociate_ucontext interface implementation
>
> Fixed issues in v2:
> - No "extern" to function declarations.
> - No inline functions in .c files, no void casting for functions with
> return values.
> - Based on siw's newest kernel version, rewrite the code (mainly CM and
> CM related part) which originally based on an old siw version.
> version.
> - remove debugfs.
> - fix issues reported by kernel test rebot.
> - Using RDMA_NLDEV_CMD_NEWLINK instead of binding in net notifiers.
>
> [1] INVALID URI REMOVED
> 2Drdma_rdma-2Dcore_pull_1126&d=DwIDAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-
> r8ZO1PP1alNwU_QJcRRLfmYTAgd3QCvqSc&m=9gxrBXG7blRayxtoienSVg7YyPPJMAAu4E7n8
> f0iVb0&s=kts7JTxieb-ucOD3UuuDRc5Uj11VfZzDaGgM4sqSja8&e=
> [2] INVALID URI REMOVED
> 3A__lore.kernel.org_all_20220118141324.GF8034-
> 40ziepe.ca_&d=DwIDAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-
> r8ZO1PP1alNwU_QJcRRLfmYTAgd3QCvqSc&m=9gxrBXG7blRayxtoienSVg7YyPPJMAAu4E7n8
> f0iVb0&s=pVyZk1TUR5kwTKlSwxvwRyqKCt8ABKRxNf5Qze860Lc&e=
>
> Thanks,
> Cheng Xu
>
> Cheng Xu (12):
> RDMA: Add ERDMA to rdma_driver_id definition
> RDMA/core: Allow calling query_port when netdev isn't attached in
> iWarp
> RDMA/erdma: Add the hardware related definitions
> RDMA/erdma: Add main include file
> RDMA/erdma: Add cmdq implementation
> RDMA/erdma: Add event queue implementation
> RDMA/erdma: Add verbs header file
> RDMA/erdma: Add verbs implementation
> RDMA/erdma: Add connection management (CM) support
> RDMA/erdma: Add the erdma module
> RDMA/erdma: Add the ABI definitions
> RDMA/erdma: Add driver to kernel build environment
>
> MAINTAINERS | 8 +
> drivers/infiniband/Kconfig | 1 +
> drivers/infiniband/core/device.c | 7 +-
> drivers/infiniband/hw/Makefile | 1 +
> drivers/infiniband/hw/erdma/Kconfig | 10 +
> drivers/infiniband/hw/erdma/Makefile | 4 +
> drivers/infiniband/hw/erdma/erdma.h | 288 ++++
> drivers/infiniband/hw/erdma/erdma_cm.c | 1440 ++++++++++++++++++++
> drivers/infiniband/hw/erdma/erdma_cm.h | 167 +++
> drivers/infiniband/hw/erdma/erdma_cmdq.c | 512 ++++++++
> drivers/infiniband/hw/erdma/erdma_cq.c | 202 +++
> drivers/infiniband/hw/erdma/erdma_eq.c | 366 ++++++
> drivers/infiniband/hw/erdma/erdma_hw.h | 480 +++++++
> drivers/infiniband/hw/erdma/erdma_main.c | 629 +++++++++
> drivers/infiniband/hw/erdma/erdma_qp.c | 567 ++++++++
> drivers/infiniband/hw/erdma/erdma_verbs.c | 1447 +++++++++++++++++++++
> drivers/infiniband/hw/erdma/erdma_verbs.h | 345 +++++
> include/uapi/rdma/erdma-abi.h | 49 +
> include/uapi/rdma/ib_user_ioctl_verbs.h | 1 +
> 19 files changed, 6523 insertions(+), 1 deletion(-)
> create mode 100644 drivers/infiniband/hw/erdma/Kconfig
> create mode 100644 drivers/infiniband/hw/erdma/Makefile
> create mode 100644 drivers/infiniband/hw/erdma/erdma.h
> create mode 100644 drivers/infiniband/hw/erdma/erdma_cm.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_cm.h
> create mode 100644 drivers/infiniband/hw/erdma/erdma_cmdq.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_cq.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_eq.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_hw.h
> create mode 100644 drivers/infiniband/hw/erdma/erdma_main.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_qp.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_verbs.c
> create mode 100644 drivers/infiniband/hw/erdma/erdma_verbs.h
> create mode 100644 include/uapi/rdma/erdma-abi.h
>
> --
> 2.27.0
