On 12/21/21 9:09 PM, Leon Romanovsky wrote:
On Tue, Dec 21, 2021 at 10:48:47AM +0800, Cheng Xu wrote:
Hello all,
This patch set introduces the Elastic RDMA Adapter (ERDMA) driver, which
released in Apsara Conference 2021 by Alibaba.
ERDMA enables large-scale RDMA acceleration capability in Alibaba ECS
environment, initially offered in g7re instance. It can improve the
efficiency of large-scale distributed computing and communication
significantly and expand dynamically with the cluster scale of Alibaba
Cloud.
ERDMA is a RDMA networking adapter based on the Alibaba MOC hardware. It
works in the VPC network environment (overlay network), and uses iWarp
tranport protocol. ERDMA supports reliable connection (RC). ERDMA also
supports both kernel space and user space verbs. Now we have already
supported HPC/AI applications with libfabric, NoF and some other internal
verbs libraries, such as xrdma, epsl, etc,.
We will need to get erdma provider implementation in the rdma-core too,
in order to consider to merge it.
Sure, I will submit erdma userspace provider implementation within 2
days.
For the ECS instance with RDMA enabled, there are two kinds of devices
allocated, one for ERDMA, and one for the original netdev (virtio-net).
They are different PCI deivces. ERDMA driver can get the information about
which netdev attached to in its PCIe barspace (by MAC address matching).
This is very questionable. The netdev part should be kept in the
drivers/ethernet/... part of the kernel.
Thanks
The net device used in Alibaba ECS instance is virtio-net device, driven
by virtio-pci/virtio-net drivers. ERDMA device does not need its own net
device, and will be attached to an existed virtio-net device. The
relationship between ibdev and netdev in erdma is similar to siw/rxe.
Thanks,
Cheng Xu
Cheng Xu (11):
RDMA: Add ERDMA to rdma_driver_id definition
RDMA/erdma: Add the hardware related definitions
RDMA/erdma: Add main include file
RDMA/erdma: Add cmdq implementation
RDMA/erdma: Add event queue implementation
RDMA/erdma: Add verbs header file
RDMA/erdma: Add verbs implementation
RDMA/erdma: Add connection management (CM) support
RDMA/erdma: Add the erdma module
RDMA/erdma: Add the ABI definitions
RDMA/erdma: Add driver to kernel build environment
MAINTAINERS | 8 +
drivers/infiniband/Kconfig | 1 +
drivers/infiniband/hw/Makefile | 1 +
drivers/infiniband/hw/erdma/Kconfig | 10 +
drivers/infiniband/hw/erdma/Makefile | 5 +
drivers/infiniband/hw/erdma/erdma.h | 381 +++++
drivers/infiniband/hw/erdma/erdma_cm.c | 1585 +++++++++++++++++++++
drivers/infiniband/hw/erdma/erdma_cm.h | 158 ++
drivers/infiniband/hw/erdma/erdma_cmdq.c | 489 +++++++
drivers/infiniband/hw/erdma/erdma_cq.c | 201 +++
drivers/infiniband/hw/erdma/erdma_debug.c | 314 ++++
drivers/infiniband/hw/erdma/erdma_debug.h | 18 +
drivers/infiniband/hw/erdma/erdma_eq.c | 346 +++++
drivers/infiniband/hw/erdma/erdma_hw.h | 474 ++++++
drivers/infiniband/hw/erdma/erdma_main.c | 711 +++++++++
drivers/infiniband/hw/erdma/erdma_qp.c | 624 ++++++++
drivers/infiniband/hw/erdma/erdma_verbs.c | 1477 +++++++++++++++++++
drivers/infiniband/hw/erdma/erdma_verbs.h | 366 +++++
include/uapi/rdma/erdma-abi.h | 49 +
include/uapi/rdma/ib_user_ioctl_verbs.h | 1 +
20 files changed, 7219 insertions(+)
create mode 100644 drivers/infiniband/hw/erdma/Kconfig
create mode 100644 drivers/infiniband/hw/erdma/Makefile
create mode 100644 drivers/infiniband/hw/erdma/erdma.h
create mode 100644 drivers/infiniband/hw/erdma/erdma_cm.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_cm.h
create mode 100644 drivers/infiniband/hw/erdma/erdma_cmdq.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_cq.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_debug.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_debug.h
create mode 100644 drivers/infiniband/hw/erdma/erdma_eq.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_hw.h
create mode 100644 drivers/infiniband/hw/erdma/erdma_main.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_qp.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_verbs.c
create mode 100644 drivers/infiniband/hw/erdma/erdma_verbs.h
create mode 100644 include/uapi/rdma/erdma-abi.h
--
2.27.0
