[PATCH 0/1] Request for Comments on SoftiWarp

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This patch set introduces the SoftiWarp driver.

Originally introduced to linux-rdma in 2011, SoftiWarp (siw) is
around as a project for a while. It aims at implementing the iWARP protocol
suite (IETF-RFC 5044/5041/5040/6581) on top of kernel TCP sockets.
It integrates with the linux-rdma  (or "OpenFabrics") framework. When our
initial attempt to contribute siw to Linux somehow sanded up, we
continued to use and further develop it as an open source project,
currently hosted at https://github.com/zrlio/softiwarp.

We found siw being useful, if applications are using
an RDMA API for communication (in particular, the ibverbs API), and
there is no local RDMA hardware support. This might be relevant for RDMA
application development and testing, as well as setups where
RDMA client applications at hosts without hardware RDMA support
want to communicate with servers which want to offload communication
to dedicated RDMA hardware. With the advent of NVME storage technology,
and NVME over Fabrics as a remote storage access method, use cases
may expand within that area (we recently added the driver support
needed to run siw at both NVMeF initiator and target side).
Last but not least, since supporting different (asynchronous, one sided)
communication patterns, we found pure siw client-server deployments can
outperform sockets based communication setups.

With all that, siw complements Soft-RoCE, also a pure software implementation
of an RDMA provider, but running the RoCE/RoCEv2 protocol. In contradiction
to Soft-RoCE, SoftiWarp per definition implements only the RC (reliable
connected) RDMA service, as defined by iWARP.

SoftiWarp comprises a kernel module and a user space library, both
plugging into the linux-rdma kernel environment, or libibverbs and librdmacm
respectively. It supports both kernel and user level RDMA applications.
For efficiency, user level communication endpoint resources such as
send/receive and completion queues are shared (memory mapped) between
siw kernel component and siw user library. We tested siw interoperability
with hardware iWARP products, such as Chelsio's T3/T4/T5/T6 adapters.

We approach the list with this RFC, since we are seeking for advice on
how to make siw ready for Linux acceptance. So, we are prepared -
and hoping for - constructive criticism on what we are presenting here.

The current code has limitations we are aware of. In particular, we are
currently working on:

1) IPv6 addressing support.

2) Revised debug code. The currently used siw specific debug macros seem to
   be obsolete and should probably get substituted by state of the art driver
   debug code.

3) NUMA awareness. We expect better performance results if siw
   would do a better job here.

4) Module parametrization. All settable driver parameters are currently set
   via module parameters. It might be a good idea to have those within /sys.

5) Transmit path implementation. We experimented with different approaches
   to implement a low latency transmit path. The current design of having
   one per CPU core transmit thread might be not the best idea. Advice
   is appreciated!

For the patch set we provide, we added siw as another software driver to
drivers/infiniband/sw.



To experiment with the current siw code, we suggest cloning our out-of-tree
development repo at https://github.com/zrlio/softiwarp.
Here, the branch 'dev-siw.mem_ext' contains the most recent code
development status, which is currently in sync with the patch.
Testing siw requires the installation of both user library and kernel
module, located within userlib/ and kernel/ respectively. That siw
version expects a running kernel of version >= 4.12.

>From Friday night on, I will be on vacation for two weeks with only
limited access to email. I will try to stay connected, and hope for a lively
discussion. I will of course address all comments latest when I am back on
October 24th, but I wanted to get the code out now - to get more eyes
on it.

Thank you,
Bernard.


Bernard Metzler (1):
  Adding SoftiWarp to the RDMA subsystem

 drivers/infiniband/Kconfig            |    1 +
 drivers/infiniband/sw/Makefile        |    1 +
 drivers/infiniband/sw/siw/Kconfig     |   18 +
 drivers/infiniband/sw/siw/Makefile    |   15 +
 drivers/infiniband/sw/siw/iwarp.h     |  381 ++++++
 drivers/infiniband/sw/siw/siw.h       |  785 ++++++++++++
 drivers/infiniband/sw/siw/siw_ae.c    |  113 ++
 drivers/infiniband/sw/siw/siw_cm.c    | 2270 +++++++++++++++++++++++++++++++++
 drivers/infiniband/sw/siw/siw_cm.h    |  154 +++
 drivers/infiniband/sw/siw/siw_cq.c    |  164 +++
 drivers/infiniband/sw/siw/siw_debug.c |  442 +++++++
 drivers/infiniband/sw/siw/siw_debug.h |  178 +++
 drivers/infiniband/sw/siw/siw_main.c  |  754 +++++++++++
 drivers/infiniband/sw/siw/siw_mem.c   |  403 ++++++
 drivers/infiniband/sw/siw/siw_obj.c   |  428 +++++++
 drivers/infiniband/sw/siw/siw_obj.h   |  113 ++
 drivers/infiniband/sw/siw/siw_qp.c    | 1172 +++++++++++++++++
 drivers/infiniband/sw/siw/siw_qp_rx.c | 1381 ++++++++++++++++++++
 drivers/infiniband/sw/siw/siw_qp_tx.c | 1342 +++++++++++++++++++
 drivers/infiniband/sw/siw/siw_verbs.c | 1933 ++++++++++++++++++++++++++++
 drivers/infiniband/sw/siw/siw_verbs.h |  119 ++
 include/uapi/rdma/siw_user.h          |  220 ++++
 22 files changed, 12387 insertions(+)
 create mode 100644 drivers/infiniband/sw/siw/Kconfig
 create mode 100644 drivers/infiniband/sw/siw/Makefile
 create mode 100644 drivers/infiniband/sw/siw/iwarp.h
 create mode 100644 drivers/infiniband/sw/siw/siw.h
 create mode 100644 drivers/infiniband/sw/siw/siw_ae.c
 create mode 100644 drivers/infiniband/sw/siw/siw_cm.c
 create mode 100644 drivers/infiniband/sw/siw/siw_cm.h
 create mode 100644 drivers/infiniband/sw/siw/siw_cq.c
 create mode 100644 drivers/infiniband/sw/siw/siw_debug.c
 create mode 100644 drivers/infiniband/sw/siw/siw_debug.h
 create mode 100644 drivers/infiniband/sw/siw/siw_main.c
 create mode 100644 drivers/infiniband/sw/siw/siw_mem.c
 create mode 100644 drivers/infiniband/sw/siw/siw_obj.c
 create mode 100644 drivers/infiniband/sw/siw/siw_obj.h
 create mode 100644 drivers/infiniband/sw/siw/siw_qp.c
 create mode 100644 drivers/infiniband/sw/siw/siw_qp_rx.c
 create mode 100644 drivers/infiniband/sw/siw/siw_qp_tx.c
 create mode 100644 drivers/infiniband/sw/siw/siw_verbs.c
 create mode 100644 drivers/infiniband/sw/siw/siw_verbs.h
 create mode 100644 include/uapi/rdma/siw_user.h

-- 
2.13.5

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux