Hi all,
here is V5 of the RTRS (former IBTRS) rdma transport library and the
corresponding RNBD (former IBNBD) rdma network block device.
Main changes are the following:
1. Fix the security problem pointed out by Jason
2. Implement code-style/readability/API/etc suggestions by Bart van Assche
3. Rename IBTRS and IBNBD to RTRS and RNBD accordingly
4. Fileio mode support in rnbd-srv has been removed.
The main functional change is a fix for the security problem pointed out by
Jason and discussed both on the mailing list and during the last LPC RDMA MC 2019.
On the server side we now invalidate in RTRS each rdma buffer before we hand it
over to RNBD server and in turn to the block layer. A new rkey is generated and
registered for the buffer after it returns back from the block layer and RNBD
server. The new rkey is sent back to the client along with the IO result.
The procedure is the default behaviour of the driver. This invalidation and
registration on each IO causes performance drop of up to 20%. A user of the
driver may choose to load the modules with this mechanism switched off
(always_invalidate=N), if he understands and can take the risk of a malicious
client being able to corrupt memory of a server it is connected to. This might
be a reasonable option in a scenario where all the clients and all the servers
are located within a secure datacenter.
Huge thanks to Bart van Assche for the very detailed review of both RNBD and
RTRS. These included suggestions for style fixes, better readability and
documentation, code simplifications, eliminating usage of deprecated APIs,
too many to name.
The transport library and the network block device using it have been renamed to
RTRS and RNBD accordingly in order to reflect the fact that they are based on
the rdma subsystem and not bound to InfiniBand only.
Fileio mode support in rnbd-server is not so efficent as pointed out by Bart,
and we can use loop device in between if there is need, hence we just
removed the fileio mode support.
Introduction
-------------
RTRS (RDMA Transport) is a reliable high speed transport library
which allows for establishing connection between client and server
machines via RDMA. It is based on RDMA-CM, so expect also to support RoCE
and iWARP, but we mainly tested in IB environment. It is optimized to
transfer (read/write) IO blocks in the sense that it follows the BIO
semantics of providing the possibility to either write data from a
scatter-gather list to the remote side or to request ("read") data
transfer from the remote side into a given set of buffers.
RTRS is multipath capable and provides I/O fail-over and load-balancing
functionality, i.e. in RTRS terminology, an RTRS path is a set of RDMA
connections and particular path is selected according to the load-balancing policy.
It can be used for other components not bind to RNBD.
RNBD (InfiniBand Network Block Device) is a pair of kernel modules
(client and server) that allow for remote access of a block device on
the server over RTRS protocol. After being mapped, the remote block
devices can be accessed on the client side as local block devices.
Internally RNBD uses RTRS as an RDMA transport library.
Commits for kernel can be found here:
https://github.com/ionos-enterprise/ibnbd/commits/linux-5.5-rc2-ibnbd-v5
The out-of-tree modules are here:
https://github.com/ionos-enterprise/ibnbd
LPC 2019 presentation:
https://linuxplumbersconf.org/event/4/contributions/367/attachments/331/555/LPC_2019_RMDA_MC_IBNBD_IBTRS_Upstreaming.pdf
Performance results for the v5.5-rc1 kernel
link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1
Jack Wang (25):
sysfs: export sysfs_remove_file_self()
rtrs: public interface header to establish RDMA connections
rtrs: private headers with rtrs protocol structs and helpers
rtrs: core: lib functions shared between client and server modules
rtrs: client: private header with client structs and functions
rtrs: client: main functionality
rtrs: client: statistics functions
rtrs: client: sysfs interface functions
rtrs: server: private header with server structs and functions
rtrs: server: main functionality
rtrs: server: statistics functions
rtrs: server: sysfs interface functions
rtrs: include client and server modules into kernel compilation
rtrs: a bit of documentation
rnbd: private headers with rnbd protocol structs and helpers
rnbd: client: private header with client structs and functions
rnbd: client: main functionality
rnbd: client: sysfs interface functions
rnbd: server: private header with server structs and functions
rnbd: server: main functionality
rnbd: server: functionality for IO submission to file or block dev
rnbd: server: sysfs interface functions
rnbd: include client and server modules into kernel compilation
rnbd: a bit of documentation
MAINTAINERS: Add maintainers for RNBD/RTRS modules
Documentation/ABI/testing/sysfs-block-rnbd | 51 +
.../ABI/testing/sysfs-class-rnbd-client | 117 +
.../ABI/testing/sysfs-class-rnbd-server | 57 +
.../ABI/testing/sysfs-class-rtrs-client | 190 ++
.../ABI/testing/sysfs-class-rtrs-server | 81 +
MAINTAINERS | 14 +
drivers/block/Kconfig | 2 +
drivers/block/Makefile | 1 +
drivers/block/rnbd/Kconfig | 28 +
drivers/block/rnbd/Makefile | 17 +
drivers/block/rnbd/README | 80 +
drivers/block/rnbd/rnbd-clt-sysfs.c | 659 ++++
drivers/block/rnbd/rnbd-clt.c | 1761 ++++++++++
drivers/block/rnbd/rnbd-clt.h | 169 +
drivers/block/rnbd/rnbd-common.c | 43 +
drivers/block/rnbd/rnbd-log.h | 61 +
drivers/block/rnbd/rnbd-proto.h | 325 ++
drivers/block/rnbd/rnbd-srv-dev.c | 162 +
drivers/block/rnbd/rnbd-srv-dev.h | 130 +
drivers/block/rnbd/rnbd-srv-sysfs.c | 234 ++
drivers/block/rnbd/rnbd-srv.c | 882 +++++
drivers/block/rnbd/rnbd-srv.h | 99 +
drivers/infiniband/Kconfig | 1 +
drivers/infiniband/ulp/Makefile | 1 +
drivers/infiniband/ulp/rtrs/Kconfig | 27 +
drivers/infiniband/ulp/rtrs/Makefile | 17 +
drivers/infiniband/ulp/rtrs/README | 137 +
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 453 +++
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 519 +++
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2952 +++++++++++++++++
drivers/infiniband/ulp/rtrs/rtrs-clt.h | 314 ++
drivers/infiniband/ulp/rtrs/rtrs-log.h | 50 +
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 426 +++
drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c | 109 +
drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 315 ++
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 2187 ++++++++++++
drivers/infiniband/ulp/rtrs/rtrs-srv.h | 159 +
drivers/infiniband/ulp/rtrs/rtrs.c | 646 ++++
drivers/infiniband/ulp/rtrs/rtrs.h | 334 ++
fs/sysfs/file.c | 1 +
40 files changed, 13811 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-block-rnbd
create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-client
create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-server
create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-client
create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-server
create mode 100644 drivers/block/rnbd/Kconfig
create mode 100644 drivers/block/rnbd/Makefile
create mode 100644 drivers/block/rnbd/README
create mode 100644 drivers/block/rnbd/rnbd-clt-sysfs.c
create mode 100644 drivers/block/rnbd/rnbd-clt.c
create mode 100644 drivers/block/rnbd/rnbd-clt.h
create mode 100644 drivers/block/rnbd/rnbd-common.c
create mode 100644 drivers/block/rnbd/rnbd-log.h
create mode 100644 drivers/block/rnbd/rnbd-proto.h
create mode 100644 drivers/block/rnbd/rnbd-srv-dev.c
create mode 100644 drivers/block/rnbd/rnbd-srv-dev.h
create mode 100644 drivers/block/rnbd/rnbd-srv-sysfs.c
create mode 100644 drivers/block/rnbd/rnbd-srv.c
create mode 100644 drivers/block/rnbd/rnbd-srv.h
create mode 100644 drivers/infiniband/ulp/rtrs/Kconfig
create mode 100644 drivers/infiniband/ulp/rtrs/Makefile
create mode 100644 drivers/infiniband/ulp/rtrs/README
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-log.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-pri.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.h
--
2.17.1
