Hi Doug, Hi Jason, Hi Jens, Hi All,
since we didn't get any new comments for the V8 prepared by Jack a
week ago do you think rnbd/rtrs could be merged in the current merge
window?
Best Regards,
Danil
On Fri, Jan 24, 2020 at 9:47 PM Jack Wang <jinpuwang@xxxxxxxxx> wrote:
>
> Hi all,
>
> Here is v8 of the RTRS (former IBTRS) RDMA Transport Library and the
> corresponding RNBD (former IBNBD) RDMA Network Block Device, which includes
> changes to address comments from the community.
>
> Introduction
> -------------
>
> RTRS (RDMA Transport) is a reliable high speed transport library
> which allows for establishing connection between client and server
> machines via RDMA. It is based on RDMA-CM, so expect also to support RoCE
> and iWARP, but we mainly tested in IB environment. It is optimized to
> transfer (read/write) IO blocks in the sense that it follows the BIO
> semantics of providing the possibility to either write data from a
> scatter-gather list to the remote side or to request ("read") data
> transfer from the remote side into a given set of buffers.
>
> RTRS is multipath capable and provides I/O fail-over and load-balancing
> functionality, i.e. in RTRS terminology, an RTRS path is a set of RDMA
> connections and particular path is selected according to the load-balancing
> policy. It can be used for other components beside RNBD.
>
> Module parameter always_invalidate is introduced for the security problem
> discussed in LPC RDMA MC 2019. When always_invalidate=Y, on the server side we
> invalidate each rdma buffer before we hand it over to RNBD server and
> then pass it to the block layer. A new rkey is generated and registered for the
> buffer after it returns back from the block layer and RNBD server.
> The new rkey is sent back to the client along with the IO result.
> The procedure is the default behaviour of the driver. This invalidation and
> registration on each IO causes performance drop of up to 20%. A user of the
> driver may choose to load the modules with this mechanism switched off
> (always_invalidate=N), if he understands and can take the risk of a malicious
> client being able to corrupt memory of a server it is connected to. This might
> be a reasonable option in a scenario where all the clients and all the servers
> are located within a secure datacenter.
>
> RNBD (RDMA Network Block Device) is a pair of kernel modules
> (client and server) that allow for remote access of a block device on
> the server over RTRS protocol. After being mapped, the remote block
> devices can be accessed on the client side as local block devices.
> Internally RNBD uses RTRS as an RDMA transport library.
>
> Commits for kernel can be found here:
> https://github.com/ionos-enterprise/ibnbd/commits/linux-5.5-rc6-ibnbd-v7
>
> Testing
> -------
>
> All the changes have been tested with our regression testsuite in our staging environment
> in IONOS data center. it's around 200 testcases, for both always_invalidate=N and
> always_invalidate=Y configurations.
>
> Changelog
> ---------
> v8:
> o Rebased to linux-5.5-rc7
> o Reviewed likey/unlikely usage, only keep the one in IO path suggested by Leon Romanovsky
> o Reviewed inline usage, remove inline for functions longer than 5 lines of code suggested by Leon
> o Removed 2 WARN_ON suggested by Leon
> o Removed 2 empty lines between copyright suggested by Leon
> o Makefile: remove compat include for upstream suggested by Leon
> o rtrs-clt: remove module paramters suggested by Leon
> o drop rnbd_clt_dev_is_mapped
> o rnbd-clt: clean up rnbd_rerun_if_needed
> o rtrs-srv: remove reset_all sysfs
> o rtrs stats: remove wc_completion stats
> o rtrs-clt: enhance doc for rtrs_clt_change_state
> o rtrs-clt: remove unused rtrs_permit_from_pdu
> v7:
> o Rebased to linux-5.5-rc6
> o Implement code-style/readability/API/Documentation etc suggestions by Bart van Assche
> o Make W=1 clean
> o New benchmark results for Mellanox ConnectX-5
> o second try adding MAINTAINERS entries in alphabetical order as Gal Pressman suggested
> * https://lore.kernel.org/linux-block/20200116125915.14815-1-jinpuwang@xxxxxxxxx/
> v6:
> o Rebased to linux-5.5-rc4
> o Fix typo in my email address in first patch
> o Cleanup copyright as suggested by Leon Romanovsky
> o Remove 2 redudant kobject_del in error path as suggested by Leon Romanovsky
> o Add MAINTAINERS entries in alphabetical order as Gal Pressman suggested
> * https://lore.kernel.org/linux-block/20191230102942.18395-1-jinpuwang@xxxxxxxxx/
> v5:
> o Fix the security problem pointed out by Jason
> o Implement code-style/readability/API/etc suggestions by Bart van Assche
> o Rename IBTRS and IBNBD to RTRS and RNBD accordingly
> o Fileio mode support in rnbd-srv has been removed.
> * https://lore.kernel.org/linux-block/20191220155109.8959-1-jinpuwang@xxxxxxxxx/
> v4:
> o Protocol extended to transport IO priorities
> o Support for Mellanox ConnectX-4/X-5
> o Minor sysfs extentions (display access mode on server side)
> o Bug fixes: cleaning up sysfs folders, race on deallocation of resources
> o Style fixes
> * https://lore.kernel.org/linux-block/20190620150337.7847-1-jinpuwang@xxxxxxxxx/
> v3:
> o Sparse fixes:
> - le32 -> le16 conversion
> - pcpu and RCU wrong declaration
> - sysfs: dynamically alloc array of sockaddr structures to reduce
> size of a stack frame
> o Rename sysfs folder on client and server sides to show source and
> destination addresses of the connection, i.e.:
> .../<session-name>/paths/<src@dst>/
> o Remove external inclusions from Makefiles.
> * https://lore.kernel.org/linux-block/20180606152515.25807-1-roman.penyaev@xxxxxxxxxxxxxxxx/
> v2:
> o IBNBD:
> - No legacy request IO mode, only MQ is left.
> o IBTRS:
> - No FMR registration, only FR is left.
> * https://lore.kernel.org/linux-block/20180518130413.16997-1-roman.penyaev@xxxxxxxxxxxxxxxx/
> v1:
> o IBTRS: load-balancing and IO fail-over using multipath features were added.
> o Major parts of the code were rewritten, simplified and overall code
> size was reduced by a quarter.
> * https://lore.kernel.org/linux-block/20180202140904.2017-1-roman.penyaev@xxxxxxxxxxxxxxxx/
> v0:
> o Initial submission
> * https://lore.kernel.org/linux-block/1490352343-20075-1-git-send-email-jinpu.wangl@xxxxxxxxxxxxxxxx/
>
> As always, please review and share your comments, and consider to merge to
> upstream.
>
> Thanks!
>
> Jack Wang (25):
> sysfs: export sysfs_remove_file_self()
> RDMA/rtrs: public interface header to establish RDMA connections
> RDMA/rtrs: private headers with rtrs protocol structs and helpers
> RDMA/rtrs: core: lib functions shared between client and server
> modules
> RDMA/rtrs: client: private header with client structs and functions
> RDMA/rtrs: client: main functionality
> RDMA/rtrs: client: statistics functions
> RDMA/rtrs: client: sysfs interface functions
> RDMA/rtrs: server: private header with server structs and functions
> RDMA/rtrs: server: main functionality
> RDMA/rtrs: server: statistics functions
> RDMA/rtrs: server: sysfs interface functions
> RDMA/rtrs: include client and server modules into kernel compilation
> RDMA/rtrs: a bit of documentation
> block/rnbd: private headers with rnbd protocol structs and helpers
> block/rnbd: client: private header with client structs and functions
> block/rnbd: client: main functionality
> block/rnbd: client: sysfs interface functions
> block/rnbd: server: private header with server structs and functions
> block/rnbd: server: main functionality
> block/rnbd: server: functionality for IO submission to file or block
> dev
> block/rnbd: server: sysfs interface functions
> block/rnbd: include client and server modules into kernel compilation
> block/rnbd: a bit of documentation
> MAINTAINERS: Add maintainers for RNBD/RTRS modules
>
> Documentation/ABI/testing/sysfs-block-rnbd | 46 +
> .../ABI/testing/sysfs-class-rnbd-client | 111 +
> .../ABI/testing/sysfs-class-rnbd-server | 50 +
> .../ABI/testing/sysfs-class-rtrs-client | 131 +
> .../ABI/testing/sysfs-class-rtrs-server | 53 +
> MAINTAINERS | 14 +
> drivers/block/Kconfig | 2 +
> drivers/block/Makefile | 1 +
> drivers/block/rnbd/Kconfig | 28 +
> drivers/block/rnbd/Makefile | 15 +
> drivers/block/rnbd/README | 92 +
> drivers/block/rnbd/rnbd-clt-sysfs.c | 619 ++++
> drivers/block/rnbd/rnbd-clt.c | 1721 ++++++++++
> drivers/block/rnbd/rnbd-clt.h | 148 +
> drivers/block/rnbd/rnbd-common.c | 23 +
> drivers/block/rnbd/rnbd-log.h | 41 +
> drivers/block/rnbd/rnbd-proto.h | 305 ++
> drivers/block/rnbd/rnbd-srv-dev.c | 142 +
> drivers/block/rnbd/rnbd-srv-dev.h | 110 +
> drivers/block/rnbd/rnbd-srv-sysfs.c | 211 ++
> drivers/block/rnbd/rnbd-srv.c | 862 +++++
> drivers/block/rnbd/rnbd-srv.h | 79 +
> drivers/infiniband/Kconfig | 1 +
> drivers/infiniband/ulp/Makefile | 1 +
> drivers/infiniband/ulp/rtrs/Kconfig | 27 +
> drivers/infiniband/ulp/rtrs/Makefile | 15 +
> drivers/infiniband/ulp/rtrs/README | 213 ++
> drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 209 ++
> drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 480 +++
> drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2929 +++++++++++++++++
> drivers/infiniband/ulp/rtrs/rtrs-clt.h | 247 ++
> drivers/infiniband/ulp/rtrs/rtrs-log.h | 28 +
> drivers/infiniband/ulp/rtrs/rtrs-pri.h | 401 +++
> drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c | 47 +
> drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 285 ++
> drivers/infiniband/ulp/rtrs/rtrs-srv.c | 2164 ++++++++++++
> drivers/infiniband/ulp/rtrs/rtrs-srv.h | 138 +
> drivers/infiniband/ulp/rtrs/rtrs.c | 594 ++++
> drivers/infiniband/ulp/rtrs/rtrs.h | 310 ++
> fs/sysfs/file.c | 1 +
> 40 files changed, 12894 insertions(+)
> create mode 100644 Documentation/ABI/testing/sysfs-block-rnbd
> create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-client
> create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-server
> create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-client
> create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-server
> create mode 100644 drivers/block/rnbd/Kconfig
> create mode 100644 drivers/block/rnbd/Makefile
> create mode 100644 drivers/block/rnbd/README
> create mode 100644 drivers/block/rnbd/rnbd-clt-sysfs.c
> create mode 100644 drivers/block/rnbd/rnbd-clt.c
> create mode 100644 drivers/block/rnbd/rnbd-clt.h
> create mode 100644 drivers/block/rnbd/rnbd-common.c
> create mode 100644 drivers/block/rnbd/rnbd-log.h
> create mode 100644 drivers/block/rnbd/rnbd-proto.h
> create mode 100644 drivers/block/rnbd/rnbd-srv-dev.c
> create mode 100644 drivers/block/rnbd/rnbd-srv-dev.h
> create mode 100644 drivers/block/rnbd/rnbd-srv-sysfs.c
> create mode 100644 drivers/block/rnbd/rnbd-srv.c
> create mode 100644 drivers/block/rnbd/rnbd-srv.h
> create mode 100644 drivers/infiniband/ulp/rtrs/Kconfig
> create mode 100644 drivers/infiniband/ulp/rtrs/Makefile
> create mode 100644 drivers/infiniband/ulp/rtrs/README
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.h
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-log.h
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-pri.h
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.h
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.h
>
> --
> 2.17.1
>
