Re: [PATCH v12 00/25] RTRS (former IBTRS) RDMA Transport Library and RNBD (former IBNBD) RDMA Network Block Device

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, Apr 15, 2020 at 11:21 AM Danil Kipnis
<danil.kipnis@xxxxxxxxxxxxxxx> wrote:
>
> Hi all,
>
> Here is v12 of  the RTRS (former IBTRS) RDMA Transport Library and the
> corresponding RNBD (former IBNBD) RDMA Network Block Device, which includes
> changes to address comments from the community.
>
> Introduction
> -------------
>
> RTRS (RDMA Transport) is a reliable high speed transport library
> which allows for establishing connection between client and server
> machines via RDMA. It is based on RDMA-CM, so expect also to support RoCE
> and iWARP, but we mainly tested in IB environment. It is optimized to
> transfer (read/write) IO blocks in the sense that it follows the BIO
> semantics of providing the possibility to either write data from a
> scatter-gather list to the remote side or to request ("read") data
> transfer from the remote side into a given set of buffers.
>
> RTRS is multipath capable and provides I/O fail-over and load-balancing
> functionality, i.e. in RTRS terminology, an RTRS path is a set of RDMA
> connections and particular path is selected according to the load-balancing
> policy. It can be used for other components beside RNBD.
>
> Module parameter always_invalidate is introduced for the security problem
> discussed in LPC RDMA MC 2019. When always_invalidate=Y, on the server side we
> invalidate each rdma buffer before we hand it over to RNBD server and
> then pass it to the block layer. A new rkey is generated and registered for the
> buffer after it returns back from the block layer and RNBD server.
> The new rkey is sent back to the client along with the IO result.
> The procedure is the default behaviour of the driver. This invalidation and
> registration on each IO causes performance drop of up to 20%. A user of the
> driver may choose to load the modules with this mechanism switched off
> (always_invalidate=N), if he understands and can take the risk of a malicious
> client being able to corrupt memory of a server it is connected to. This might
> be a reasonable option in a scenario where all the clients and all the servers
> are located within a secure datacenter.
>
> RNBD (RDMA Network Block Device) is a pair of kernel modules
> (client and server) that allow for remote access of a block device on
> the server over RTRS protocol. After being mapped, the remote block
> devices can be accessed on the client side as local block devices.
> Internally RNBD uses RTRS as an RDMA transport library.
>
> Commits for kernel can be found here:
>    https://github.com/ionos-enterprise/ibnbd/commits/linux-5.7-rc1-ibnbd-v12
>
> Testing
> -------
>
> All the changes have been tested with our regression testsuite in our staging environment
> in IONOS data center. it's around 200 testcases, for both always_invalidate=N and
> always_invalidate=Y configurations.
>
> Changelog
> ---------
> v12:
>  o Rebased to linux-5.7-rc1
>  o rtrs/rnbd: add release call back for kobject suggested by Bart & Jason
>  o rtrs-clt: drop reexport bio_map_kern, open-code it using bio_add_page
>  suggested by Christoph
>  o rnbd-srv: replace rwlock with RCU.
>  o rnbd-srv: get rid of sysfs_release_compl suggested by Bart.
>  o rnbd-clt: add a option to specify the destination port in map_device operation
>  suggested by Bart.
>  o rnbd-srv: remove io_cb use rnbd_endio directly
>  o Address other comments from Bart for naming/typo/comments/coding style/etc
>
> v11:
>  o Rebased to linux-5.6-rc6
>  o rtrs-clt: use rcu_dereference_protected to avoid warning suggested by Jason
>  o rtrs-clt: get rid of the second cancel_delayed_work_sync(reconnect_work) call
>  suggested by Jason
>  o rtrs-clt: remove unneeded synchronize_rcu suggested by Jason
>  o rtrs-clt: removing the opened flag suggested by Jason
>  o rtrs-clt: postpone uevent until sysfs is created suggested by Jason
>  o rtrs-srv: postpone uevent until sysfs is created.
>  o rtrs: remove unnecessary module_get/put call suggested by Jason
>  o rtrs-clt: fix up error path in alloc_clt() reported by Jason
>  o rtrs: move kdocs to .c files suggested by Jason and Bart
>  o rtrs/rnbd: remove print during load/unload module reported by Jason
>  o rtrs/rnbd: add missing mutex_destroy
>  o rtrs: cleanup dead code
>  o rtrs-srv: fix error handling
>  o other misc cleanup
>  * https://lore.kernel.org/linux-block/20200320121657.1165-1-jinpu.wang@xxxxxxxxxxxxxxx/
> v10:
>  o Rebased to linux-5.6-rc5
>  o Collect Reviewed-by from Bart
>  o Update description in Kconfig for RNBD
>  o Address comments from Bart for naming/typo/comments/etc
>  o removal of rnbd_bio_map_kern by reexporting bio_map_kern suggested by Bart
>  o kill some inline wrappers suggested by Bart
>  o rtrs: use mutex more consistently reported by Leon
>  o rtrs/rnbd: remove prints for allocation failure suggested by Leon
>  o rtrs/rnbd: avoid typedefs for function callbacks suggested by Leon
>  o rtrs-srv: handle sq_full situation suggested by Jason
>  o rtrs-clt: remove useless get_cpu()/put_cpu() in __rtrs_get_permit suggested
>  by Jason and Bart.
>  o rtrs-srv: inline rtrs_srv_update_rdma_stats
>  o other minor cleanup
>  * https://lore.kernel.org/linux-block/20200311161240.30190-1-jinpu.wang@xxxxxxxxxxxxxxx/
> v9:
>  o Rebased to linux-5.6-rc2
>  o Update Date/Kernel version in Documentation
>  o Update description in Kconfig for RNBD
>  o rtrs-clt: inline rtrs_clt_decrease_inflight
>  o rtrs-clt: only track inflight for Min_inflight policy
>  * https://lore.kernel.org/linux-block/20200221104721.350-1-jinpuwang@xxxxxxxxx/
> v8:
>  o Rebased to linux-5.5-rc7
>  o Reviewed likey/unlikely usage, only keep the one in IO path suggested by Leon Romanovsky
>  o Reviewed inline usage, remove inline for functions longer than 5 lines of code suggested by Leon
>  o Removed 2 WARN_ON suggested by Leon
>  o Removed 2 empty lines between copyright suggested by Leon
>  o Makefile: remove compat include for upstream suggested by Leon
>  o rtrs-clt: remove module parameters suggested by Leon
>  o drop rnbd_clt_dev_is_mapped
>  o rnbd-clt: clean up rnbd_rerun_if_needed
>  o rtrs-srv: remove reset_all sysfs
>  o rtrs stats: remove wc_completion stats
>  o rtrs-clt: enhance doc for rtrs_clt_change_state
>  o rtrs-clt: remove unused rtrs_permit_from_pdu
>  * https://lore.kernel.org/linux-block/20200124204753.13154-1-jinpuwang@xxxxxxxxx/
> v7:
>  o Rebased to linux-5.5-rc6
>  o Implement code-style/readability/API/Documentation etc suggestions by Bart van Assche
>  o Make W=1 clean
>  o New benchmark results for Mellanox ConnectX-5
>  o second try adding MAINTAINERS entries in alphabetical order as Gal Pressman suggested
>  * https://lore.kernel.org/linux-block/20200116125915.14815-1-jinpuwang@xxxxxxxxx/
> v6:
>   o Rebased to linux-5.5-rc4
>   o Fix typo in my email address in first patch
>   o Cleanup copyright as suggested by Leon Romanovsky
>   o Remove 2 redudant kobject_del in error path as suggested by Leon Romanovsky
>   o Add MAINTAINERS entries in alphabetical order as Gal Pressman suggested
>   * https://lore.kernel.org/linux-block/20191230102942.18395-1-jinpuwang@xxxxxxxxx/
> v5:
>   o Fix the security problem pointed out by Jason
>   o Implement code-style/readability/API/etc suggestions by Bart van Assche
>   o Rename IBTRS and IBNBD to RTRS and RNBD accordingly
>   o Fileio mode support in rnbd-srv has been removed.
>   * https://lore.kernel.org/linux-block/20191220155109.8959-1-jinpuwang@xxxxxxxxx/
> v4:
>   o Protocol extended to transport IO priorities
>   o Support for Mellanox ConnectX-4/X-5
>   o Minor sysfs extentions (display access mode on server side)
>   o Bug fixes: cleaning up sysfs folders, race on deallocation of resources
>   o Style fixes
>   * https://lore.kernel.org/linux-block/20190620150337.7847-1-jinpuwang@xxxxxxxxx/
> v3:
>   o Sparse fixes:
>      - le32 -> le16 conversion
>      - pcpu and RCU wrong declaration
>      - sysfs: dynamically alloc array of sockaddr structures to reduce
>            size of a stack frame
>   o Rename sysfs folder on client and server sides to show source and
>     destination addresses of the connection, i.e.:
>            .../<session-name>/paths/<src@dst>/
>   o Remove external inclusions from Makefiles.
>   * https://lore.kernel.org/linux-block/20180606152515.25807-1-roman.penyaev@xxxxxxxxxxxxxxxx/
> v2:
>   o IBNBD:
>      - No legacy request IO mode, only MQ is left.
>   o IBTRS:
>      - No FMR registration, only FR is left.
>   * https://lore.kernel.org/linux-block/20180518130413.16997-1-roman.penyaev@xxxxxxxxxxxxxxxx/
> v1:
>   o IBTRS: load-balancing and IO fail-over using multipath features were added.
>   o Major parts of the code were rewritten, simplified and overall code
>     size was reduced by a quarter.
>   * https://lore.kernel.org/linux-block/20180202140904.2017-1-roman.penyaev@xxxxxxxxxxxxxxxx/
> v0:
>   o Initial submission
>   * https://lore.kernel.org/linux-block/1490352343-20075-1-git-send-email-jinpu.wangl@xxxxxxxxxxxxxxxx/
>
> As always, please review, share your comments, and consider to merge to
> upstream.
>
> Thanks.
>
> Jack Wang (25):
>   sysfs: export sysfs_remove_file_self()
>   RDMA/rtrs: public interface header to establish RDMA connections
>   RDMA/rtrs: private headers with rtrs protocol structs and helpers
>   RDMA/rtrs: core: lib functions shared between client and server
>     modules
>   RDMA/rtrs: client: private header with client structs and functions
>   RDMA/rtrs: client: main functionality
>   RDMA/rtrs: client: statistics functions
>   RDMA/rtrs: client: sysfs interface functions
>   RDMA/rtrs: server: private header with server structs and functions
>   RDMA/rtrs: server: main functionality
>   RDMA/rtrs: server: statistics functions
>   RDMA/rtrs: server: sysfs interface functions
>   RDMA/rtrs: include client and server modules into kernel compilation
>   RDMA/rtrs: a bit of documentation
>   block/rnbd: private headers with rnbd protocol structs and helpers
>   block/rnbd: client: private header with client structs and functions
>   block/rnbd: client: main functionality
>   block/rnbd: client: sysfs interface functions
>   block/rnbd: server: private header with server structs and functions
>   block/rnbd: server: main functionality
>   block/rnbd: server: functionality for IO submission to block dev
>   block/rnbd: server: sysfs interface functions
>   block/rnbd: include client and server modules into kernel compilation
>   block/rnbd: a bit of documentation
>   MAINTAINERS: Add maintainers for RNBD/RTRS modules
>
>  Documentation/ABI/testing/sysfs-block-rnbd    |   46 +
>  .../ABI/testing/sysfs-class-rnbd-client       |  111 +
>  .../ABI/testing/sysfs-class-rnbd-server       |   50 +
>  .../ABI/testing/sysfs-class-rtrs-client       |  131 +
>  .../ABI/testing/sysfs-class-rtrs-server       |   53 +
>  MAINTAINERS                                   |   14 +
>  drivers/block/Kconfig                         |    2 +
>  drivers/block/Makefile                        |    1 +
>  drivers/block/rnbd/Kconfig                    |   28 +
>  drivers/block/rnbd/Makefile                   |   15 +
>  drivers/block/rnbd/README                     |   92 +
>  drivers/block/rnbd/rnbd-clt-sysfs.c           |  636 ++++
>  drivers/block/rnbd/rnbd-clt.c                 | 1729 ++++++++++
>  drivers/block/rnbd/rnbd-clt.h                 |  156 +
>  drivers/block/rnbd/rnbd-common.c              |   23 +
>  drivers/block/rnbd/rnbd-log.h                 |   41 +
>  drivers/block/rnbd/rnbd-proto.h               |  303 ++
>  drivers/block/rnbd/rnbd-srv-dev.c             |  134 +
>  drivers/block/rnbd/rnbd-srv-dev.h             |   92 +
>  drivers/block/rnbd/rnbd-srv-sysfs.c           |  215 ++
>  drivers/block/rnbd/rnbd-srv.c                 |  861 +++++
>  drivers/block/rnbd/rnbd-srv.h                 |   79 +
>  drivers/infiniband/Kconfig                    |    1 +
>  drivers/infiniband/ulp/Makefile               |    1 +
>  drivers/infiniband/ulp/rtrs/Kconfig           |   27 +
>  drivers/infiniband/ulp/rtrs/Makefile          |   15 +
>  drivers/infiniband/ulp/rtrs/README            |  213 ++
>  drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c  |  200 ++
>  drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c  |  481 +++
>  drivers/infiniband/ulp/rtrs/rtrs-clt.c        | 2995 +++++++++++++++++
>  drivers/infiniband/ulp/rtrs/rtrs-clt.h        |  251 ++
>  drivers/infiniband/ulp/rtrs/rtrs-log.h        |   28 +
>  drivers/infiniband/ulp/rtrs/rtrs-pri.h        |  399 +++
>  drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c  |   38 +
>  drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c  |  320 ++
>  drivers/infiniband/ulp/rtrs/rtrs-srv.c        | 2175 ++++++++++++
>  drivers/infiniband/ulp/rtrs/rtrs-srv.h        |  148 +
>  drivers/infiniband/ulp/rtrs/rtrs.c            |  612 ++++
>  drivers/infiniband/ulp/rtrs/rtrs.h            |  195 ++
>  fs/sysfs/file.c                               |    1 +
>  40 files changed, 12912 insertions(+)
>  create mode 100644 Documentation/ABI/testing/sysfs-block-rnbd
>  create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-client
>  create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-server
>  create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-client
>  create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-server
>  create mode 100644 drivers/block/rnbd/Kconfig
>  create mode 100644 drivers/block/rnbd/Makefile
>  create mode 100644 drivers/block/rnbd/README
>  create mode 100644 drivers/block/rnbd/rnbd-clt-sysfs.c
>  create mode 100644 drivers/block/rnbd/rnbd-clt.c
>  create mode 100644 drivers/block/rnbd/rnbd-clt.h
>  create mode 100644 drivers/block/rnbd/rnbd-common.c
>  create mode 100644 drivers/block/rnbd/rnbd-log.h
>  create mode 100644 drivers/block/rnbd/rnbd-proto.h
>  create mode 100644 drivers/block/rnbd/rnbd-srv-dev.c
>  create mode 100644 drivers/block/rnbd/rnbd-srv-dev.h
>  create mode 100644 drivers/block/rnbd/rnbd-srv-sysfs.c
>  create mode 100644 drivers/block/rnbd/rnbd-srv.c
>  create mode 100644 drivers/block/rnbd/rnbd-srv.h
>  create mode 100644 drivers/infiniband/ulp/rtrs/Kconfig
>  create mode 100644 drivers/infiniband/ulp/rtrs/Makefile
>  create mode 100644 drivers/infiniband/ulp/rtrs/README
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.h
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-log.h
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-pri.h
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.h
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.c
>  create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.h
>
> --
> 2.20.1
>
Hi Jason, hi Leon, hi Doug, hi all,

We now have Reviewed-by for RNBD part from Bart (Thanks again), Do you
have new comments regarding RTRS, should we send another round to do a
rebase to latest rc?

Thanks!



[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux