Hi all,
This is v2 of series, which introduces IBNBD/IBTRS modules.
This cover letter is split on three parts:
1. Introduction, which almost repeats everything from previous cover
letters.
2. Changelog.
3. Performance measurements on linux-4.17.0-rc2 and on two different
Mellanox cards: ConnectX-2 and ConnectX-3 and CPUs: Intel and AMD.
Introduction
-------------
IBTRS (InfiniBand Transport) is a reliable high speed transport library
which allows for establishing connection between client and server
machines via RDMA. It is optimized to transfer (read/write) IO blocks
in the sense that it follows the BIO semantics of providing the
possibility to either write data from a scatter-gather list to the
remote side or to request ("read") data transfer from the remote side
into a given set of buffers.
IBTRS is multipath capalbdke and provides I/O fail-over and load-balancing
functionality, i.e. in IBTRS terminology, an IBTRS path is a set of RDMA
CMs and particular path is selected according to the load-balancing policy.
IBNBD (InfiniBand Network Block Device) is a pair of kernel modules
(client and server) that allow for remote access of a block device on
the server over IBTRS protocol. After being mapped, the remote block
devices can be accessed on the client side as local block devices.
Internally IBNBD uses IBTRS as an RDMA transport library.
Why?
- IBNBD/IBTRS is developed in order to map thin provisioned volumes,
thus internal protocol is simple.
- IBTRS was developed as an independent RDMA transport library, which
supports fail-over and load-balancing policies using multipath, thus
it can be used for any other IO needs rather than only for block
device.
- IBNBD/IBTRS is faster than NVME over RDMA.
Old comparison results:
https://www.spinics.net/lists/linux-rdma/msg48799.html
New comparison results: see performance measurements section below.
Key features of IBTRS transport library and IBNBD block device:
o High throughput and low latency due to:
- Only two RDMA messages per IO.
- IMM InfiniBand messages on responses to reduce round trip latency.
- Simplified memory management: memory allocation happens once on
server side when IBTRS session is established.
o IO fail-over and load-balancing by using multipath. According to
our test loads additional path brings ~20% of bandwidth.
o Simple configuration of IBNBD:
- Server side is completely passive: volumes do not need to be
explicitly exported.
- Only IB port GID and device path needed on client side to map
a block device.
- A device is remapped automatically i.e. after storage reboot.
Commits for kernel can be found here:
https://github.com/profitbricks/ibnbd/commits/linux-4.17-rc2
The out-of-tree modules are here:
https://github.com/profitbricks/ibnbd/
Vault 2017 presentation:
http://events.linuxfoundation.org/sites/events/files/slides/IBNBD-Vault-2017.pdf
Changelog
---------
v2:
o IBNBD:
- No legacy request IO mode, only MQ is left.
o IBTRS:
- No FMR registration, only FR is left.
- By default memory is always registered for the sake of the security,
i.e. by default no pd is created with IB_PD_UNSAFE_GLOBAL_RKEY.
- Server side (target) always does memory registration and exchanges
MRs dma addresses with client for direct writes from client side.
- Client side (initiator) has `noreg_cnt` module option, which specifies
sg number, from which read IO should be registered. By default 0
is set, i.e. always register memory for read IOs. (IBTRS protocol
does not require registration for writes, which always go directly
to server memory).
- Proper DMA sync with ib_dma_sync_single_for_(cpu|device) calls.
- Do signalled IB_WR_LOCAL_INV.
- Avoid open-coding of string conversion to IPv4/6 sockaddr,
inet_pton_with_scope() is used instead.
- Introduced block device namespaces configuration on server side
(target) to avoid security gap in not trusted environment, when
client can map a block device which does not belong to him.
When device namespaces are enabled on server side, server opens
device using client's session name in the device path, where
session name is a random token, e.g. GUID. If server is configured
to find device namespaces in a folder /run/ibnbd-guid/, then
request to map device 'sda1' from client with session 'A' (or any
token) will be resolved by path /run/ibnbd-guid/A/sda1.
- README is extended with description of IBTRS and IBNBD protocol,
e.g. how IB IMM field is used to acknowledge IO requests or
heartbeats.
- IBTRS/IBNBD client and server modules are registered as devices in
the kernel in order to have all sysfs configuration entries under
/sys/devices/virtual/ in order not to spoil /sys/kernel directory.
I failed to switch configuration to configfs, because of the
several reasons:
a) configfs entries created from kernel side using
configfs_register_group() API call can't be removed from
userspace side using rmdir() syscall. That is required
behaviour for IBTRS when session is created by API call and
not from userspace.
Actually, I have a patch for configfs to solve a), but then
b) comes.
b) configfs show/store callbacks are racy by design (in
contradiction to kernfs), i.e. even dentry is unhashed, opener
of it can be faster and in few moments later those callbacks
can be invoked. To guarantee that all openers left and nobody
is able to access an entry after configfs_drop_dentry() is
returned additional hairy code should be written with wait
queues, locks, etc. I didn't like at all what I eventually
got, gave up and left as is, i.e. sysfs.
What is left unchanged on IBTRS side but was suggested to modify:
- Bart suggested to use sbitmap instead of calling find_first_zero_bit()
and friends. I found calling pure bit API is more explicit in
comparison to sbitmap - there is no need in using sbitmap_queue
and all the power of wait queues, no benefits in terms of LoC
as well.
- I did several attempts to unify approach of wrapping ib_device
with ULP device structure (e.g. device pool or using ib_client
API) but it turns out to be that none of these approaches bring
simplicity, so IBTRS still creates ULP specific device on demand
and keeps it in the list.
- Sagi suggested to extend inet_pton_with_scope() with gid to
sockaddr conversion, but after IPv6 conversion (gid is compliant
with IPv6) special RDMA magic should be done in order to setup
IB port space range, which is very specific and does not fit to
be some generic library helper. And am I right that gid is not
used and seems dying?
v1:
- IBTRS: load-balancing and IO fail-over using multipath features were added.
- Major parts of the code were rewritten, simplified and overall code
size was reduced by a quarter.
* https://lwn.net/Articles/746342/
v0:
- Initial submission
* https://lwn.net/Articles/718181/
Performance measurements
------------------------
o FR and FMR:
Firstly I would like to start performance measurements with (probably
well known) observations that FR is slower than FMR by ~40% on Mellanox
ConnectX-2 and by ~15% on ConnectX-3. That is a huge numbers, e.g. FIO
results on IBNBD:
- on ConnectX-2 (MT26428)
x64 CPUs AMD Opteron(tm) Processor 6282 SE
rw=randread, bandwidth in Kbytes:
jobs IBNBD (FMR) IBNBD (FR) Change
x1 1037624 932951 -10.1%
x8 2569649 1543074 -40.0%
x16 2751461 1531282 -44.3%
x24 2360887 1396153 -40.9%
x32 1873174 1215334 -35.1%
x40 1995846 1255781 -37.1%
x48 2004740 1240931 -38.1%
x56 2076871 1250333 -39.8%
x64 2051668 1229389 -40.1%
- on ConnectX-3 (MT4099)
x40 CPUs Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
rw=randread, bandwidth in Kbytes:
jobs IBNBD (FMR) IBNBD (FR) Change
x1 2243322 1961216 -12.6%
x8 4389048 4012912 -8.6%
x16 4473103 4033837 -9.8%
x24 4570209 3939186 -13.8%
x32 4576757 3843434 -16.0%
x40 4468110 3696896 -17.3%
x48 4848049 4106259 -15.3%
x56 4872790 4141374 -15.0%
x64 4967287 4207317 -15.3%
I missed the whole history why FMR is considered as outdated, why FMR
is a no way, I would very much appreciate if someone would explain me
why FR should be prefered. Is there a link with a clear explanation?
o IBNBD and NVMEoRDMA
Here I would like to publish IBNBD and NVMEoRDMA comparison results
with FR memory registration on each IO (i.e. with the following modules
params: 'register_always=Y' for NVME and 'noreg_cnt=0' for IBTRS).
- on ConnectX-2 (MT26428)
x64 CPUs AMD Opteron(tm) Processor 6282 SE
rw=randread, bandwidth in Kbytes:
jobs IBNBD NVMEoRDMA Change
x1 932951 975425 +4.6%
x8 1543074 1504416 -2.5%
x16 1531282 1432937 -6.4%
x24 1396153 1244858 -10.8%
x32 1215334 1066607 -12.2%
x40 1255781 1076841 -14.2%
x48 1240931 1066453 -14.1%
x56 1250333 1065879 -14.8%
x64 1229389 1064199 -13.4%
rw=randwrite, bandwidth in Kbytes:
jobs IBNBD NVMEoRDMA Change
x1 1416413 1181102 -16.6%
x8 2438615 1977051 -18.9%
x16 2436924 1854223 -23.9%
x24 2430527 1714580 -29.5%
x32 2425552 1641288 -32.3%
x40 2378784 1592788 -33.0%
x48 2202260 1511895 -31.3%
x56 2207013 1493400 -32.3%
x64 2098949 1432951 -31.7%
- on ConnectX-3 (MT4099)
x40 CPUs Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
rw=randread, bandwidth in Kbytes:
jobs IBNBD NVMEoRDMA Change
x1 1961216 2046572 +4.4%
x8 4012912 4059410 +1.2%
x16 4033837 3968410 -1.6%
x24 3939186 3770729 -4.3%
x32 3843434 3623869 -5.7%
x40 3696896 3448772 -6.7%
x48 4106259 3729201 -9.2%
x56 4141374 3732954 -9.9%
x64 4207317 3805638 -9.5%
rw=randwrite, bandwidth in Kbytes:
jobs IBNBD NVMEoRDMA Change
x1 3195637 2479068 -22.4%
x8 4576924 4541743 -0.8%
x16 4581528 4555459 -0.6%
x24 4692540 4595963 -2.1%
x32 4686968 4540456 -3.1%
x40 4583814 4404859 -3.9%
x48 4969587 4710902 -5.2%
x56 4996101 4701814 -5.9%
x64 5083460 4759663 -6.4%
The interesting observation is that on machine with Intel CPUs and
ConnectX-3 card the difference between IBNBD and NVME bandwidth is
significantly smaller comparing to AMD and ConnectX-2. I did not
thoroughly investiage that behaviour, but suspect that the devil
is in Intel vs AMD architecture and probably how NUMAs are organized,
i.e. Intel has 2 NUMA nodes against 8 on AMD. If someone is interested
in those results and can point me out where to dig on NVME side I can
investigate deeply why exactly NVME bandwidth significantly drops on
AMD machine with Connect-X2.
Shiny graphs are here:
https://docs.google.com/spreadsheets/d/1vxSoIvfjPbOWD61XMeN2_gPGxsxrbIUOZADk1UX5lj0
Roman Pen (26):
rculist: introduce list_next_or_null_rr_rcu()
sysfs: export sysfs_remove_file_self()
ibtrs: public interface header to establish RDMA connections
ibtrs: private headers with IBTRS protocol structs and helpers
ibtrs: core: lib functions shared between client and server modules
ibtrs: client: private header with client structs and functions
ibtrs: client: main functionality
ibtrs: client: statistics functions
ibtrs: client: sysfs interface functions
ibtrs: server: private header with server structs and functions
ibtrs: server: main functionality
ibtrs: server: statistics functions
ibtrs: server: sysfs interface functions
ibtrs: include client and server modules into kernel compilation
ibtrs: a bit of documentation
ibnbd: private headers with IBNBD protocol structs and helpers
ibnbd: client: private header with client structs and functions
ibnbd: client: main functionality
ibnbd: client: sysfs interface functions
ibnbd: server: private header with server structs and functions
ibnbd: server: main functionality
ibnbd: server: functionality for IO submission to file or block dev
ibnbd: server: sysfs interface functions
ibnbd: include client and server modules into kernel compilation
ibnbd: a bit of documentation
MAINTAINERS: Add maintainer for IBNBD/IBTRS modules
MAINTAINERS | 14 +
drivers/block/Kconfig | 2 +
drivers/block/Makefile | 1 +
drivers/block/ibnbd/Kconfig | 22 +
drivers/block/ibnbd/Makefile | 13 +
drivers/block/ibnbd/README | 299 +++
drivers/block/ibnbd/ibnbd-clt-sysfs.c | 669 ++++++
drivers/block/ibnbd/ibnbd-clt.c | 1818 +++++++++++++++
drivers/block/ibnbd/ibnbd-clt.h | 171 ++
drivers/block/ibnbd/ibnbd-log.h | 71 +
drivers/block/ibnbd/ibnbd-proto.h | 364 +++
drivers/block/ibnbd/ibnbd-srv-dev.c | 410 ++++
drivers/block/ibnbd/ibnbd-srv-dev.h | 149 ++
drivers/block/ibnbd/ibnbd-srv-sysfs.c | 242 ++
drivers/block/ibnbd/ibnbd-srv.c | 922 ++++++++
drivers/block/ibnbd/ibnbd-srv.h | 100 +
drivers/infiniband/Kconfig | 1 +
drivers/infiniband/ulp/Makefile | 1 +
drivers/infiniband/ulp/ibtrs/Kconfig | 20 +
drivers/infiniband/ulp/ibtrs/Makefile | 15 +
drivers/infiniband/ulp/ibtrs/README | 358 +++
drivers/infiniband/ulp/ibtrs/ibtrs-clt-stats.c | 455 ++++
drivers/infiniband/ulp/ibtrs/ibtrs-clt-sysfs.c | 482 ++++
drivers/infiniband/ulp/ibtrs/ibtrs-clt.c | 2814 ++++++++++++++++++++++++
drivers/infiniband/ulp/ibtrs/ibtrs-clt.h | 304 +++
drivers/infiniband/ulp/ibtrs/ibtrs-log.h | 91 +
drivers/infiniband/ulp/ibtrs/ibtrs-pri.h | 458 ++++
drivers/infiniband/ulp/ibtrs/ibtrs-srv-stats.c | 110 +
drivers/infiniband/ulp/ibtrs/ibtrs-srv-sysfs.c | 271 +++
drivers/infiniband/ulp/ibtrs/ibtrs-srv.c | 1980 +++++++++++++++++
drivers/infiniband/ulp/ibtrs/ibtrs-srv.h | 174 ++
drivers/infiniband/ulp/ibtrs/ibtrs.c | 609 +++++
drivers/infiniband/ulp/ibtrs/ibtrs.h | 331 +++
fs/sysfs/file.c | 1 +
include/linux/rculist.h | 19 +
35 files changed, 13761 insertions(+)
create mode 100644 drivers/block/ibnbd/Kconfig
create mode 100644 drivers/block/ibnbd/Makefile
create mode 100644 drivers/block/ibnbd/README
create mode 100644 drivers/block/ibnbd/ibnbd-clt-sysfs.c
create mode 100644 drivers/block/ibnbd/ibnbd-clt.c
create mode 100644 drivers/block/ibnbd/ibnbd-clt.h
create mode 100644 drivers/block/ibnbd/ibnbd-log.h
create mode 100644 drivers/block/ibnbd/ibnbd-proto.h
create mode 100644 drivers/block/ibnbd/ibnbd-srv-dev.c
create mode 100644 drivers/block/ibnbd/ibnbd-srv-dev.h
create mode 100644 drivers/block/ibnbd/ibnbd-srv-sysfs.c
create mode 100644 drivers/block/ibnbd/ibnbd-srv.c
create mode 100644 drivers/block/ibnbd/ibnbd-srv.h
create mode 100644 drivers/infiniband/ulp/ibtrs/Kconfig
create mode 100644 drivers/infiniband/ulp/ibtrs/Makefile
create mode 100644 drivers/infiniband/ulp/ibtrs/README
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-clt-stats.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-clt-sysfs.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-clt.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-clt.h
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-log.h
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-pri.h
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-srv-stats.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-srv-sysfs.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-srv.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs-srv.h
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs.c
create mode 100644 drivers/infiniband/ulp/ibtrs/ibtrs.h
Signed-off-by: Roman Pen <roman.penyaev@xxxxxxxxxxxxxxxx>
Cc: Danil Kipnis <danil.kipnis@xxxxxxxxxxxxxxxx>
Cc: Jack Wang <jinpu.wang@xxxxxxxxxxxxxxxx>
--
2.13.1
