Hi Bart, Sagi and all, By current email I would like to share some fresh RDMA performance results of IBNBD, SCST and NVMEof, based on 4.10 kernel and variety of configurations. All fio runs are grouped by the name of a project, crucial config differencies (e.g. CPU pinning or register_always=N) and two testing modes: MANY-DISKS and MANY-JOBS. In each group of results amount of simultaneous fio jobs is increasing starting from 1 up to 128. E.g. in MANY-DISKS testing mode 1 fio job is dedicated to 1 disk, where amount of jobs (and disks) is growing, in its turn, in MANY-JOBS testing mode each fio job produces IO for the same disk, i.e.: MANY-DISKS: x1: numjobs=1 [job1] filename=/dev/nvme0n1 ... x128: numjobs=1 [job1] filename=/dev/nvme0n1 [job2] filename=/dev/nvme0n2 ... [job128] filename=/dev/nvme0n128 MANY-JOBS: x1: numjobs=1 [job1] filename=/dev/nvme0n1 ... x128: numjobs=128 [job1] filename=/dev/nvme0n1 Each group of results represents itself as a performance measurement, which can be easily plotted, taking number of jobs as X axis and iops, overall IO latencies or anything else extracted from fio json result files as Y axis. FIO configurations were generated and saved along with produced fio json results by the fio-runner.py script [1]. Complete archive with FIO configs and results can be downloaded here [2]. The following metrics were taken from fio json results: write/iops - IOPS write/lat/mean - average latency (μs) Here I would like to present reduced results table taking into account only runs with CPU pinning in MANY-DISKS testing mode, since CPU pinning makes more sense in terms of performance and MANY-DISKS and MANY-JOBS results look very much similar: write/iops (MANY-DISKS) IBNBD_pin NVME_noreg_pin NVME_pin SCST_noreg_pin SCST_pin x1 80398.96 75577.24 54110.19 59555.04 48446.05 x2 109018.60 96478.45 69176.77 73925.81 55557.59 x4 169164.56 140558.75 93700.96 75419.91 56294.61 x8 197725.44 159887.33 99773.05 79750.92 55938.84 x16 176782.36 150448.33 99644.05 92964.23 56463.14 x32 139666.00 123198.38 81845.30 81287.98 50590.86 x64 125666.16 82231.77 72117.67 72023.32 45121.17 x128 120253.63 73911.97 65665.08 74642.27 47268.46 write/lat/mean (MANY-DISKS) IBNBD_pin NVME_noreg_pin NVME_pin SCST_noreg_pin SCST_pin x1 647.78 697.91 1032.97 925.51 1173.04 x2 973.20 1104.38 1612.75 1462.18 2047.11 x4 1279.49 1528.09 2452.22 3188.41 4235.95 x8 2356.92 2929.87 4891.70 6248.85 8907.10 x16 5605.62 6575.70 10046.4 10830.50 17945.57 x32 14489.54 16516.60 24849.16 24984.26 40335.09 x64 32364.39 49481.42 56615.23 56559.02 90590.84 x128 67570.88 110768.70 124249.4 109321.84 171390.00 * Where suffixes mean: _pin - CPU pinning _noreg - modules on initiator side (ib_srp, nvme_rdma) were loaded with 'register_always=N' param Complete table results and corresponding graphs are presented on Google sheet [3]. Conclusion: IBNBD outperforms in average by: NVME_noreg_pin NVME_pin SCST_noreg_pin SCST_pin iops 41% 72% 61% 155% lat/mean 28% 42% 38% 60% * Complete tables results [3] were taken into account for average percentage calculation. Test setup is the following: Initiator and target HW configuration: AMD Opteron 6386 SE, 64CPU, 128Gb InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] Initiator and target SW configuration: vanilla Linux 4.10 + IBNBD patches + SCST from https://github.com/bvanassche/scst, master branch Initiator side: IBNBD and NVME: MQ mode SRP: default RQ, on attempt to set 'use_blk_mq=Y' IO hangs. FIO generic configuration pattern: bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4 fadvise_hint=0 rw=randrw:2 direct=1 random_distribution=zipf:1.2 time_based=1 runtime=10 ioengine=libaio iodepth=128 iodepth_batch_submit=128 iodepth_batch_complete=128 group_reporting Target side: 128 null_blk devices with default configuration, opened as blockio. NVMEoF configuration script [4]. SCST configuration script [5]. Would be great to receive any feedback. I am open for further perf tuning and testing with other possible configurations and options. Thanks. -- Roman [1] FIO runner and results extractor script: https://drive.google.com/open?id=0B8_SivzwHdgSS2RKcmc4bWg0YjA [2] Archive with FIO configurations and results: https://drive.google.com/open?id=0B8_SivzwHdgSaDlhMXV6THhoRXc [3] Google sheet with performance measurements: https://drive.google.com/open?id=1sCTBKLA5gbhhkgd2USZXY43VL3zLidzdqDeObZn9Edc [4] NVMEoF configuration: https://drive.google.com/open?id=0B8_SivzwHdgSTzRjbGtmaVR6LWM [5] SCST configuration: https://drive.google.com/open?id=0B8_SivzwHdgSM1B5eGpKWmFJMFk