On Wed, 2018-05-02 at 13:16 +0300, Leon Romanovsky wrote:
> From: Idan Burstein <idanb@xxxxxxxxxxxx>
>
> As most kernel RDMA ULPs, (e.g. NVMe over Fabrics in its default
> "register_always=Y" mode) registers and invalidates user buffer
> upon each IO.
>
> Today the mlx5 driver is posting the registration work
> request using scatter/gather entry for the MTT/KLM list.
> The fetch of the MTT/KLM list becomes the bottleneck in
> number of IO operation could be done by NVMe over Fabrics
> host driver on a single adapter as shown below.
>
> This patch is adding the support for inline registration
> work request upon MTT/KLM list of size <=64B.
>
> The result for NVMe over Fabrics is increase of > x3.5 for small
> IOs as shown below, I expect other ULPs (e.g iSER, SRP, NFS over RDMA)
> performance to be enhanced as well.
>
> The following results were taken against a single NVMe-oF (RoCE link layer)
> subsystem with a single namespace backed by null_blk using fio benchmark
> (with rw=randread, numjobs=48, iodepth={16,64}, ioengine=libaio direct=1):
>
> ConnectX-5 (pci Width x16)
> ---------------------------
>
> Block Size s/g reg_wr inline reg_wr
> ++++++++++ +++++++++++++++ ++++++++++++++++
> 512B 1302.8K/34.82% 4951.9K/99.02%
> 1KB 1284.3K/33.86% 4232.7K/98.09%
> 2KB 1238.6K/34.1% 2797.5K/80.04%
> 4KB 1169.3K/32.46% 1941.3K/61.35%
> 8KB 1013.4K/30.08% 1236.6K/39.47%
> 16KB 695.7K/20.19% 696.9K/20.59%
> 32KB 350.3K/9.64% 350.6K/10.3%
> 64KB 175.86K/5.27% 175.9K/5.28%
>
> ConnectX-4 (pci Width x8)
> ---------------------------
>
> Block Size s/g reg_wr inline reg_wr
> ++++++++++ +++++++++++++++ ++++++++++++++++
> 512B 1285.8K/42.66% 4242.7K/98.18%
> 1KB 1254.1K/41.74% 3569.2K/96.00%
> 2KB 1185.9K/39.83% 2173.9K/75.58%
> 4KB 1069.4K/36.46% 1343.3K/47.47%
> 8KB 755.1K/27.77% 748.7K/29.14%
>
> Tested-by: Nitzan Carmi <nitzanc@xxxxxxxxxxxx>
> Signed-off-by: Idan Burstein <idanb@xxxxxxxxxxxx>
> Signed-off-by: Max Gurtovoy <maxg@xxxxxxxxxxxx>
> Signed-off-by: Leon Romanovsky <leonro@xxxxxxxxxxxx>
Thanks, applied to for-next.
--
Doug Ledford <dledford@xxxxxxxxxx>
GPG KeyID: B826A3330E572FDD
Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDDAttachment:
signature.asc
Description: This is a digitally signed message part
