Re: [PATCH net-next v2 0/6] net/smc: Introduce virtually contiguous buffers for SMC-R

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 14.07.22 11:43, Wen Gu wrote:
On long-running enterprise production servers, high-order contiguous
memory pages are usually very rare and in most cases we can only get
fragmented pages.

When replacing TCP with SMC-R in such production scenarios, attempting
to allocate high-order physically contiguous sndbufs and RMBs may result
in frequent memory compaction, which will cause unexpected hung issue
and further stability risks.

So this patch set is aimed to allow SMC-R link group to use virtually
contiguous sndbufs and RMBs to avoid potential issues mentioned above.
Whether to use physically or virtually contiguous buffers can be set
by sysctl smcr_buf_type.

Note that using virtually contiguous buffers will bring an acceptable
performance regression, which can be mainly divided into two parts:

1) regression in data path, which is brought by additional address
    translation of sndbuf by RNIC in Tx. But in general, translating
    address through MTT is fast. According to qperf test, this part
    regression is basically less than 10% in latency and bandwidth.
    (see patch 5/6 for details)

2) regression in buffer initialization and destruction path, which is
    brought by additional MR operations of sndbufs. But thanks to link
    group buffer reuse mechanism, the impact of this kind of regression
    decreases as times of buffer reuse increases.

Patch set overview:
- Patch 1/6 and 2/6 mainly about simplifying and optimizing DMA sync
   operation, which will reduce overhead on the data path, especially
   when using virtually contiguous buffers;
- Patch 3/6 and 4/6 introduce a sysctl smcr_buf_type to set the type
   of buffers in new created link group;
- Patch 5/6 allows SMC-R to use virtually contiguous sndbufs and RMBs,
   including buffer creation, destruction, MR operation and access;
- patch 6/6 extends netlink attribute for buffer type of SMC-R link group;

v1->v2:
- Patch 5/6 fixes build issue on 32bit;
- Patch 3/6 adds description of new sysctl in smc-sysctl.rst;

Guangguan Wang (2):
   net/smc: remove redundant dma sync ops
   net/smc: optimize for smc_sndbuf_sync_sg_for_device and
     smc_rmb_sync_sg_for_cpu

Wen Gu (4):
   net/smc: Introduce a sysctl for setting SMC-R buffer type
   net/smc: Use sysctl-specified types of buffers in new link group
   net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
   net/smc: Extend SMC-R link group netlink attribute

  Documentation/networking/smc-sysctl.rst |  13 ++
  include/net/netns/smc.h                 |   1 +
  include/uapi/linux/smc.h                |   1 +
  net/smc/af_smc.c                        |  68 +++++++--
  net/smc/smc_clc.c                       |   8 +-
  net/smc/smc_clc.h                       |   2 +-
  net/smc/smc_core.c                      | 246 +++++++++++++++++++++-----------
  net/smc/smc_core.h                      |  20 ++-
  net/smc/smc_ib.c                        |  44 +++++-
  net/smc/smc_ib.h                        |   2 +
  net/smc/smc_llc.c                       |  33 +++--
  net/smc/smc_rx.c                        |  92 +++++++++---
  net/smc/smc_sysctl.c                    |  11 ++
  net/smc/smc_tx.c                        |  10 +-
  14 files changed, 404 insertions(+), 147 deletions(-)

This idea is very cool! Thank you for your effort! But we still need to verify if this solution can run well on our system. I'll come to you soon.



[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux