On 28.05.24 15:51, Guangguan Wang wrote:
SMCR_RMBE_SIZES is the upper boundary of SMC-R's snd_buf and rcv_buf.
The maximum bytes of snd_buf and rcv_buf can be calculated by 2^SMCR_
RMBE_SIZES * 16KB. SMCR_RMBE_SIZES = 5 means the upper boundary is 512KB.
TCP's snd_buf and rcv_buf max size is configured by net.ipv4.tcp_w/rmem[2]
whose defalut value is 4MB or 6MB, is much larger than SMC-R's upper
boundary.
In some scenarios, such as Recommendation System, the communication
pattern is mainly large size send/recv, where the size of snd_buf and
rcv_buf greatly affects performance. Due to the upper boundary
disadvantage, SMC-R performs poor than TCP in those scenarios. So it
is time to enlarge the upper boundary size of SMC-R's snd_buf and rcv_buf,
so that the SMC-R's snd_buf and rcv_buf can be configured to larger size
for performance gain in such scenarios.
The SMC-R rcv_buf's size will be transferred to peer by the field
rmbe_size in clc accept and confirm message. The length of the field
rmbe_size is four bits, which means the maximum value of SMCR_RMBE_SIZES
is 15. In case of frequently adjusting the value of SMCR_RMBE_SIZES
in different scenarios, set the value of SMCR_RMBE_SIZES to the maximum
value 15, which means the upper boundary of SMC-R's snd_buf and rcv_buf
is 512MB. As the real memory usage is determined by the value of
net.smc.w/rmem, not by the upper boundary, set the value of SMCR_RMBE_SIZES
to the maximum value has no side affects.
Hi Guangguan,
That is correct that the maximum buffer(snd_buf and rcv_buf) size of
SMCR is much smaller than TCP's. If I remember correctly, that was
because the 512KB was enough for the traffic and did not waist memory
space after some experiment. Sure, that was years ago, and it could be
very different nowadays. But I'm still curious if you have any concrete
scenario with performance benchmark which shows the distinguish
disadvantage of the current maximum buffer size.
Thanks,
Wenjia
Guangguan Wang (2):
net/smc: set rmb's SG_MAX_SINGLE_ALLOC limitation only when
CONFIG_ARCH_NO_SG_CHAIN is defined
net/smc: change SMCR_RMBE_SIZES from 5 to 15
net/smc/smc_core.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
