On 2024/7/11 23:57, Wenjia Zhang wrote: > > > On 09.07.24 18:05, Guangguan Wang wrote: >> When sending large size data in TCP, the data will be split into >> several segments(packets) to transfer due to MTU config. And in >> the receive side, application can be woken up to recv data every >> packet arrived, the data transmission and data recv copy are >> pipelined. >> >> But for SMC-R, it will transmit as many data as possible in one >> RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive >> size, the application only be woken up to recv data when all RDMA >> WRITE data and the followed CDC msg arrived. The data transmission >> and data recv copy are sequential. >> >> This patch introduce autosplit for SMC, which can automatic split >> data into several segments and every segment transmitted by one RDMA >> WRITE when sending large size data in SMC. Because of the split, the >> data transmission and data send copy can be pipelined in the send side, >> and the data transmission and data recv copy can be pipelined in the >> receive side. Thus autosplit helps improving latency performance when >> sending large size data. The autosplit also works for SMC-D. >> >> This patch also introduce a sysctl names autosplit_size for configure >> the max size of the split segment, whose default value is 128KiB >> (128KiB perform best in my environment). >> >> The sockperf benchmark shows 17%-28% latency improvement when msgsize >>> = 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB >> for SMC-D with smc-loopback. >> >> Test command: >> sockperf sr --tcp -m 1048575 >> sockperf pp --tcp -i <server ip> -m <msgsize> -t 20 >> >> Test config: >> sysctl -w net.smc.wmem=524288 >> sysctl -w net.smc.rmem=524288 >> >> Test results: >> SMC-R >> msgsize noautosplit autosplit >> 128KB 55.546 us 55.763 us >> 256KB 83.537 us 69.743 us (17% improve) >> 512KB 138.306 us 100.313 us (28% improve) >> 1MB 273.702 us 197.222 us (28% improve) >> >> SMC-D with smc-loopback >> msgsize noautosplit autosplit >> 128KB 14.672 us 14.690 us >> 256KB 28.277 us 23.958 us (15% improve) >> 512KB 63.047 us 45.339 us (28% improve) >> 1MB 129.306 us 87.278 us (32% improve) >> >> Signed-off-by: Guangguan Wang <guangguan.wang@xxxxxxxxxxxxxxxxx> >> --- >> Documentation/networking/smc-sysctl.rst | 11 +++++++++++ >> include/net/netns/smc.h | 1 + >> net/smc/smc_sysctl.c | 12 ++++++++++++ >> net/smc/smc_tx.c | 19 ++++++++++++++++++- >> 4 files changed, 42 insertions(+), 1 deletion(-) >> > > Hi Guangguan, > > If I remember correctly, the intention to use one RDMA-write for a possible large data is to reduce possible many partial stores. Since many year has gone, I'm not that sure if it would still be an issue. I need some time to check on it. > Did you mean too many partial stores will result in some issue? What's the issue? > BTW, I don't really like the idea to use sysctl to set the autosplit_size in any value at will. That makes no sense to improve the performance. Although 128KB autosplit_size have a good performance in most scenario, I still found some better autosplit_size for some specific network configurations. For example, 128KB autosplit_size have a good performance whether the MTU is 1500 or 8500, but for 8500 MTU, 64KB autosplit_size performs better. Maybe the sysctl is not the best way, but I think it should have a way to set the value of autosplit_size for possible performance tuning. Thanks, Guangguan Wang > > Thanks, > Wenjia
