On 2024/7/15 10:53, Guangguan Wang wrote: > > > On 2024/7/11 23:57, Wenjia Zhang wrote: >> >> >> On 09.07.24 18:05, Guangguan Wang wrote: >>> When sending large size data in TCP, the data will be split into >>> several segments(packets) to transfer due to MTU config. And in >>> the receive side, application can be woken up to recv data every >>> packet arrived, the data transmission and data recv copy are >>> pipelined. >>> >>> But for SMC-R, it will transmit as many data as possible in one >>> RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive >>> size, the application only be woken up to recv data when all RDMA >>> WRITE data and the followed CDC msg arrived. The data transmission >>> and data recv copy are sequential. >>> >>> This patch introduce autosplit for SMC, which can automatic split >>> data into several segments and every segment transmitted by one RDMA >>> WRITE when sending large size data in SMC. Because of the split, the >>> data transmission and data send copy can be pipelined in the send side, >>> and the data transmission and data recv copy can be pipelined in the >>> receive side. Thus autosplit helps improving latency performance when >>> sending large size data. The autosplit also works for SMC-D. >>> >>> This patch also introduce a sysctl names autosplit_size for configure >>> the max size of the split segment, whose default value is 128KiB >>> (128KiB perform best in my environment). >>> >>> The sockperf benchmark shows 17%-28% latency improvement when msgsize >>>> = 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB >>> for SMC-D with smc-loopback. >>> >>> Test command: >>> sockperf sr --tcp -m 1048575 >>> sockperf pp --tcp -i <server ip> -m <msgsize> -t 20 >>> >>> Test config: >>> sysctl -w net.smc.wmem=524288 >>> sysctl -w net.smc.rmem=524288 >>> >>> Test results: >>> SMC-R >>> msgsize noautosplit autosplit >>> 128KB 55.546 us 55.763 us >>> 256KB 83.537 us 69.743 us (17% improve) >>> 512KB 138.306 us 100.313 us (28% improve) >>> 1MB 273.702 us 197.222 us (28% improve) >>> >>> SMC-D with smc-loopback >>> msgsize noautosplit autosplit >>> 128KB 14.672 us 14.690 us >>> 256KB 28.277 us 23.958 us (15% improve) >>> 512KB 63.047 us 45.339 us (28% improve) >>> 1MB 129.306 us 87.278 us (32% improve) >>> >>> Signed-off-by: Guangguan Wang <guangguan.wang@xxxxxxxxxxxxxxxxx> >>> --- >>> Documentation/networking/smc-sysctl.rst | 11 +++++++++++ >>> include/net/netns/smc.h | 1 + >>> net/smc/smc_sysctl.c | 12 ++++++++++++ >>> net/smc/smc_tx.c | 19 ++++++++++++++++++- >>> 4 files changed, 42 insertions(+), 1 deletion(-) >>> >> >> Hi Guangguan, >> >> If I remember correctly, the intention to use one RDMA-write for a possible large data is to reduce possible many partial stores. Since many year has gone, I'm not that sure if it would still be an issue. I need some time to check on it. >> > > Did you mean too many partial stores will result in some issue? What's the issue? > > >> BTW, I don't really like the idea to use sysctl to set the autosplit_size in any value at will. That makes no sense to improve the performance. > > Although 128KB autosplit_size have a good performance in most scenario, I still found some better autosplit_size for some specific network configurations. > For example, 128KB autosplit_size have a good performance whether the MTU is 1500 or 8500, but for 8500 MTU, 64KB autosplit_size performs better. > > Maybe the sysctl is not the best way, but I think it should have a way to set the value of autosplit_size for possible performance tuning. > > Thanks, > Guangguan Wang > Hi Wenjia, Is there any update comment or information about this patch? >> >> Thanks, >> Wenjia
