On 08.08.24 08:26, Guangguan Wang wrote:
On 2024/7/15 10:53, Guangguan Wang wrote:
On 2024/7/11 23:57, Wenjia Zhang wrote:
On 09.07.24 18:05, Guangguan Wang wrote:
When sending large size data in TCP, the data will be split into
several segments(packets) to transfer due to MTU config. And in
the receive side, application can be woken up to recv data every
packet arrived, the data transmission and data recv copy are
pipelined.
But for SMC-R, it will transmit as many data as possible in one
RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive
size, the application only be woken up to recv data when all RDMA
WRITE data and the followed CDC msg arrived. The data transmission
and data recv copy are sequential.
This patch introduce autosplit for SMC, which can automatic split
data into several segments and every segment transmitted by one RDMA
WRITE when sending large size data in SMC. Because of the split, the
data transmission and data send copy can be pipelined in the send side,
and the data transmission and data recv copy can be pipelined in the
receive side. Thus autosplit helps improving latency performance when
sending large size data. The autosplit also works for SMC-D.
This patch also introduce a sysctl names autosplit_size for configure
the max size of the split segment, whose default value is 128KiB
(128KiB perform best in my environment).
The sockperf benchmark shows 17%-28% latency improvement when msgsize
= 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB
for SMC-D with smc-loopback.
Test command:
sockperf sr --tcp -m 1048575
sockperf pp --tcp -i <server ip> -m <msgsize> -t 20
Test config:
sysctl -w net.smc.wmem=524288
sysctl -w net.smc.rmem=524288
Test results:
SMC-R
msgsize noautosplit autosplit
128KB 55.546 us 55.763 us
256KB 83.537 us 69.743 us (17% improve)
512KB 138.306 us 100.313 us (28% improve)
1MB 273.702 us 197.222 us (28% improve)
SMC-D with smc-loopback
msgsize noautosplit autosplit
128KB 14.672 us 14.690 us
256KB 28.277 us 23.958 us (15% improve)
512KB 63.047 us 45.339 us (28% improve)
1MB 129.306 us 87.278 us (32% improve)
Signed-off-by: Guangguan Wang <guangguan.wang@xxxxxxxxxxxxxxxxx>
---
Documentation/networking/smc-sysctl.rst | 11 +++++++++++
include/net/netns/smc.h | 1 +
net/smc/smc_sysctl.c | 12 ++++++++++++
net/smc/smc_tx.c | 19 ++++++++++++++++++-
4 files changed, 42 insertions(+), 1 deletion(-)
Hi Guangguan,
If I remember correctly, the intention to use one RDMA-write for a possible large data is to reduce possible many partial stores. Since many year has gone, I'm not that sure if it would still be an issue. I need some time to check on it.
Did you mean too many partial stores will result in some issue? What's the issue?
Forget it, I did verify that the partial stores should not be problem now.
BTW, I don't really like the idea to use sysctl to set the autosplit_size in any value at will. That makes no sense to improve the performance.
Although 128KB autosplit_size have a good performance in most scenario, I still found some better autosplit_size for some specific network configurations.
For example, 128KB autosplit_size have a good performance whether the MTU is 1500 or 8500, but for 8500 MTU, 64KB autosplit_size performs better.
Maybe the sysctl is not the best way, but I think it should have a way to set the value of autosplit_size for possible performance tuning.
mhhh, that could be a good reason to use sysctl.
Thanks,
Guangguan Wang
Hi Wenjia,
Is there any update comment or information about this patch?
Hi Guangguan,
sorry for the delayed answer. In the last time it is really difficult
for me to find time to look into it and test it. With more thinking, I'm
kind of convinced with this idea. But test is still needed. I'll be in
vacation next 3 weeks. I hope it is okay for you that I'll test it as
soon as possible when I'm back. If everything is ok, I think we can let
it go upstream.
Thanks,
Wenjia
Thanks,
Wenjia
