在 2022/2/10 13:13, Pearson, Robert B 写道:
Christian,
There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s
Thanks, I have also reached the big bandwidth with the same methods.
How about latency of soft roce?
Zhu Yanjun
in local loopback on a single 12 core box (AMD 3900x) using 12 IO
threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head
of tree with some patches not yet upstream.
Bob Pearson
rpearsonhpe@xxxxxxxxx
rpearson@xxxxxxx
-----Original Message-----
From: Christian Blume <chr.blume@xxxxxxxxx>
Sent: Wednesday, February 9, 2022 9:34 PM
To: RDMA mailing list <linux-rdma@xxxxxxxxxxxxxxx>
Subject: Soft-RoCE performance
Hello!
I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with?
When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests.
This is between two machines running Ubuntu20.04 with the 5.11 kernel.
Cheers,
Christian
