Hi Bob, Thanks for clarifying! I am looking forward to testing larger MTUs with the new functionality! Cheers, Christian On Fri, Feb 11, 2022 at 3:04 AM Yanjun Zhu <yanjun.zhu@xxxxxxxxx> wrote: > > 在 2022/2/10 13:13, Pearson, Robert B 写道: > > Christian, > > > > There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s > > Thanks, I have also reached the big bandwidth with the same methods. > How about latency of soft roce? > > Zhu Yanjun > > > in local loopback on a single 12 core box (AMD 3900x) using 12 IO > threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head > of tree with some patches not yet upstream. > > > > Bob Pearson > > rpearsonhpe@xxxxxxxxx > > rpearson@xxxxxxx > > > > > > -----Original Message----- > > From: Christian Blume <chr.blume@xxxxxxxxx> > > Sent: Wednesday, February 9, 2022 9:34 PM > > To: RDMA mailing list <linux-rdma@xxxxxxxxxxxxxxx> > > Subject: Soft-RoCE performance > > > > Hello! > > > > I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with? > > > > When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests. > > > > This is between two machines running Ubuntu20.04 with the 5.11 kernel. > > > > Cheers, > > Christian >
