On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote: > Dear experts on RDMA, > We are sorry to disturb you. Because of a project, we need to > integrate soft-RoCE in our system. However ,we are very confused by our > soft-RoCE throughput results, which are quite low compared with TCP > throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw > and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the > two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via > back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The > throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps, > respectively. But in the figure 6 in the attached paper "A Performance > Comparison of Container Networking Alternatives", the throughput of > soft-RoCE can be up to 23 Gbps. In our tests, we get the open-source > soft-RoCE from github in https://github.com/linux-rdma. Do you know how can > we get such high bandwidth? Do we need to configure some OS system settings? > We find that in 2017, someone finds the same problem and he posts all > his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951 ; > . But it remains unsolved. His results are nearly the same with our's. For > simplicity, we do not post our results in this email. You can get very > detailed information in the web page listed above. > We are very confused by our results. We will very appreciate it if we > can receive your early reply. Best wishes, > Wang Qi Can you please fix your email client? The email text looks like one big sentence. >From the perf report attached to this bugzilla, looks like RXE does a lot of CRC32 calculations and it is consistent with what Matan said a long time ago, RXE "stuck" in ICRC calculations required by spec. I'm curios what are your CONFIG_CRYPTO_* configs? Thanks >
