Why our soft-RoCE throughput is quite low compared with TCP

[QWang <3100102071@xxxxxxxxxx>]

Dear experts on RDMA,
      We are sorry to disturb you. Because of a project, we need to 
integrate soft-RoCE in our system. However ,we are very confused by our 
soft-RoCE throughput results, which are quite low compared with TCP 
throughput. The throughput of soft-RoCE in our tests measured by 
ib_send_bw and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 
Gbps and the two Xeon E5 servers with Mellanox ConnectX-4 cards are 
connected via back-to-back, the OS is ubuntu16.04 with kernel 
4.15.0-041500-generic). The throughput of hard-RoCE and TCP are normal, 
which are 100 Gbps and 20 Gbps, respectively. But in the figure 6 in the 
attached paper "A Performance Comparison of Container Networking 
Alternatives", the throughput of soft-RoCE can be up to 23 Gbps.  In our 
tests, we get the open-source soft-RoCE from github in 
https://github.com/linux-rdma. Do you know how can we get such high 
bandwidth? Do we need to configure some OS system settings?
      We find that in 2017, someone finds the same problem and he posts 
all his detailed results on 
https://bugzilla.kernel.org/show_bug.cgi?id=190951  ; . But it remains 
unsolved. His results are nearly the same with our's. For simplicity,  
we do not post our results in this email. You can get very detailed 
information in the web page listed above.
      We are very confused by our results. We will very appreciate it 
if we can receive your early reply. Best wishes,
Wang Qi