On Thu, May 18, 2023 5:22 PM Daisuke Matsuda wrote:
> Both responder and completer need to sleep to execute page-fault when used
> with ODP. It can happen when they are going to access user MRs, so tasks
> must be executed in process context for such cases.
>
> Additionally, current implementation seldom defers tasks to workqueue, but
> instead defers to a softirq context running do_task(). It is called from
> rxe_resp_queue_pkt() and rxe_comp_queue_pkt() in SOFTIRQ_NET_RX context and
> can last until maximum RXE_MAX_ITERATIONS (=1024) loops are executed. The
> problem is the that task execuion appears to be anonymous loads in the
> system and that the loop can throttle other softirqs on the same CPU.
>
> This patch makes responder and completer codes run in process context for
> ODP and the problem described above.
>
> Signed-off-by: Daisuke Matsuda <matsuda-daisuke@xxxxxxxxxxx>
> ---
Some people may be concerned about performance change caused by this patch,
so I measured the performance using 2 VMs on the same host.
Here are the settings:
Host:
CPU cores: 32
MEM 64GiB
Hypervisor: KVM
Guest:
vCPUs: 8
Mem: 12 GiB
OS: CentOS 9 Stream
kernel vers: 6.4.0-rc2 (latest jgg-for-next as of 5/18, 2023)
Conditions:
Operations: send, write, read
I measured the performance of the latest jgg-for-next kernel. After that I applied all ODP
patches and measured the performance again. The results are shown below. The performance
often fluctuates, so it is difficult to say which one is better. However, it seems there
is no regression after the change.
## latest jgg-for-next (Current)
[send]
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 258.32 0.004133
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 335.09 0.005361
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 240.24 0.003844
---------------------------------------------------------------------------------------
[write]
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 281.40 0.004502
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 347.02 0.005552
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 280.36 0.004486
---------------------------------------------------------------------------------------
[read]
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 271.30 0.004341
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 266.44 0.004263
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 226.94 0.003631
---------------------------------------------------------------------------------------
## After applying the ODP patches
send
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 289.26 0.004628
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 268.85 0.004302
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 320.81 0.005133
---------------------------------------------------------------------------------------
write
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 288.98 0.004624
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 261.98 0.004192
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 274.59 0.004393
---------------------------------------------------------------------------------------
read
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 324.56 0.005193
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 294.21 0.004707
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 200000 0.00 256.60 0.004106
---------------------------------------------------------------------------------------
