On 10/21/21 3:17 PM, zhenwei pi wrote:
On 10/21/21 1:03 PM, Wanpeng Li wrote:
On Thu, 21 Oct 2021 at 11:05, zhenwei pi <pizhenwei@xxxxxxxxxxxxx> wrote:
On 10/21/21 4:12 AM, Sean Christopherson wrote:
On Wed, Oct 20, 2021, Wanpeng Li wrote:
On Wed, 20 Oct 2021 at 20:08, zhenwei pi <pizhenwei@xxxxxxxxxxxxx>
wrote:
Although host side exposes KVM PV SEND IPI feature to guest side,
guest should still have a chance to disable it.
A typicall case of this parameter:
If the host AMD server enables AVIC feature, the flat mode of APIC
get better performance in the guest.
Hmm, I didn't find enough valuable information in your posting. We
observe AMD a lot before.
https://lore.kernel.org/all/CANRm+Cx597FNRUCyVz1D=B6Vs2GX3Sw57X7Muk+yMpi_hb+v1w@xxxxxxxxxxxxxx/T/#u
I too would like to see numbers. I suspect the answer is going to
be that
AVIC performs poorly in CPU overcommit scenarios because of the cost
of managing
the tables and handling "failed delivery" exits, but that AVIC does
quite well
when vCPUs are pinned 1:1 and IPIs rarely require an exit to the host.
Test env:
CPU: AMD EPYC 7642 48-Core Processor
Kmod args(enable avic and disable nested):
modprobe kvm-amd nested=0 avic=1 npt=1
QEMU args(disable x2apic):
... -cpu host,x2apic=off ...
Benchmark tool:
https://github.com/bytedance/kvm-utils/tree/master/microbenchmark/apic-ipi
~# insmod apic_ipi.ko options=5 && dmesg -c
apic_ipi: 1 NUMA node(s)
apic_ipi: apic [flat]
apic_ipi: apic->send_IPI[default_send_IPI_single+0x0/0x40]
apic_ipi: apic->send_IPI_mask[kvm_send_ipi_mask+0x0/0x10]
apic_ipi: IPI[kvm_send_ipi_mask] from CPU[0] to CPU[1]
apic_ipi: total cycles 375671259, avg 3756
apic_ipi: IPI[flat_send_IPI_mask] from CPU[0] to CPU[1]
apic_ipi: total cycles 221961822, avg 2219
apic->send_IPI_mask[kvm_send_ipi_mask+0x0/0x10]
-> This line show current send_IPI_mask is kvm_send_ipi_mask(because
of PV SEND IPI FEATURE)
apic_ipi: IPI[kvm_send_ipi_mask] from CPU[0] to CPU[1]
apic_ipi: total cycles 375671259, avg 3756
-->These lines show the average cycles of each kvm_send_ipi_mask:
3756
apic_ipi: IPI[flat_send_IPI_mask] from CPU[0] to CPU[1]
apic_ipi: total cycles 221961822, avg 2219
-->These lines show the average cycles of each
flat_send_IPI_mask: 2219
Just single target IPI is not eough.
Wanpeng
Benchmark smp_call_function_single
(https://github.com/bytedance/kvm-utils/blob/master/microbenchmark/ipi-bench/ipi_bench.c):
Test env:
CPU: AMD EPYC 7642 48-Core Processor
Kmod args(enable avic and disable nested):
modprobe kvm-amd nested=0 avic=1 npt=1
QEMU args(disable x2apic):
... -cpu host,x2apic=off ...
1> without no-kvm-pvipi:
ipi_bench_single wait[1], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
elapsed = 424945631 cycles, average = 4249 cycles
ipitime = 385246136 cycles, average = 3852 cycles
ipi_bench_single wait[0], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
elapsed = 419057953 cycles, average = 4190 cycles
2> with no-kvm-pvipi:
ipi_bench_single wait[1], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
elapsed = 321756407 cycles, average = 3217 cycles
ipitime = 299433550 cycles, average = 2994 cycles
ipi_bench_single wait[0], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
elapsed = 295382146 cycles, average = 2953 cycles
Hi, Wanpeng & Sean
Also benchmark redis(by 127.0.0.1) in a guest(2vCPU), 'no-kvm-pvipi'
gets better performance.
Test env:
Host side: pin 2vCPU on 2core in a die.
Guest side: run command:
taskset -c 1 ./redis-server --appendonly no
taskset -c 0 ./redis-benchmark -h 127.0.0.1 -d 1024 -n 10000000 -t get
1> without no-kvm-pvipi:
redis QPS: 193203.12 requests per second
kvm_pv_send_ipi exit: ~18K/s
2> with no-kvm-pvipi:
redis QPS: 196028.47 requests per second
avic_incomplete_ipi_interception exit: ~5K/s
--
zhenwei pi
