Hi, Bibo, I have done an off-list discussion with some KVM experts, and they think user-space have its right to know PV features, so cpucfg solution is acceptable. And I applied this series with some modifications at https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.git/log/?h=loongarch-kvm You can test it now. But it seems the upstream qemu cannot enable PV IPI now. I will reply to other patches about my modifications. Huacai On Sun, Apr 28, 2024 at 6:05 PM Bibo Mao <maobibo@xxxxxxxxxxx> wrote: > > On physical machine, ipi HW uses IOCSR registers, however there is trap > into hypervisor when vcpu accesses IOCSR registers if system is in VM > mode. SWI is a interrupt mechanism like SGI on ARM, software can send > interrupt to CPU, only that on LoongArch SWI can only be sent to local CPU > now. So SWI can not used for IPI on real HW system, however it can be used > on VM when combined with hypercall method. IPI can be sent with hypercall > method and SWI interrupt is injected to vcpu, vcpu can treat SWI > interrupt as IPI. > > With PV IPI supported, there is one trap with IPI sending, however with IPI > receiving there is no trap. with IOCSR HW ipi method, there will be one > trap with IPI sending and two trap with ipi receiving. > > Also IPI multicast support is added for VM, the idea comes from x86 PV ipi. > IPI can be sent to 128 vcpus in one time. With IPI multicast support, trap > will be reduced greatly. > > Here is the microbenchmarck data with "perf bench futex wake" testcase on > 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine, > VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16 > threads, the performance is better if data is smaller. > > physical machine 0.0176 ms > VM original 0.1140 ms > VM with pv ipi patch 0.0481 ms > > It passes to boot with 128/256 vcpus, and passes to run runltp command > with package ltp-20230516. > > --- > v7 --- v8: > 1. Remove kernel PLV mode checking with cpucfg emulation for hypervisor > feature inquiry. > 2. Remove document about loongarch hypercall ABI per request of huacai, > will add English/Chinese doc at the same time in later. > > v6 --- v7: > 1. Refine LoongArch virt document by review comments. > 2. Add function kvm_read_reg()/kvm_write_reg() in hypercall emulation, > and later it can be used for other trap emulations. > > v5 --- v6: > 1. Add privilege checking when emulating cpucfg at index 0x4000000 -- > 0x400000FF, return 0 if not executed at kernel mode. > 2. Add document about LoongArch pv ipi with new creatly directory > Documentation/virt/kvm/loongarch/ > 3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(), > where min should plus BITS_PER_LONG with second bitmap, otherwise > VM with more than 64 vpus fails to boot. > 4. Adjust patch order and code refine with review comments. > > v4 --- v5: > 1. Refresh function/macro name from review comments. > > v3 --- v4: > 1. Modfiy pv ipi hook function name call_func_ipi() and > call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv > ipi is used for both remote function call and reschedule notification. > 2. Refresh changelog. > > v2 --- v3: > 1. Add 128 vcpu ipi multicast support like x86 > 2. Change cpucfg base address from 0x10000000 to 0x40000000, in order > to avoid confliction with future hw usage > 3. Adjust patch order in this patchset, move patch > Refine-ipi-ops-on-LoongArch-platform to the first one. > > v1 --- v2: > 1. Add hw cpuid map support since ipi routing uses hw cpuid > 2. Refine changelog description > 3. Add hypercall statistic support for vcpu > 4. Set percpu pv ipi message buffer aligned with cacheline > 5. Refine pv ipi send logic, do not send ipi message with if there is > pending ipi message. > --- > Bibo Mao (6): > LoongArch/smp: Refine some ipi functions on LoongArch platform > LoongArch: KVM: Add hypercall instruction emulation support > LoongArch: KVM: Add cpucfg area for kvm hypervisor > LoongArch: KVM: Add vcpu search support from physical cpuid > LoongArch: KVM: Add pv ipi support on kvm side > LoongArch: Add pv ipi support on guest kernel side > > arch/loongarch/Kconfig | 9 + > arch/loongarch/include/asm/Kbuild | 1 - > arch/loongarch/include/asm/hardirq.h | 5 + > arch/loongarch/include/asm/inst.h | 1 + > arch/loongarch/include/asm/irq.h | 10 +- > arch/loongarch/include/asm/kvm_host.h | 27 +++ > arch/loongarch/include/asm/kvm_para.h | 155 ++++++++++++++++++ > arch/loongarch/include/asm/kvm_vcpu.h | 11 ++ > arch/loongarch/include/asm/loongarch.h | 11 ++ > arch/loongarch/include/asm/paravirt.h | 27 +++ > .../include/asm/paravirt_api_clock.h | 1 + > arch/loongarch/include/asm/smp.h | 31 ++-- > arch/loongarch/include/uapi/asm/Kbuild | 2 - > arch/loongarch/kernel/Makefile | 1 + > arch/loongarch/kernel/irq.c | 24 +-- > arch/loongarch/kernel/paravirt.c | 151 +++++++++++++++++ > arch/loongarch/kernel/perf_event.c | 14 +- > arch/loongarch/kernel/smp.c | 62 ++++--- > arch/loongarch/kernel/time.c | 12 +- > arch/loongarch/kvm/exit.c | 132 +++++++++++++-- > arch/loongarch/kvm/vcpu.c | 94 ++++++++++- > arch/loongarch/kvm/vm.c | 11 ++ > 22 files changed, 690 insertions(+), 102 deletions(-) > create mode 100644 arch/loongarch/include/asm/kvm_para.h > create mode 100644 arch/loongarch/include/asm/paravirt.h > create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h > delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild > create mode 100644 arch/loongarch/kernel/paravirt.c > > > base-commit: 5eb4573ea63d0c83bf58fb7c243fc2c2b6966c02 > -- > 2.39.3 > >
