> -----Original Message----- > From: Marc Zyngier [mailto:maz@xxxxxxxxxx] > Sent: 07 September 2023 11:09 > To: kvmarm@xxxxxxxxxxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; > kvm@xxxxxxxxxxxxxxx > Cc: James Morse <james.morse@xxxxxxx>; Suzuki K Poulose > <suzuki.poulose@xxxxxxx>; Oliver Upton <oliver.upton@xxxxxxxxx>; > yuzenghui <yuzenghui@xxxxxxxxxx>; Xu Zhao <zhaoxu.35@xxxxxxxxxxxxx> > Subject: [PATCH 0/5] KVM: arm64: Accelerate lookup of vcpus by MPIDR > values > > Xu Zhao recently reported[1] that sending SGIs on large VMs was slower > than expected, specially if targeting vcpus that have a high vcpu > index. They root-caused it to the way we walk the vcpu xarray in the > search of the correct MPIDR, one vcpu at a time, which is of course > grossly inefficient. > > The solution they proposed was, unfortunately, less than ideal, but I > was "nerd snipped" into doing something about it. > > The main idea is to build a small hash table of MPIDR to vcpu > mappings, using the fact that most of the time, the MPIDR values only > use a small number of significant bits and that we can easily compute > a compact index from it. Once we have that, accelerating vcpu lookup > becomes pretty cheap, and we can in turn make SGIs great again. > > It must be noted that since the MPIDR values are controlled by > userspace, it isn't always possible to allocate the hash table > (userspace could build a 32 vcpu VM and allocate one bit of affinity > to each of them, making all the bits significant). We thus always have > an iterative fallback -- if it hurts, don't do that. > > Performance wise, this is very significant: using the KUT micro-bench > test with the following patch (always IPI-ing the last vcpu of the VM) > and running it with large number of vcpus shows a large improvement > (from 3832ns to 2593ns for a 64 vcpu VM, a 32% reduction, measured on > an Ampere Altra). I expect that IPI-happy workloads could benefit from > this. Hi Marc, Tested on a HiSilicon D06 test board using KUT micro-bench(+ the changes) with a 64 vCPU VM. From an avg. of 5 runs, observed around ~54% improvement for IPI (from 5309ns to 2413ns). FWIW, Tested-by: Shameer Kolothum <shameerali.kolothum.thodi@xxxxxxxxxx> Thanks, Shameer
