The unfortunate reality is there are increasingly large systems that are shipping today without support for GICv4 vLPI injection. Serialization in KVM's LPI routing/injection code has been a significant bottleneck for VMs on these machines when under a high load of LPIs (e.g. a multi-queue NIC). Even though the long-term solution is quite clearly **direct injection**, we really ought to do something about the LPI scaling issues within KVM. This series aims to improve the performance of LPI routing/injection in KVM by moving readers of LPI configuration data away from the lpi_list_lock in favor or using RCU. Patches 1-5 change out the representation of LPIs in KVM from a linked-list to an xarray. While not strictly necessary for making the locking improvements, this seems to be an opportune time to switch to a data structure that can actually be indexed. Patches 6-10 transition vgic_get_lpi() and vgic_put_lpi() away from taking the lpi_list_lock in favor of using RCU for protection. Note that this requires some rework to the way references are taken on LPIs and how reclaim works to be RCU safe. Lastly, patches 11-15 rework the LRU policy on the LPI translation cache to not require moving elements in the linked-list and take advantage of this to make it an rculist readable outside of the lpi_list_lock. All of this was tested on top of v6.8-rc1. Apologies if any of the changelogs are a bit too light, I'm happy to rework those further in subsequent revisions. I would've liked to have benchmark data showing the improvement on top of upstream with this series, but I'm currently having issues with our internal infrastructure and upstream kernels. However, this series has been found to have a near 2x performance improvement to redis-memtier [*] benchmarks on our kernel tree. [*] https://github.com/RedisLabs/memtier_benchmark Oliver Upton (15): KVM: arm64: vgic: Store LPIs in an xarray KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi() KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list() KVM: arm64: vgic: Get rid of the LPI linked-list KVM: arm64: vgic: Use atomics to count LPIs KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi() KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq() KVM: arm64: vgic-its: Lazily allocate LPI translation cache KVM: arm64: vgic-its: Pick cache victim based on usage count KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads arch/arm64/kvm/vgic/vgic-debug.c | 2 +- arch/arm64/kvm/vgic/vgic-init.c | 7 +- arch/arm64/kvm/vgic/vgic-its.c | 190 ++++++++++++++++++------------- arch/arm64/kvm/vgic/vgic-v3.c | 3 +- arch/arm64/kvm/vgic/vgic.c | 56 +++------ arch/arm64/kvm/vgic/vgic.h | 12 +- include/kvm/arm_vgic.h | 9 +- 7 files changed, 146 insertions(+), 133 deletions(-) base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d -- 2.43.0.429.g432eaa2c6b-goog
