Dear KVM developers and maintainers, In our research work presented last week at the VEE 2021 conference [1], we found out that a lot of continuous Pause-Loop-Exiting (PLE) events occur due to three problems we have identified: 1) Linux CFS ignores hints from KVM; 2) IPI receiver vCPUs in user-mode are not boosted; 3) IPI-receiver that has halted is always a candidate for boost. We have intoduced two mitigations against the problems. To solve problem (1), patch 1 increases the vruntime of yielded vCPU to pass the check `if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1)` in `struct sched_entity * pick_next_entity()` if the cfs_rq's skip and next are both vCPUs in the same VM. To keep fairness it does not prioritize the guest VM which causes PLE, however it improves the performance by eliminating unnecessary PLE. Also we have confirmed `yield_to_task_fair` is called only from KVM. To solve problems (2) and (3), patch 2 monitors IPI communication between vCPUs and leverages the relationship between vCPUs to select boost candidates. The "[PATCH] KVM: Boost vCPU candidiate in user mode which is delivering interrupt" patch (https://lore.kernel.org/kvm/CANRm+Cy-78UnrkX8nh5WdHut2WW5NU=UL84FRJnUNjsAPK+Uww@xxxxxxxxxxxxxx/T/) seems to be effective for (2) while it only uses the IPI receiver information. Our approach reduces the total number of PLE events by up to 87.6 % in four 8-vCPU VMs in over-subscribed scenario with the Linux kernel 5.6.0. Please find the patch below. We would greatly appreciate your valuable feedback on our approach and patch. Thank you very much for your consideration Kenta Ishiguro [1] Kenta Ishiguro, Naoki Yasuno, Pierre-Louis Aublin, and Kenji Kono. "Mitigating excessive vCPU spinning in VM-agnostic KVM". In Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2021). Association for Computing Machinery, New York, NY, USA, 139--152. https://dl.acm.org/doi/abs/10.1145/3453933.3454020 Kenta Ishiguro (2): Prevent CFS from ignoring boost requests from KVM Boost vCPUs based on IPI-sender and receiver information arch/x86/kvm/lapic.c | 14 ++++++++++++++ arch/x86/kvm/vmx/vmx.c | 2 ++ include/linux/kvm_host.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 26 ++++++++++++++++++++++++-- 5 files changed, 76 insertions(+), 2 deletions(-) -- 2.30.2
