On Wed, May 03, 2023, Peter Xu wrote: > Oops, bounced back from the list.. > > Forward with no attachment this time - I assume the information is still > enough in the paragraphs even without the flamegraphs. The flamegraphs are definitely useful beyond what is captured here. Not sure how to get them accepted on the list though. > > From what I got there, vmx_vcpu_load() gets more highlights than the > > spinlocks. I think that's the tlb flush broadcast. No, it's KVM dealing with the vCPU being migrated to a different pCPU. The smp_call_function_single() that shows up is from loaded_vmcs_clear() and is triggered when KVM needs to VMCLEAR the VMCS on the _previous_ pCPU (yay for the VMCS caches not being coherent). Task migration can also trigger IBPB (if mitigations are enabled), and also does an "all contexts" INVEPT, i.e. flushes all TLB entries for KVM's MMU. Can you trying 1:1 pinning of vCPUs to pCPUs? That _should_ eliminate the vmx_vcpu_load_vmcs() hotspot, and for large VMs is likely represenative of a real world configuration.
