On Mon, Mar 21, 2022, Paolo Bonzini wrote: > On 3/18/22 17:48, Paolo Bonzini wrote: > > This reverts commit cf3e26427c08ad9015956293ab389004ac6a338e. > > > > Multi-vCPU Hyper-V guests started crashing randomly on boot with the > > latest kvm/queue and the problem can be bisected the problem to this > > particular patch. Basically, I'm not able to boot e.g. 16-vCPU guest > > successfully anymore. Both Intel and AMD seem to be affected. Reverting > > the commit saves the day. > > > > Reported-by: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> > > Signed-off-by: Paolo Bonzini <pbonzini@xxxxxxxxxx> > > This is not enough, the following is also needed to account > for "KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow > pages": > > ------------------- 8< ---------------- > From: Paolo Bonzini <pbonzini@xxxxxxxxxx> > Subject: [PATCH] kvm: x86/mmu: Flush TLB before zap_gfn_range releases RCU > > Since "KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()" > is going to be reverted, it's not going to be true anymore that > the zap-page flow does not free any 'struct kvm_mmu_page'. Introduce > an early flush before tdp_mmu_zap_leafs() returns, to preserve > bisectability. Can I have 1-2 weeks to try and root cause and fix the underlying issue before sending reverts to Linus? I really don't want to paper over a TLB flushing bug or an off-by-one bug, and I really, really don't want to end up with another scenario where KVM zaps everything just because. Vitaly, can you provide repro instructions? A nearly-complete QEMU command line would be wonderful :-) Is the issue unique to any particular guest kernel? I've been unable to repro with a 112 vCPU Linux guest with these Hyper-V enlightenments: $ : dm | grep -i hyper-v [ 0.000000] Hypervisor detected: Microsoft Hyper-V [ 0.000000] Hyper-V: privilege flags low 0x2aff, high 0x830, hints 0x4e2c, misc 0x80d12 [ 0.000000] Hyper-V Host Build:14393-10.0-0-0.0 [ 0.000000] Hyper-V: Nested features: 0x80101 [ 0.000000] Hyper-V: LAPIC Timer Frequency: 0x3d0900 [ 0.000000] Hyper-V: Using hypercall for remote TLB flush [ 0.000004] tsc: Marking TSC unstable due to running on Hyper-V [ 0.129376] Booting paravirtualized kernel on Hyper-V [ 0.140419] Hyper-V: PV spinlocks disabled [ 0.247500] Hyper-V: Using IPI hypercalls [ 0.247502] Hyper-V: Using enlightened APIC (x2apic mode) Actually, since this is apparently specific to kvm_zap_gfn_range(), can you add printk "tracing" in update_mtrr(), kvm_post_set_cr0(), and __kvm_request_apicv_update() to see what is actually triggering zaps? Capturing the start and end GFNs would be very helpful for the MTRR case. The APICv update seems unlikely to affect only Hyper-V guests, though there is the auto EOI crud. And the other two only come into play with non-coherent DMA. In other words, figuring out exactly what sequence leads to failure should be straightforward.
