On Tue, May 30, 2023 at 1:10 PM Jim Mattson <jmattson@xxxxxxxxxx> wrote: > > On Mon, May 29, 2023 at 6:44 AM Maxim Levitsky <mlevitsk@xxxxxxxxxx> wrote: > > > > У пн, 2023-05-29 у 14:58 +0200, jwarren@xxxxxxxxxxxx пише: > > > Hello, > > > Since kernel 5.16 users can't start VMware VMs when it is nested under KVM on AMD CPUs. > > > > > > User reports are here: > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583 > > > https://forums.unraid.net/topic/128868-vmware-7x-will-not-start-any-vms-under-unraid-6110/ > > > > > > I've pinpointed it to commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 (appeared in 5.16rc1) > > > "nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB" > > > > > > I've confirmed that VMware errors out when it checks for TLB_CONTROL_FLUSH_ASID support and gets a 'false' answer. > > > > > > First revisions of the patch in question had some support for TLB_CONTROL_FLUSH_ASID, but it was removed: > > > https://lore.kernel.org/kvm/f7c2d5f5-3560-8666-90be-3605220cb93c@xxxxxxxxxx/ > > > > > > I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". > > > Committer's email is dead, so I'm writing here. > > > > > > > I have to say that I know about this for long time, because some time ago I used to play with VMware player in a > > VM on AMD on my spare time, on weekends > > (just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc). > > > > I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID > > which states that KVM doesn't support setting this bit. > > I am pretty sure that bit 1 is supposed to be ignored on hardware > without FlushByASID, but I'll have to see if I can dig up an old APM > to verify that. I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN. While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical. I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause. > > Supporting FLUSHBYASID would fix this, and make nesting faster too, > > but it is far from a trivial job. > > > > I hope that I will find time to do this soon. > > > > Best regards, > > Maxim Levitsky > > > >
