On Wed, Jul 26, 2023, Yahya Sohail wrote: > On 7/26/23 12:17, Sean Christopherson wrote: > > > If so, what fields in the kvm_run struct should I check that could cause such > > > an issue? > > > > Heh, all of them. I'm only somewhat joking. Root causing "invalid control field" > > errors on bare metal is painfully difficult, bordering on impossible if you don't > > have something to give you a hint as to what might be going wrong. > > I suppose that's what I was expecting, but was hoping it could be narrowed > down a bit. Could the values of the CPU control registers or other special > registers set with KVM_SET_SREGS also cause this error (with > hardware_entry_failure_reason = 7)? I'd expect this not to be possible > because I don't think the CPU registers are part of the VMCS, but I'm not > very familiar with VMX. > > I do know that the emulator I'm copying state from likely doesn't consider > all bits in the control fields, so it's possible that they're in an invalid > state. When I ran the model before with the value for cr0 copied out of the > emulator I also got KVM_EXIT_FAIL_ENTRY, but with a different value for > hardware_entry_failure_reason = 0x80000021. I fixed this by changing the > value of cr0 to be (hopefully) valid. What were the before and after values of CR0? > > If you can, try running a nested setup, i.e. run a normal Linux guest as your L1 > > VM (L0 is bare metal), and then run your problematic x86 emulator VM within that > > L1 guest (that's your L2). Then, in L0 (your bare metal host), enable the > > kvm_nested_vmenter_failed tracepoint. > > > > The kvm_nested_vmenter_failed tracepoint logs all VM-Enter failures that _KVM_ > > detects when L1 attempts a nested VM-Enter from L1 to L2. If you're at all lucky, > > KVM in L0 (acting a the CPU from L1's perspective) will detect the invalid state > > and explicitly log which consistency check failed. > > I did this and had an interesting result. Instead of exiting with > KVM_EXIT_FAIL_ENTRY, it exited with KVM_EXIT_UNkNOWN, and > hardware_exit_reason = 0. Hrm, what kernel version are you running as L1? KVM on x86 doesn't explicitly return KVM_EXIT_UNKNOWN except in a few paths that I highly doubt you are hitting. > I also didn't get anything logged from the kvm_nested_vmenter_failed trace > point. When I checked the value of rip after KVM_RUN, it was the same as the > starting value, so it probably failed without executing any instructions. > > I then tried setting the kvm_nested_vmexit tracepoint to see if I could get > any more information about the vmexit. When the vmexit occurred, I got a > line in the log that looked like this: > > CPU 3/KVM-9310 [013] .... 6076.453278: kvm_nested_vmexit: vcpu 3 reason > EPT_VIOLATION rip 0x103c00 info1 0x0000000000000781 info2 0x000000008000030d > intr_info 0x00000000 error_code 0x00000000 So getting an EPT violation VM-Exit means the VM-Entry was successful. Are you running different kernel versions for L0 versus L1? If so, it's possible that there's a bug (or bug fix) in one kernel and not the other. > It appears this occurred due to an EPT_VIOLATION. I have some questions: > I believe an EPT_VIOLATION is caused by trying to access physical memory > that is not mapped. Is that correct? Yep. The "info1 0x0000000000000781" from above is the EXIT_QUALIFICATION field, which for EPT violations is equivalent to a #PF error code. 0x781 means a read access faulted and the mapping was !present, e.g. as opposed to the mapping being !readable (EPT supports execute-only mappings). The other interesting bit is "info 0x000000008000030d", which is the vectoring info. That value means that the EPT violation occurred while the CPU was trying to deliver a #GP in the guest. In and of itself, that isn't fatal, but it does suggest that something might be going wrong in the emulator. > Also, could this be the same error that causes the KVM_EXIT_FAIL_ENTRY when > running the VM as L1, or must that be a separate issue? Maybe? EPT violations themselves are not errors (ignore the "violation" part, it's not as scary as it sounds). But if the exit to userspace is related to the EPT violation, I would expect uuuKVM_EXIT_MMIO, not KVM_EXIT_UNKNOWN. > I know that the paging code of the emulator the state is from is a little > suspect (in fact, one of my reasons to get this VM working in KVM is to help > debug the emulator), and it is possible that the page tables of the VM are > not setup properly and are mapping linear addresses to unexpected physical > addresses and causing an EPT_VIOLATION. I'll have to look into that further. Turn on the kvm_page_fault tracepoint, that will give the gpa on which the fault occurs.
