To test support for nested virtualization I was running a VM (L2) on a
debug build of ESX (L1) on VMware Workstation/KVM (L0). This
consistently resulted in an ASSERT in L1 firing as the interrupt
shadow bit in the VMCB was set on an #NPF exit that occurred when
vectoring through the IDT to deliver an interrupt to L2.
Some details from our exit recorder are below. Basically what
happened is that L1 resumed L2 after handling an I/O exit and
attempted to inject an internal interrupt with vector 0x68. This
resulted in a #NPF exit when vectoring through the IDT to deliver the
interrupt to the guest with the interrupt shadow bit set which our
code is not expecting. There is no reason for the interrupt shadow
bit to be set and neither L1 or L0 were setting it.
This turns out to be due to a quirk where on AMD 'vmrun' after an
'sti' will cause the interrupt shadow bit to leak into the guest state
in the VMCB. Jim Mattson discovered this back when he was with VMware
and checked in a fix to make sure that our 'vmrun' is not immediately
after an 'sti':
sti /* Enable interrupts during guest execution */
mov svmPhysCurrentVMCB(%rip), %rax
vmrun /* Must not immediately follow STI. See PR 150935 */
PR 150935 describes exactly the same problem I am seeing with KVM.
For KVM the 'vmrun' is immediately after a 'sti' though:
/* Enter guest mode */
sti
1: vmrun %rax
I confirmed that moving the 'sti' after the mov instruction in the
VMware code causes the same exact ASSERT to fire. I discussed this
with Jim and Sean and they suggested sending an e-mail to this list.
Jim also mentioned that this was introduced by [1] a few years back.
It would be hard to argue that this isn't an AMD bug but it seems best
to workaround it in SW. It would be great if someone could fix this
but if folks are too busy I can ask Zach to include it in the patches
he is working on.
Thanks.
Doug
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/kvm/svm/vmenter.S?id=fb0c4a4fee5a35b4e531b57e42231868d1fedb18
(gdb) p svmDbgExitRecs[5]->rip
$9 = 0x5b5f723
(gdb) p svmDbgExitRecs[5]->intrState
$10 = 0x0
(gdb) p svmDbgExitRecs[5]->exitCode
$11 = 0x7b
(gdb) p svmDbgExitRecs[5]->exitIntInfo
$12 = 0x0
(gdb) p svmDbgResumeRecs[5]->rip
$13 = 0x5b5f724
(gdb) p svmDbgResumeRecs[5]->intrState
$14 = 0x0
(gdb) p svmDbgResumeRecs[5]->exitCode
$15 = 0x7b
(gdb) p svmDbgResumeRecs[5]->exitIntInfo
$16 = 0x0
(gdb) p svmDbgResumeRecs[5]->eventInj
$22 = 0x80000068
(gdb) p svmDbgExitRecs[6]->rip
$17 = 0x5b5f724
(gdb) p svmDbgExitRecs[6]->intrState
$18 = 0x1 <<< should be 0
(gdb) p svmDbgExitRecs[6]->exitCode
$19 = 0x400
(gdb) p svmDbgExitRecs[6]->exitIntInfo
$20 = 0x80000068
0x5b5f71f: mov %edx,%eax
0x5b5f721: mov %ecx,%edx
0x5b5f723: out %al,(%dx)
=> 0x5b5f724: retq
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
