Separate followup message with the symptoms I managed to gather about
the failure on KVM.
On 08/06/15 15:42, Laszlo Ersek wrote:
> Hi Star,
>
> On 08/06/15 11:44, Zeng, Star wrote:
>> Hi Laszlo,
>>
>> Could you help take a try with the attached patch on your VM before I
>> send it for formal review?
>
> I got your patch via your first (public) message as well -- not the
> one reflected by the list software, but on the direct route. So, I can
> test it and even comment on it a little bit below (in Thunderbird
> plaintext attachments are displayed inline, and if you select the full
> contents of the "body" window before hitting Reply All, then the
> entire selection will be quoted, including the inline displayed text
> attachments).
>
> ** So, test results:
>
> - With X64 DXE, your patch solves the issue for me.
>
> - With Ia32 DXE, your patch also soves the issue.
>
> - With this patch applied, if I use Ia32 DXE *plus* my SMM series,
> then:
>
> - Using TCG in QEMU (= software emulation), the patch solves the
> issue.
>
> - Using KVM with QEMU (= hardware virtualization), I continue seeing
> reboots, but they hit in a different place. Somewhere inside or
> after the SMBASE relocation. I believe this is a KVM bug. I will
> send a separate email about this.
** Bottom-up investigation:
I captured a KVM trace. The interesting part is not very long (fewer
than 300 lines), I just searched for a triple fault, and then searched
backwards for an SMI:
> kvm_apic_ipi: dst 0 vec 0 (SMI|physical|assert|edge|dst)
> kvm_apic_accept_irq: apicid 0 vec 0 (SMI|edge)
> kvm_enter_smm: vcpu 0: entering SMM, smbase 0x30000
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x8000 info 184 0
> kvm_page_fault: address 38000 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffa78ca info 184 0
> kvm_page_fault: address 7ffd78ca error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason CR_ACCESS rip 0x7ffa78d0 info 3 0
> kvm_cr: cr_write 3 = 0x7fe64000
> kvm_entry: vcpu 0
> kvm_exit: reason CR_ACCESS rip 0x7ffa78e0 info 4 0
> kvm_cr: cr_write 4 = 0x660
> kvm_entry: vcpu 0
> kvm_exit: reason CR_ACCESS rip 0x7ffa78ec info 0 0
> kvm_cr: cr_write 0 = 0x80000033
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffa78ef info 81 0
> kvm_page_fault: address 7fe66ff8 error_code 81
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffa78ef info 81 0
> kvm_page_fault: address 7fe63eb8 error_code 81
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffa78ef info 181 0
> kvm_page_fault: address 7ffdf710 error_code 181
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd7906 info 182 0
> kvm_page_fault: address 7ffc1ff8 error_code 182
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd2f20 info 184 0
> kvm_page_fault: address 7ffd2f20 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd8ec4 info 184 0
> kvm_page_fault: address 7ffd8ec4 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd8ec4 info 181 0
> kvm_page_fault: address 7ffdb8c8 error_code 181
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffda2a8 info 184 0
> kvm_page_fault: address 7ffda2a8 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd9fe4 info 184 0
> kvm_page_fault: address 7ffd9fe4 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason CPUID rip 0x7ffd8923 info 0 0
> kvm_cpuid: func 1 rax 6e8 rbx 800 rcx 80200001 rdx f89fbff
> kvm_entry: vcpu 0
> kvm_exit: reason MSR_READ rip 0x7ffd886f info 0 0
> kvm_msr: msr_read 1b = 0xfee00900
> kvm_entry: vcpu 0
> kvm_exit: reason CPUID rip 0x7ffd8923 info 0 0
> kvm_cpuid: func 1 rax 6e8 rbx 800 rcx 80200001 rdx f89fbff
> kvm_entry: vcpu 0
> kvm_exit: reason MSR_READ rip 0x7ffd886f info 0 0
> kvm_msr: msr_read 1b = 0xfee00900
> kvm_entry: vcpu 0
> kvm_exit: reason CPUID rip 0x7ffd8923 info 0 0
> kvm_cpuid: func 0 rax a rbx 756e6547 rcx 6c65746e rdx 49656e69
> kvm_entry: vcpu 0
> kvm_exit: reason CPUID rip 0x7ffd8923 info 0 0
> kvm_cpuid: func 1 rax 6e8 rbx 800 rcx 80200001 rdx f89fbff
> kvm_entry: vcpu 0
> kvm_exit: reason CPUID rip 0x7ffd8923 info 0 0
> kvm_cpuid: func 1 rax 6e8 rbx 800 rcx 80200001 rdx f89fbff
> kvm_entry: vcpu 0
> kvm_exit: reason MSR_READ rip 0x7ffd886f info 0 0
> kvm_msr: msr_read 1b = 0xfee00900
> kvm_entry: vcpu 0
> kvm_exit: reason CPUID rip 0x7ffd8923 info 0 0
> kvm_cpuid: func 1 rax 6e8 rbx 800 rcx 80200001 rdx f89fbff
> kvm_entry: vcpu 0
> kvm_exit: reason MSR_READ rip 0x7ffd886f info 0 0
> kvm_msr: msr_read 1b = 0xfee00900
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffdb669 info 81 0
> kvm_page_fault: address 7fe68fb8 error_code 81
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffdb669 info 181 0
> kvm_page_fault: address fee00020 error_code 181
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd2f6c info 181 0
> kvm_page_fault: address 7ffdea60 error_code 181
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd43b4 info 184 0
> kvm_page_fault: address 7ffd43b4 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd4421 info 81 0
> kvm_page_fault: address 7fe65000 error_code 81
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd4421 info 182 0
> kvm_page_fault: address 3fef8 error_code 182
> kvm_entry: vcpu 0
> kvm_exit: reason MSR_READ rip 0x7ffd886f info 0 0
> kvm_msr: msr_read fe = 0x508
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffdb2b8 info 181 0
> kvm_page_fault: address 7ffdc000 error_code 181
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x49
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x73
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x53
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6d
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x72
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x72
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x53
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x75
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x70
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x70
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6f
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x72
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x74
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x65
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x64
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x3a
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x20
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x53
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x4d
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x52
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x52
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x20
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x73
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x75
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x70
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x70
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6f
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x72
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x74
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x20
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x77
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x61
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x73
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x20
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6e
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6f
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x74
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x20
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x66
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6f
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x75
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x6e
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0x64
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0xd
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason IO_INSTRUCTION rip 0x7ffd8de3 info 4020000 0
> kvm_pio: pio_write at 0x402 size 1 count 1 val 0xa
> kvm_userspace_exit: reason KVM_EXIT_IO (2)
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd537d info 184 0
> kvm_page_fault: address 7ffd537d error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EPT_VIOLATION rip 0x7ffd3015 info 184 0
> kvm_page_fault: address 7ffd3015 error_code 184
> kvm_entry: vcpu 0
> kvm_exit: reason EXCEPTION_NMI rip 0x7ffd790b info 0 80000306
> kvm_emulate_insn: 0:7ffd790b: 0f aa
> kvm_cpuid: func 80000001 rax 6e8 rbx 0 rcx 0 rdx 100000
> kvm_enter_smm: vcpu 0: leaving SMM, smbase 0x7ffc0000
> kvm_entry: vcpu 0
> kvm_exit: reason TRIPLE_FAULT rip 0x7ffdb6b2 info 0 0
> kvm_userspace_exit: reason KVM_EXIT_SHUTDOWN (8)
The emulated instruction at 7ffd790b, [0f aa], is RSM.
I correlated that address with the load address of the PiSmmCpuDxeSmm
driver, logged by the firmware:
> Loading SMM driver at 0x0007FFD2000 EntryPoint=0x0007FFD2253 PiSmmCpuDxeSmm.efi
The difference (0x590B) matches the disassembly of the PiSmmCpuDxeSmm
module:
> 00005902 <gSmmInitStack>:
> 5902: 00 00 add %al,(%eax)
> 5904: 00 00 add %al,(%eax)
> 5906: e8 15 b6 ff ff call f20 <SmmInitHandler>
> 590b: 0f aa rsm
This disassembly snippet comes form
"OvmfPkg/QuarkPort/PiSmmCpuDxeSmm/Ia32/SmmInit.S":
> ASM_PFX(gSmmInitStack): .space 4
> call ASM_PFX(SmmInitHandler)
> rsm
The SmmInitHandler() function runs and returns, the RSM is also
executed, and then the instruction at 0x7ffdb6b2 seems to cause a triple
fault.
Again from the disassembly (0x7ffdb6b2 - 0x0007FFD2000 = 0x96B2):
> 00009677 <MmioWrite32>:
> 9677: 55 push %ebp
> 9678: 89 e5 mov %esp,%ebp
> 967a: 56 push %esi
> 967b: 8b 75 08 mov 0x8(%ebp),%esi
> 967e: 53 push %ebx
> 967f: 8b 5d 0c mov 0xc(%ebp),%ebx
> 9682: e8 3d d8 ff ff call 6ec4 <DebugAssertEnabled>
> 9687: 84 c0 test %al,%al
> 9689: 74 20 je 96ab <MmioWrite32+0x34>
> 968b: f7 c6 03 00 00 00 test $0x3,%esi
> 9691: 74 18 je 96ab <MmioWrite32+0x34>
> 9693: 50 push %eax
> 9694: 68 a7 c7 00 00 push $0xc7a7
> 9699: 68 f7 00 00 00 push $0xf7
> 969e: 68 44 c7 00 00 push $0xc744
> 96a3: e8 ae d7 ff ff call 6e56 <DebugAssert>
> 96a8: 83 c4 10 add $0x10,%esp
> 96ab: e8 9b d1 ff ff call 684b <MemoryFence>
> 96b0: 89 1e mov %ebx,(%esi)
> 96b2: e8 94 d1 ff ff call 684b <MemoryFence> <======== boom
> 96b7: 8d 65 f8 lea -0x8(%ebp),%esp
> 96ba: 89 d8 mov %ebx,%eax
> 96bc: 5b pop %ebx
> 96bd: 5e pop %esi
> 96be: 5d pop %ebp
> 96bf: c3 ret
The corresponding source code is
("MdePkg/Library/BaseIoLibIntrinsic/IoLib.c"):
> UINT32
> EFIAPI
> MmioWrite32 (
> IN UINTN Address,
> IN UINT32 Value
> )
> {
> ASSERT ((Address & 3) == 0);
>
> MemoryFence ();
> *(volatile UINT32*)Address = Value;
> MemoryFence ();
>
> return Value;
> }
** Top-down investigation:
I added a bunch of debug statements to the PiSmmCpuDxeSmm source code,
and this is the call stack that blows up:
SmmRelocateBases() [OvmfPkg/QuarkPort/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c]
//
// Relocate BSP's SM base
//
SendSmiIpi() [UefiCpuPkg/Library/BaseXApicLib/BaseXApicLib.c]
SendIpi() [UefiCpuPkg/Library/BaseXApicLib/BaseXApicLib.c]
WriteLocalApicReg() [UefiCpuPkg/Library/BaseXApicLib/BaseXApicLib.c]
//
// LocalApicBaseAddress=0xFEE00000 MmioOffset=0x310 Value=0x0
// (XAPIC_ICR_HIGH_OFFSET = 0x310)
//
MmioWrite32() [MdePkg/Library/BaseIoLibIntrinsic/IoLib.c]
WriteLocalApicReg() [UefiCpuPkg/Library/BaseXApicLib/BaseXApicLib.c]
//
// LocalApicBaseAddress=0xFEE00000 MmioOffset=0x300 Value=0x4200
// (XAPIC_ICR_LOW_OFFSET = 0x300)
//
MmioWrite32() [MdePkg/Library/BaseIoLibIntrinsic/IoLib.c]
This is when the "gSmmInitStack" code that I quoted from
"OvmfPkg/QuarkPort/PiSmmCpuDxeSmm/Ia32/SmmInit.S" runs (in response to
the write to XAPIC_ICR_LOW_OFFSET).
"gSmmInitStack" calls back to C (invoking the SmmInitHandler()
function). After SmmInitHandler() returns, the RSM instruction from the
assembly source (at the end of "gSmmInitStack") is executed.
However, the *very first* instruction executed after the RSM -- ie. the
second MemoryFence() call in MmioWrite32() -- causes a triple fault.
Thanks
Laszlo
On 08/06/15 15:42, Laszlo Ersek wrote:
>
> ** Comments on the patch:
>
>> CpuDxe.diff
>>
>>
>> 9238355e710a898a148efa072c67693fac631e42
>> UefiCpuPkg/CpuDxe/ApStartup.c | 140 +++++++++++++++++++++++++++++++++++++++++-
>> 1 file changed, 138 insertions(+), 2 deletions(-)
>>
>> diff --git a/UefiCpuPkg/CpuDxe/ApStartup.c b/UefiCpuPkg/CpuDxe/ApStartup.c
>> index 7613b47..59cf8e6 100644
>> --- a/UefiCpuPkg/CpuDxe/ApStartup.c
>> +++ b/UefiCpuPkg/CpuDxe/ApStartup.c
>> @@ -19,6 +19,47 @@
>> #pragma pack(1)
>>
>> typedef struct {
>> + UINT8 MoveIa32EferMsrToEcx[5];
>> + UINT8 ReadIa32EferMsr[2];
>> + UINT8 SetExecuteDisableBitEnableBit[4];
>> + UINT8 WriteIa32EferMsr[2];
>> +
>> +#if defined (MDE_CPU_IA32)
>> + UINT8 MovEaxCr3;
>> + UINT32 Cr3Value;
>> + UINT8 MovCr3Eax[3];
>> +
>> + UINT8 MoveCr4ToEax[3];
>> + UINT8 SetCr4Bit5[4];
>> + UINT8 MoveEaxToCr4[3];
>> +
>> + UINT8 MoveCr0ToEax[3];
>> + UINT8 SetCr0PagingBit[4];
>> + UINT8 MoveEaxToCr0[3];
>> +#endif
>> +} ENABLE_EXECUTE_DISABLE_CODE;
>> +
>> +ENABLE_EXECUTE_DISABLE_CODE mEnableExecuteDisbleCodeTemplate = {
>
> (1) This should be STATIC. (Although, I realize mStartupCodeTemplate is
> not STATIC either, so feel free to ignore this.)
>
>> + { 0xB9, 0x80, 0x00, 0x00, 0xC0 }, // mov ecx, 0xc0000080
>> + { 0x0F, 0x32 }, // rdmsr
>> + { 0x0F, 0xBA, 0xE8, 0x0B }, // bts eax, 11
>> + { 0x0F, 0x30 }, // wrmsr
>> +
>> +#if defined (MDE_CPU_IA32)
>> + 0xB8, 0x00000000, // mov eax, cr3 value
>> + { 0x0F, 0x22, 0xd8 }, // mov cr3, eax
>> +
>> + { 0x0F, 0x20, 0xE0 }, // mov eax, cr4
>> + { 0x0F, 0xBA, 0xE8, 0x05 }, // bts eax, 5
>> + { 0x0F, 0x22, 0xE0 }, // mov cr4, eax
>> +
>> + { 0x0F, 0x20, 0xC0 }, // mov eax, cr0
>> + { 0x0F, 0xBA, 0xE8, 0x1F }, // bts eax, 31
>> + { 0x0F, 0x22, 0xC0 }, // mov cr0, eax
>> +#endif
>> +};
>> +
>> +typedef struct {
>> UINT8 JmpToCli[2];
>>
>> UINT16 GdtLimit;
>> @@ -56,6 +97,12 @@ typedef struct {
>> //
>> // Transition to X64
>> //
>> +
>> + //
>> + // Code placeholder to enable Execute Disable Bit
>> + //
>> + ENABLE_EXECUTE_DISABLE_CODE EnableExecuteDisble;
>> +
>> UINT8 MovEaxCr3;
>> UINT32 Cr3Value;
>> UINT8 MovCr3Eax[3];
>> @@ -75,7 +122,12 @@ typedef struct {
>> //UINT8 DeadLoop[2];
>>
>> UINT8 FarJmp32LongMode; UINT32 LongJmpOffset; UINT16 LongJmpSelector;
>> -#endif // defined (MDE_CPU_X64)
>> +#else
>> + //
>> + // Code placeholder to enable IA32 PAE Execute Disable
>> + //
>> + ENABLE_EXECUTE_DISABLE_CODE EnableExecuteDisble;
>> +#endif
>
> (2) I think that you can eliminate the #else branch (more precisely, you
> don't need to introduce it at all). Simply add the
> ENABLE_EXECUTE_DISABLE_CODE field above the
>
> #if defined (MDE_CPU_X64)
>
> line, so that it becomes common to X64 and Ia32. I agree that the
> comment / purpose of the field differs between X64 and Ia32, but that
> could be spelled out in one, common comment block.
>
> (3) Also, please search the whole patch for occurrences of "Disble",
> that's a typo.
>
>>
>> #if defined (MDE_CPU_X64)
>> UINT8 MovEaxOrRaxCpuDxeEntry[2]; UINTN CpuDxeEntryValue;
>> @@ -207,6 +259,17 @@ STARTUP_CODE mStartupCodeTemplate = {
>> { 0x8e, 0xd0 }, // mov ss, ax
>>
>> #if defined (MDE_CPU_X64)
>> + //
>> + // Code placeholder to enable Execute Disable Bit
>> + // Default is all NOP¡ªNo Operation
>
> (4) Your patch should be pure ASCII. There seems to be a UTF-8 sequence
> between "NOP" and "No Operation", 0xA1 0xAA.
>
>> + //
>> + {
>> + { 0x90, 0x90, 0x90, 0x90, 0x90 }, // mov ecx, 0xc0000080
>> + { 0x90, 0x90 }, // rdmsr
>> + { 0x90, 0x90, 0x90, 0x90 }, // bts eax, 11
>> + { 0x90, 0x90 }, // wrmsr
>> + },
>> +
>
> (5) I don't think it is useful to preserve the assembly instructions
> here, in comments; the binary contents is just NOPs. The comments to the
> right don't match.
>
>> 0xB8, 0x00000000, // mov eax, cr3 value
>> { 0x0F, 0x22, 0xd8 }, // mov cr3, eax
>>
>> @@ -226,7 +289,29 @@ STARTUP_CODE mStartupCodeTemplate = {
>> 0xEA, // FarJmp32LongMode
>> OFFSET_OF(STARTUP_CODE, MovEaxOrRaxCpuDxeEntry),
>> LINEAR_CODE64_SEL,
>> -#endif // defined (MDE_CPU_X64)
>> +#else
>> + //
>> + // Code placeholder to enable IA32 PAE Execute Disable
>> + // Default is all NOP¡ªNo Operation
>> + //
>> + {
>> + { 0x90, 0x90, 0x90, 0x90, 0x90 }, // mov ecx, 0xc0000080
>> + { 0x90, 0x90 }, // rdmsr
>> + { 0x90, 0x90, 0x90, 0x90 }, // bts eax, 11
>> + { 0x90, 0x90 }, // wrmsr
>> +
>> + 0x90, 0x90909090, // mov eax, cr3 value
>> + { 0x90, 0x90, 0x90 }, // mov cr3, eax
>> +
>> + { 0x90, 0x90, 0x90 }, // mov eax, cr4
>> + { 0x90, 0x90, 0x90, 0x90 }, // bts eax, 5
>> + { 0x90, 0x90, 0x90 }, // mov cr4, eax
>> +
>> + { 0x90, 0x90, 0x90 }, // mov eax, cr0
>> + { 0x90, 0x90, 0x90, 0x90 }, // bts eax, 31
>> + { 0x90, 0x90, 0x90 }, // mov cr0, eax
>> + },
>> +#endif
>
> (6) Ditto.
>
>>
>> //0xeb, 0xfe, // jmp $
>> #if defined (MDE_CPU_X64)
>> @@ -241,6 +326,48 @@ STARTUP_CODE mStartupCodeTemplate = {
>> volatile STARTUP_CODE *StartupCode = NULL;
>>
>> /**
>> + The function will check if BSP Execute Disable is enabled.
>> + DxeIpl may have enabled Execute Disable for BSP,
>> + APs need to get the status and sync up the settings.
>> +
>> + @retval TRUE BSP Execute Disable is enabled.
>> + @retval FALSE BSP Execute Disable is not enabled.
>> +
>> +**/
>> +BOOLEAN
>> +IsBspExecuteDisableEnabled (
>> + VOID
>> + )
>> +{
>> + UINT32 RegEax;
>> + UINT32 RegEdx;
>> + UINT64 MsrRegisters;
>> + BOOLEAN Enabled;
>> +
>> + Enabled = FALSE;
>> + AsmCpuid (0x80000000, &RegEax, NULL, NULL, NULL);
>> + if (RegEax >= 0x80000001) {
>> + AsmCpuid (0x80000001, NULL, NULL, NULL, &RegEdx);
>> + //
>> + // Cpuid 0x80000001
>> + // Bit 20: Execute Disable Bit available.
>> + //
>> + if ((RegEdx & BIT20) != 0) {
>> + MsrRegisters = AsmReadMsr64 (0xC0000080);
>> + //
>> + // Msr 0xC0000080
>> + // Bit 11: Execute Disable Bit enable.
>> + //
>> + if ((MsrRegisters & BIT11) != 0) {
>> + Enabled = TRUE;
>> + }
>> + }
>> + }
>> +
>> + return Enabled;
>> +}
>> +
>> +/**
>> Prepares Startup Code for APs.
>> This function prepares Startup Code for APs.
>>
>> @@ -280,9 +407,18 @@ PrepareAPStartupCode (
>>
>> StartupCode->FlatJmpOffset += (UINT32) StartAddress;
>>
>> + if (IsBspExecuteDisableEnabled ()) {
>> + CopyMem (
>> + (VOID*) &StartupCode->EnableExecuteDisble,
>
> (7) No need for the (VOID*) cast.
>
>> + &mEnableExecuteDisbleCodeTemplate,
>> + sizeof (ENABLE_EXECUTE_DISABLE_CODE)
>> + );
>> + }
>> #if defined (MDE_CPU_X64)
>> StartupCode->Cr3Value = (UINT32) AsmReadCr3 ();
>> StartupCode->LongJmpOffset += (UINT32) StartAddress;
>> +#else
>> + StartupCode->EnableExecuteDisble.Cr3Value = (UINT32) AsmReadCr3 ();
>
> (8) I think the (UINT32) cast can be dropped in this branch, but it's
> not important.
>
>> #endif
>>
>> return EFI_SUCCESS;
>>
>
> So, I think the patch is good, and that we should look into the KVM
> problem separately. I'll follow up soon.
>
> Thanks
> Laszlo
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
