Ryan,
On Fri, May 29, 2009 at 11:43:26AM -0500, Ryan Harper wrote:
> Testing latest qemu-kvm.git and kvm-kmod.git, ept enabled and backing
> guests with large pages trips a BUG in the mmu code. If I disable ept,
> but still use large pages, migration succeeds. Reproduce with:
>
> hugetlbfs setup:
> % mkdir -p /hugetlbfs && mount -t hugetlbfs hugetlbfs /hugetlbfs
> % echo 10000 > /proc/sys/vm/nr_hugepages
>
> qemu commands:
>
> guest a:
> sudo x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 2048 -mempath /hugetlbfs -net nic -net tap -vnc :12 -monitor stdio -hda /scratch/images/rharper/rhel4u8-32-ide.raw
>
> guest b:
> sudo x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 2048 -mempath /hugetlbfs -net nic -net tap -vnc :13 -monitor stdio -hda /scratch/images/rharper/rhel4u8-32-ide.raw -incoming tcp:0:4444
>
> Once the guest a is up, issued migrate command:
> (qemu) migrate -d tcp:localhost:444
>
> rmap_remove: ffff880a08e00098 c0336e65c0336e5b 0->BUG
^^^^^^^^^^^^^^^^
This value looks very strange (bits 5:3 contain invalid value, for one).
Don't have access to HW at the very moment, so it would be great if you
had time to do a change equivalent to this and reproduce:
diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index 809cce0..ceb70b0 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -1759,7 +1764,7 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *shadow_pte,
child = page_header(pte & PT64_BASE_ADDR_MASK);
mmu_page_remove_parent_pte(child, shadow_pte);
} else if (pfn != spte_to_pfn(*shadow_pte)) {
- pgprintk("hfn old %lx new %lx\n",
+ printk(KERN_ERR "hfn old %lx new %lx\n",
spte_to_pfn(*shadow_pte), pfn);
rmap_remove(vcpu->kvm, shadow_pte);
} else
Avi, any hints?
> ------------[ cut here ]------------
> kernel BUG at /home/rharper/work/git/kvm-kmod/x86/mmu.c:615!
> invalid opcode: 0000 [1] SMP
> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
> CPU 6
> Modules linked in: kvm_intel(N) kvm(N) tun nfs lockd nfs_acl sunrpc ipv6 bridge stp cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq microcode fuse loop sr_mod cdrom dm_mod sg rtc_cmos thermal cdc_ether i2c_i801 rtc_core usbnet usb_storage shpchp i2c_core rtc_lib processor bnx2 pcspkr button pci_hotplug mii mptctl joydev usbhid hid ff_memless uhci_hcd ehci_hcd usbcore sd_mod crc_t10dif edd fan thermal_sys hwmon ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas scsi_mod [last unloaded: kvm]
> Supported: No
> Pid: 17635, comm: qemu-system-x86 Tainted: G 2.6.27.19-5-default #1
> RIP: 0010:[<ffffffffa012d8dc>] [<ffffffffa012d8dc>] rmap_remove+0xc9/0x19e [kvm]
> RSP: 0018:ffff880c7a1cbba8 EFLAGS: 00010296
> RAX: 0000000000000039 RBX: 00000036e65c0336 RCX: ffff880c7b405e60
> RDX: ffffffff806e0d08 RSI: 0000000000000092 RDI: ffffffff806e0d00
> RBP: ffff880a08e00098 R08: ffffffff806e0cf0 R09: 0000000100000000
> R10: 0000000000000046 R11: 000000000000000a R12: ffff880c7b066a20
> R13: ffff8806778e0000 R14: 0000000000000000 R15: 0000000000000007
> FS: 00007f298b4ad950(0000) GS:ffff880c7cd83f40(0000) knlGS:0000000000000000
> CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
> CR2: 0000000000879ba0 CR3: 0000000679da8000 CR4: 00000000000026e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process qemu-system-x86 (pid: 17635, threadinfo ffff880c7a1ca000, task ffff8809ebce4880)
> Stack: ffff88069822b888 0000000000000000 ffff8803f1000040 ffff880a08e00098
> 0000000000000001 ffffffffa012f5d3 ffff880c7a1cbc58 ffffffff8023661a
> 0000000000000000 8000000a08e000e7 ffffffff80228db7 00007f298e413fff
> Call Trace:
> [<ffffffffa012f5d3>] mmu_set_spte+0x98/0x302 [kvm]
> [<ffffffffa012ffa3>] __direct_map+0xee/0x1b8 [kvm]
> [<ffffffffa013014b>] tdp_page_fault+0xde/0x114 [kvm]
> [<ffffffffa0130f16>] kvm_mmu_page_fault+0x19/0x81 [kvm]
> [<ffffffffa012a64b>] kvm_arch_vcpu_ioctl_run+0x89b/0xaf2 [kvm]
> [<ffffffffa0123540>] kvm_vcpu_ioctl+0xf1/0x46b [kvm]
> [<ffffffff802bd249>] vfs_ioctl+0x21/0x6c
> [<ffffffff802bd4b6>] do_vfs_ioctl+0x222/0x231
> [<ffffffff802bd516>] sys_ioctl+0x51/0x73
> [<ffffffff8020bfbb>] system_call_fastpath+0x16/0x1b
> [<00007f298c3c3b77>] 0x7f298c3c3b77
>
>
> Code: 80 00 00 00 48 8b 34 c1 e8 0c ff ff ff 49 89 c1 48 8b 00 48 85 c0 75 17 48 8b 55 00 48 89 ee 48 c7 c7 2f db 13 a0 e8 6d cc 36 e0 <0f> 0b eb fe a8 01 75 2a 48 39 c5 74 19 48 8b 55 00 48 89 ee 48
> RIP [<ffffffffa012d8dc>] rmap_remove+0xc9/0x19e [kvm]
> RSP <ffff880c7a1cbba8>
> ---[ end trace 91e1d7963caa34a7 ]---
>
> hugepage info:
> HugePages_Total: 10000
> HugePages_Free: 7944
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> module info:
> filename: /lib/modules/2.6.27.19-5-default/extra/kvm-intel.ko
> license: GPL
> author: Qumranet
> version: kvm-devel
> srcversion: 9F14ECEFD8109654DFA20D2
> depends: kvm
> vermagic: 2.6.27.19-5-default SMP mod_unload modversions
> parm: bypass_guest_pf:bool
> parm: vpid:bool
> parm: flexpriority:bool
> parm: ept:bool
> parm: emulate_invalid_guest_state:bool
>
> filename: /lib/modules/2.6.27.19-5-default/extra/kvm.ko
> license: GPL
> author: Qumranet
> version: kvm-devel
> srcversion: 157F8CB48FC31BC2F44847B
> depends:
> vermagic: 2.6.27.19-5-default SMP mod_unload modversions
> parm: oos_shadow:bool
>
>
>
> --
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> ryanh@xxxxxxxxxx
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
