On 2014/12/11 19:48, Paolo Bonzini wrote:
On 11/12/2014 09:07, zhanghailiang wrote:
User space (i.e. QEMU) should be able to check whether KVM
supports apic-v. User space will use this to decide whether enable
emulated MSR-based APIC (i.e. hyperv-vapic).
Did you see an improvement in some workload?
Yes, compared with using hyperv-vapic, using apic-v reduces lots of VM exit.
The bellow is the qemu command and perf data:
qemu/x86_64-softmmu/qemu-system-x86_64 -name win2008_st_r2_64_2U -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -m 4096 -realtime mlock=off -smp 6,maxcpus=12,sockets=1,cores=12,threads=1 -uuid e7fca4b5-10a2-4fa5-a71b-b5e9eaa41d4d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_st_r2_64_2U.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/data/win2008_st_r2_64_2U.DataCenter.40G.base.virtio.02,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=!
52:54:00
:48:70:80,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,vgamem_mb=8,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
(note: -cpu is Haswell, so x2apic is enable in default)
(1) Using Hyperv-vapic
UVP:/data # perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
1215390 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
0 kvm:kvm_hv_hypercall [100.00%]
8397 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
693288 kvm:kvm_apic [100.00%]
1228026 kvm:kvm_exit [100.00%]
0 kvm:kvm_inj_virq [100.00%]
3 kvm:kvm_inj_exception [100.00%]
39 kvm:kvm_page_fault [100.00%]
536450 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
1340 kvm:kvm_pic_set_irq [100.00%]
164441 kvm:kvm_apic_ipi [100.00%]
376239 kvm:kvm_apic_accept_irq [100.00%]
377100 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
528 kvm:kvm_emulate_insn [100.00%]
444 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
630 kvm:kvm_userspace_exit [100.00%]
1394 kvm:kvm_set_irq [100.00%]
1396 kvm:kvm_ioapic_set_irq [100.00%]
212003 kvm:kvm_msi_set_irq [100.00%]
4092 kvm:kvm_ack_irq [100.00%]
901 kvm:kvm_mmio [100.00%]
710971 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.000959015 seconds time elapsed
(2) Using apic-v
perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
888948 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
6 kvm:kvm_hv_hypercall [100.00%]
12510 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
324926 kvm:kvm_apic [100.00%]
898841 kvm:kvm_exit [100.00%]
0 kvm:kvm_inj_virq [100.00%]
0 kvm:kvm_inj_exception [100.00%]
259 kvm:kvm_page_fault [100.00%]
60 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
2046 kvm:kvm_pic_set_irq [100.00%]
164641 kvm:kvm_apic_ipi [100.00%]
371867 kvm:kvm_apic_accept_irq [100.00%]
665 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
1418 kvm:kvm_emulate_insn [100.00%]
465 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
368303 kvm:kvm_ple_window [100.00%]
3948 kvm:kvm_userspace_exit [100.00%]
2088 kvm:kvm_set_irq [100.00%]
2090 kvm:kvm_ioapic_set_irq [100.00%]
0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%]
208975 kvm:kvm_msi_set_irq [100.00%]
688 kvm:kvm_ack_irq [100.00%]
949 kvm:kvm_mmio [100.00%]
743624 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.000829758 seconds time elapsed
I think it's a Windows bug---it should prefer x2apic to hv-vapic if both
are available.
No, i don't think it is a windows bug, it has nothing to do with x2apic, but apic-v
(need hardware support, i.e. Haswell cpu). When we don't passthough host cpu model to Guest os,
it has no idea about whether it supports apic-v in host, so the choose should be done by qemu.
Actually, qemu has a option 'hv_vapic' for -cpu, we can choose not to configure it if we know
there is apic-v support in host. But IMHO, we'd better to do it automatically.
PS:
This is the perf data when we configure x2apic but without configure hv-vapic and disable apic-v in kvm.
It is the worst result, compared with the above two cases.
UVP:/data # perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
1616665 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
0 kvm:kvm_hv_hypercall [100.00%]
8766 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
896485 kvm:kvm_apic [100.00%]
1634073 kvm:kvm_exit [100.00%]
363649 kvm:kvm_inj_virq [100.00%]
0 kvm:kvm_inj_exception [100.00%]
38 kvm:kvm_page_fault [100.00%]
0 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
1444 kvm:kvm_pic_set_irq [100.00%]
181544 kvm:kvm_apic_ipi [100.00%]
368012 kvm:kvm_apic_accept_irq [100.00%]
368745 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
556213 kvm:kvm_emulate_insn [100.00%]
463 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
378556 kvm:kvm_ple_window [100.00%]
1125 kvm:kvm_userspace_exit [100.00%]
1490 kvm:kvm_set_irq [100.00%]
1492 kvm:kvm_ioapic_set_irq [100.00%]
0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%]
187316 kvm:kvm_msi_set_irq [100.00%]
4137 kvm:kvm_ack_irq [100.00%]
568249 kvm:kvm_mmio [100.00%]
763264 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.001000787 seconds time elapsed
Thanks,
zhanghailiang
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
