On 25/11/2014 12:20, Zhang Haoyu wrote: >>> I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", >>> this problem still happened, about 200,000 vmexits per-second, >>> bringing very bad experience, just like being stuck. >> >> Please upload a full trace somewhere, or at least the "perf report" output. >> > # perf kvm stat report --event=vmexit > Warning: > Processed 4912765 events and lost 81 chunks! > Check IO/CPU overload! > > Analyze events for all VCPUs: > > VM-EXIT Samples Samples% Time% Avg time > > PAUSE_INSTRUCTION 1248620 60.74% 23.33% 0.83us ( +- 0.02% ) > IO_INSTRUCTION 799619 38.89% 74.49% 4.13us ( +- 6.93% ) > EXTERNAL_INTERRUPT 2629 0.13% 0.47% 7.93us ( +- 1.27% ) > VMCALL 1785 0.09% 0.04% 0.98us ( +- 0.30% ) > APIC_ACCESS 1495 0.07% 0.11% 3.38us ( +- 1.65% ) > EXCEPTION_NMI 1188 0.06% 0.03% 1.26us ( +- 1.66% ) > TPR_BELOW_THRESHOLD 273 0.01% 0.01% 1.57us ( +- 1.32% ) > PENDING_INTERRUPT 226 0.01% 0.01% 1.38us ( +- 2.59% ) > HLT 6 0.00% 1.51% 11153.81us ( +- 15.24% ) > EPT_MISCONFIG 2 0.00% 0.00% 15.46us ( +- 23.93% ) > > Total Samples:2055843, Total events handled time:4438794.59us. > > # perf kvm stat report --event=ioport > Warning: > Processed 4912765 events and lost 81 chunks! > Check IO/CPU overload! > > Analyze events for all VCPUs: > > IO Port Access Samples Samples% Time% Avg time > > 0x64:PIN 799032 99.93% 99.84% 3.42us ( +- 8.35% ) > 0x3fd:PIN 300 0.04% 0.08% 6.88us ( +- 3.45% ) > 0x60:PIN 274 0.03% 0.09% 8.53us ( +- 3.07% ) > 0xb000:POUT 2 0.00% 0.00% 7.96us ( +- 3.16% ) > 0xb000:PIN 2 0.00% 0.00% 8.16us ( +- 2.31% ) > 0xb008:PIN 2 0.00% 0.00% 1.53us ( +- 13.66% ) > 0xafe0:PIN 2 0.00% 0.00% 3.62us ( +- 0.84% ) > 0xafe1:PIN 2 0.00% 0.00% 2.76us ( +- 2.99% ) > > Total Samples:799616, Total events handled time:2739038.53us. > > # trace-cmd report > version = 6 > CPU 4 is empty > CPU 5 is empty > CPU 6 is empty > CPU 7 is empty > CPU 9 is empty > CPU 11 is empty > CPU 12 is empty > CPU 13 is empty > cpus=16 > kvm-22783 [003] 721.255618: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255619: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255622: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22784 [014] 721.255623: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255623: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255623: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255623: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255625: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255626: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255626: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255627: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255628: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255628: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255628: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255629: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255630: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255631: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22784 [014] 721.255632: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22783 [003] 721.255632: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255632: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255632: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255634: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22784 [014] 721.255634: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255635: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255636: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255636: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255636: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255637: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255637: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255638: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255640: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255640: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255640: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255640: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255641: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255642: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22784 [014] 721.255643: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255643: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255645: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255645: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255645: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255645: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255646: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255647: kvm_entry: vcpu 1 > kvm-22784 [014] 721.255649: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255649: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255649: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255649: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255650: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255651: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255652: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22783 [003] 721.255653: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255653: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255653: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22784 [014] 721.255654: kvm_userspace_exit: reason KVM_EXIT_IO (2) > kvm-22783 [003] 721.255655: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > kvm-22784 [014] 721.255655: kvm_entry: vcpu 1 > kvm-22783 [003] 721.255655: kvm_entry: vcpu 0 > kvm-22784 [014] 721.255657: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0 > kvm-22784 [014] 721.255658: kvm_pio: pio_read at 0x64 size 1 count 1 > kvm-22783 [003] 721.255658: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0 > ... > > And, if I remove the commit of 0bc830b0, the problem disappeared. Please send the full trace file. If you compress it, it should be small. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
