>> I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time",
>> this problem still happened, about 200,000 vmexits per-second,
>> bringing very bad experience, just like being stuck.
>
>Please upload a full trace somewhere, or at least the "perf report" output.
>
# perf kvm stat report --event=vmexit
Warning:
Processed 4912765 events and lost 81 chunks!
Check IO/CPU overload!
Analyze events for all VCPUs:
VM-EXIT Samples Samples% Time% Avg time
PAUSE_INSTRUCTION 1248620 60.74% 23.33% 0.83us ( +- 0.02% )
IO_INSTRUCTION 799619 38.89% 74.49% 4.13us ( +- 6.93% )
EXTERNAL_INTERRUPT 2629 0.13% 0.47% 7.93us ( +- 1.27% )
VMCALL 1785 0.09% 0.04% 0.98us ( +- 0.30% )
APIC_ACCESS 1495 0.07% 0.11% 3.38us ( +- 1.65% )
EXCEPTION_NMI 1188 0.06% 0.03% 1.26us ( +- 1.66% )
TPR_BELOW_THRESHOLD 273 0.01% 0.01% 1.57us ( +- 1.32% )
PENDING_INTERRUPT 226 0.01% 0.01% 1.38us ( +- 2.59% )
HLT 6 0.00% 1.51% 11153.81us ( +- 15.24% )
EPT_MISCONFIG 2 0.00% 0.00% 15.46us ( +- 23.93% )
Total Samples:2055843, Total events handled time:4438794.59us.
# perf kvm stat report --event=ioport
Warning:
Processed 4912765 events and lost 81 chunks!
Check IO/CPU overload!
Analyze events for all VCPUs:
IO Port Access Samples Samples% Time% Avg time
0x64:PIN 799032 99.93% 99.84% 3.42us ( +- 8.35% )
0x3fd:PIN 300 0.04% 0.08% 6.88us ( +- 3.45% )
0x60:PIN 274 0.03% 0.09% 8.53us ( +- 3.07% )
0xb000:POUT 2 0.00% 0.00% 7.96us ( +- 3.16% )
0xb000:PIN 2 0.00% 0.00% 8.16us ( +- 2.31% )
0xb008:PIN 2 0.00% 0.00% 1.53us ( +- 13.66% )
0xafe0:PIN 2 0.00% 0.00% 3.62us ( +- 0.84% )
0xafe1:PIN 2 0.00% 0.00% 2.76us ( +- 2.99% )
Total Samples:799616, Total events handled time:2739038.53us.
# trace-cmd report
version = 6
CPU 4 is empty
CPU 5 is empty
CPU 6 is empty
CPU 7 is empty
CPU 9 is empty
CPU 11 is empty
CPU 12 is empty
CPU 13 is empty
cpus=16
kvm-22783 [003] 721.255618: kvm_entry: vcpu 0
kvm-22784 [014] 721.255619: kvm_entry: vcpu 1
kvm-22783 [003] 721.255622: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22784 [014] 721.255623: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255623: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255623: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255623: kvm_entry: vcpu 0
kvm-22784 [014] 721.255625: kvm_entry: vcpu 1
kvm-22783 [003] 721.255626: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255626: kvm_entry: vcpu 0
kvm-22784 [014] 721.255627: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255628: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255628: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255628: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255629: kvm_entry: vcpu 0
kvm-22784 [014] 721.255630: kvm_entry: vcpu 1
kvm-22783 [003] 721.255631: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22784 [014] 721.255632: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22783 [003] 721.255632: kvm_entry: vcpu 0
kvm-22784 [014] 721.255632: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255632: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255634: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22784 [014] 721.255634: kvm_entry: vcpu 1
kvm-22783 [003] 721.255635: kvm_entry: vcpu 0
kvm-22784 [014] 721.255636: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255636: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255636: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255637: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255637: kvm_entry: vcpu 0
kvm-22784 [014] 721.255638: kvm_entry: vcpu 1
kvm-22783 [003] 721.255640: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255640: kvm_entry: vcpu 0
kvm-22784 [014] 721.255640: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255640: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255641: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255642: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22784 [014] 721.255643: kvm_entry: vcpu 1
kvm-22783 [003] 721.255643: kvm_entry: vcpu 0
kvm-22784 [014] 721.255645: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255645: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255645: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255645: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255646: kvm_entry: vcpu 0
kvm-22784 [014] 721.255647: kvm_entry: vcpu 1
kvm-22784 [014] 721.255649: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255649: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255649: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255649: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255650: kvm_entry: vcpu 0
kvm-22784 [014] 721.255651: kvm_entry: vcpu 1
kvm-22783 [003] 721.255652: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22783 [003] 721.255653: kvm_entry: vcpu 0
kvm-22784 [014] 721.255653: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255653: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22784 [014] 721.255654: kvm_userspace_exit: reason KVM_EXIT_IO (2)
kvm-22783 [003] 721.255655: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
kvm-22784 [014] 721.255655: kvm_entry: vcpu 1
kvm-22783 [003] 721.255655: kvm_entry: vcpu 0
kvm-22784 [014] 721.255657: kvm_exit: reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
kvm-22784 [014] 721.255658: kvm_pio: pio_read at 0x64 size 1 count 1
kvm-22783 [003] 721.255658: kvm_exit: reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
...
And, if I remove the commit of 0bc830b0, the problem disappeared.
Thanks,
Zhang Haoyu
>Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
