Hi folks, I recently got a Intel Xeon E5-2609 v2 machine with APIC-v support. I did some performance tests under Linux kernel 3.11 and have some doubts about the new APICv feature. I'm appreciated for any comments and please correct me if I'm wrong. My understanding of APIC-v is that it mainly consists of 1) Virtual interrupt delivery (the same as posted interrupt), which avoids KVM to inject vAPIC interrupts manually. In other word, it "post" an interrupt to the guest without sending IPI, which causes external interrupt exit. 2) EOI virtualization. So guest acknowledging the interrupt incurs no EOI exit. (however, sometimes it exits) 3) Virtualized the APIC-registers so read/write won't trap into the hypervisor. However, "some" APIC-write still trigger VM exit, but it becomes trap-like instead of fault-like. (I don't know which APIC-write causes exit and which does not) === Experiment A Result === 1. virtio network with vhost, iperf TCP experiments, enable/disable APIC-v [With APIC-v] Total number of EXIT rate: 4351.1 exits second ------ VM EXIT Breakdown ------ reason exit/sec Avg(us) IO_INSTRUCTION 1428 81.471931 EXCEPTION_NMI 69 7.906276 EXTERNAL_INTERRUPT 1866 7.317781 MSR_WRITE 970 1.504932 [Without APIC-v] Total number of EXIT rate: 83510.1 exits per second ------ VM EXIT Breakdown ------ reason exit/sec Avg(us) IO_INSTRUCTION 18428 81.471931 EXTERNAL_INTERRUPT 31166 7.317781 MSR_WRITE 30970 1.504932 VM exit rate reduces from 83k/sec to 4.3k/sec because - the 31166 EXTERNAL_INTERRUPT mainly comes from vhost sending IPI, while APIC-v's posted interrupt avoids it. - the 30970 MSR_WRITE comes from EOI, while APIC-v's EOI virtualization avoids it - however, APIC-v still has 1866 EXTERNAL_INTERRUPT and 970 MSR_WRITE exits, I found it's due to timer. I confirm with the next experiments. === Experiment B Result === I run cyclictest in VM and measure the VM exit behavior. The cyclictest is configure to generate 1k timer per second. For with or without APIC-v, I got similar results as below total number of EXIT 156919 rate: 5225.18 exits per second ------ VM EXIT Breakdown ------ reason exit/sec Avg(us) IO_INSTRUCTION 18 47.412613 EXTERNAL_INTERRUPT 3085 3.022567 MSR_WRITE 2095 1.330987 I found APIC-v does not improve on timer interrupt delivery because posted interrupt seems not work on LAPIC timer? If not, why? So the 3085 EXTERNAL_INTERRUPT is due to timer expiration interrupt and 2095 MSR_WRITE is due to EOI and program the TMICT (part of APIC's register). Does this contradict with the APIC-v's assumption saying APIC-write is direct without VM exit? Thank you and any comments are welcome Regards, William Tu -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
