On 15/10/19 18:49, Andrea Arcangeli wrote: > On Tue, Oct 15, 2019 at 10:28:39AM +0200, Paolo Bonzini wrote: >> If you're including EXIT_REASON_EPT_MISCONFIG (MMIO access) then you >> should include EXIT_REASON_IO_INSTRUCTION too. Depending on the devices >> that are in the guest, the doorbell register might be MMIO or PIO. > > The fact outb/inb devices exists isn't the question here. The question > you should clarify is: which of the PIO devices is performance > critical as much as MMIO with virtio/vhost? virtio 0.9 uses PIO. > I mean even on real hardware those devices aren't performance critical. On virtual machines they're actually faster than MMIO because they don't need to go through page table walks. >> So, the difference between my suggested list (which I admit is just >> based on conjecture, not benchmarking) is that you add >> EXIT_REASON_PAUSE_INSTRUCTION, EXIT_REASON_PENDING_INTERRUPT, >> EXIT_REASON_EXTERNAL_INTERRUPT, EXIT_REASON_HLT, EXIT_REASON_MSR_READ, >> EXIT_REASON_CPUID. >> >> Which of these make a difference for the hrtimer testcase? It's of >> course totally fine to use benchmarks to prove that my intuition was >> bad---but you must also use them to show why your intuition is right. :) > > The hrtimer flood hits on this: > > MSR_WRITE 338793 56.54% 5.51% 0.33us 34.44us 0.44us ( +- 0.20% ) > PENDING_INTERRUPT 168431 28.11% 2.52% 0.36us 32.06us 0.40us ( +- 0.28% ) > PREEMPTION_TIMER 91723 15.31% 1.32% 0.34us 30.51us 0.39us ( +- 0.41% ) > EXTERNAL_INTERRUPT 234 0.04% 0.00% 0.25us 5.53us 0.43us ( +- 5.67% ) > HLT 65 0.01% 90.64% 0.49us 319933.79us 37562.71us ( +- 21.68% ) > MSR_READ 6 0.00% 0.00% 0.67us 1.96us 1.06us ( +- 17.97% ) > EPT_MISCONFIG 6 0.00% 0.01% 3.09us 105.50us 26.76us ( +- 62.10% ) > > PENDING_INTERRUPT is the big missing thing in your list. It probably > accounts for the bulk of slowdown from your list. Makes sense. > However I could imagine other loads with higher external > interrupt/hlt/rdmsr than the hrtimer one so I didn't drop those. External interrupts should only tick at 1 Hz on nohz_full kernels, and even at 1000 Hz (if physical CPUs are not isolated) it should not really matter. We can include it since it has such a short handler so the cost of retpolines is in % much more than other exits. HLT is certainly a slow path, the guest only invokes if things such as NAPI interrupt mitigation have failed. As long as the guest stays in halted state for a microsecond or so, the cost of retpoline will all but disappear. RDMSR again shouldn't be there, guests sometimes read the PMTimer (which is an I/O port) or TSC but for example do not really ever read the APIC TMCCT. > I'm pretty sure HLT/EXTERNAL_INTERRUPT/PENDING_INTERRUPT should be > included. > I also wonder if VMCALL should be added, certain loads hit on fairly > frequent VMCALL, but none of the one I benchmarked. I agree for external interrupt and pending interrupt, and VMCALL is fine too. In addition I'd add I/O instructions which are useful for some guests and also for benchmarking (e.g. vmexit.flat has both IN and OUT tests). Paolo
