On 17.07.17 11:26, Yang Zhang wrote:
On 2017/7/14 17:37, Alexander Graf wrote:
On 13.07.17 13:49, Yang Zhang wrote:
On 2017/7/4 22:13, Radim Krčmář wrote:
2017-07-03 17:28+0800, Yang Zhang:
The background is that we(Alibaba Cloud) do get more and more
complaints
from our customers in both KVM and Xen compare to bare-mental.After
investigations, the root cause is known to us: big cost in message
passing
workload(David show it in KVM forum 2015)
A typical message workload like below:
vcpu 0 vcpu 1
1. send ipi 2. doing hlt
3. go into idle 4. receive ipi and wake up from hlt
5. write APIC time twice 6. write APIC time twice to
to stop sched timer reprogram sched timer
One write is enough to disable/re-enable the APIC timer -- why does
Linux use two?
One is to remove the timer and another one is to reprogram the timer.
Normally, only one write to remove the timer.But in some cases, it
will reprogram it.
7. doing hlt 8. handle task and send ipi to
vcpu 0
9. same to 4. 10. same to 3
One transaction will introduce about 12 vmexits(2 hlt and 10 msr
write). The
cost of such vmexits will degrades performance severely.
Yeah, sounds like too much ... I understood that there are
IPI from 1 to 2
4 * APIC timer
IPI from 2 to 1
which adds to 6 MSR writes -- what are the other 4?
In the worst case, each timer will touch APIC timer twice.So it will
add additional 4 msr writse. But this is not always true.
Linux kernel
already provide idle=poll to mitigate the trend. But it only
eliminates the
IPI and hlt vmexit. It has nothing to do with start/stop sched
timer. A
compromise would be to turn off NOHZ kernel, but it is not the default
config for new distributions. Same for halt-poll in KVM, it only
solve the
cost from schedule in/out in host and can not help such workload much.
The purpose of this patch we want to improve current idle=poll
mechanism to
Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
down the sibling hyperthread. MWAIT solves the IPI problem, but
doesn't
get rid of the timer one.
Yes, i can try it. But MWAIT will not yield CPU, it only helps the
sibling hyperthread as you mentioned.
If you implement proper MWAIT emulation that conditionally gets en- or
disabled depending on the same halt poll dynamics that we already have
for in-host HLT handling, it will also yield the CPU.
It is hard to do . If we not intercept MWAIT instruction, there is no
chance to wake up the CPU unless an interrupt arrived or a store to the
address armed by MONITOR which is the same with idle=polling.
Yes, but you can reconfigure the VMCS/VMCB to trap on MWAIT or not trap
on it. That's something that idle=polling does not give you at all - a
guest vcpu will always use 100% CPU.
The only really tricky part is how to limit the effect of MONITOR on
nested page table maintenance. But if we just set the MONITOR cache size
to 4k, well behaved guests should ideally always give us the one same
page for wakeup - which we can then leave marked as trapping.
As for the timer - are you sure the problem is really the overhead of
the timer configuration, not the latency that it takes to actually fire
the guest timer?
No, the main cost is introduced by vmexit, includes IPIs, Timer program,
HLT. David detailed it in KVM forum, you can search "Message Passing
Workloads in KVM" in google and the first link give the whole analysis
of the problem.
During time critical message passing you want to keep both vCPUs inside
the guest, yes. That again is something that guest exposed MWAIT would
buy you.
The problem is that overcommitting CPU is very expensive with anything
that does not set the guests idle at all. And not everyone can afford to
throw more CPUs at problems :).
Alex
