2017-11-13 18:46 GMT+08:00 Peter Zijlstra <peterz@xxxxxxxxxxxxx>:
> On Mon, Nov 13, 2017 at 04:26:57PM +0800, Wanpeng Li wrote:
>> 2017-11-13 16:04 GMT+08:00 Peter Zijlstra <peterz@xxxxxxxxxxxxx>:
>
>> > So if at this point a vCPU gets preempted we'll still spin-wait for it,
>> > which is sub-optimal.
>> >
>> > I think we can come up with something to get around that 'problem' if
>> > indeed it is a problem. But we can easily do that as follow up patches.
>> > Just let me know if you think its worth spending more time on.
>>
>> You can post your idea, it is always smart. :) Then we can evaluate
>> the complexity and gains.
>
> I'm not sure I have a fully baked idea just yet, but the general idea
> would be something like:
>
> - switch (back) to a dedicated TLB invalidate IPI
>
> - introduce KVM_VCPU_IPI_PENDING
>
> - change flush_tlb_others() into something like:
>
> for_each_cpu(cpu, flushmask) {
> src = &per_cpu(steal_time, cpu);
> state = READ_ONCE(src->preempted);
> do {
> if (state & KVM_VCPU_PREEMPTED) {
> if (try_cmpxchg(&src->preempted, &state,
> state | KVM_VCPU_SHOULD_FLUSH)) {
> __cpumask_clear_cpu(cpu, flushmask);
> break;
> }
> }
> } while (!try_cmpxchg(&src->preempted, &state,
> state | KVM_VCPU_IPI_PENDING));
> }
>
> apic->send_IPI_mask(flushmask, CALL_TLB_VECTOR);
>
> for_each_cpu(cpu, flushmask) {
> src = &per_cpu(steal_time, cpu);
> smp_cond_load_acquire(&src->preempted, !(VAL & KVM_VCPU_IPI_PENDING);
> }
>
>
> - have the TLB invalidate handler do something like:
>
> state = READ_ONCE(src->preempted);
> if (!(state & KVM_VCPU_IPI_PENDING))
> return;
>
> local_flush_tlb();
>
> do {
> } while (!try_cmpxchg(&src->preempted, &state,
> state & ~KVM_VCPU_IPI_PENDING));
There are a lot of cases handled by flush_tlb_func_remote() ->
flush_tlb_function_common(), so I'm afraid to have hole.
Regards,
Wanpeng Li
>
> - then at VMEXIT time do something like:
>
> state = READ_ONCE(src->preempted);
> do {
> if (!(state & KVM_VCPU_IPI_PENDING))
> break;
> } while (!try_cmpxchg(&src->preempted, state,
> (state & ~KVM_VCPU_IPI_PENDING) |
> KVM_VCPU_SHOULD_FLUSH));
>
> and clear any possible pending TLB_VECTOR in the guest state to avoid
> raising that IPI spuriously on enter again.
>
>
> This way the preemption will clear the IPI_PENDING and the
> flush_others() wait loop will terminate.
