Liang Li <liliang324@xxxxxxxxx> writes:
>> >> > Analyze events for all VMs, all VCPUs:
>> >> > VM-EXIT Samples Samples% Time% Min Time Max
>> >> > Time Avg time
>> >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
>> >> > 65.42us 2.34us ( +- 0.11% )
>> >> > MSR_WRITE 238932 30.33% 23.07% 0.48us
>> >> > 41.05us 1.56us ( +- 0.14% )
>> >> >
>> >> > Total Samples:787803, Total events handled time:1611193.84us.
>> >> >
>> >> > I tried turning off hyper-v for the same workload and repeat the test,
>> >> > the overall virtualization overhead reduced by about of 50%:
>> >> >
>> >> > -------
>> >> >
>> >> > Analyze events for all VMs, all VCPUs:
>> >> >
>> >> > VM-EXIT Samples Samples% Time% Min Time Max
>> >> > Time Avg time
>> >> > APIC_WRITE 255152 74.43% 50.72% 0.49us
>> >> > 50.01us 1.42us ( +- 0.14% )
>> >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
>> >> > 686.05us 7.27us ( +- 0.43% )
>> >> > DR_ACCESS 35003 10.21% 4.64% 0.32us
>> >> > 40.03us 0.95us ( +- 0.32% )
>> >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
>> >> > 57.38us 2.25us ( +- 1.42% )
>> >> >
>> >> > Total Samples:342788, Total events handled time:715695.62us.
>> >> >
>> >> > For this scenario, hyper-v works really bad. stimer works better
>> >> > than hpet, but on the other hand, it relies on SynIC which has
>> >> > negative effects for IPI intensive workloads.
>> >> > Do you have any plans for improvement?
>> >> >
>> >>
>> >> Hey,
>> >>
>> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
>> >> automatically disables APICv and this can explain the overhead and the
>> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
>> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
>> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
>> >> inhibit APICv if the recommendation was ignored. This is implemented in
>> >> the following KVM patch series:
>> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@xxxxxxxxxx/
>> >>
>> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it
>> >> can be tested with 'hv-passthrough'.
>> >>
>> >> It would be great if you could give it a spin!
>> >>
>> >> --
>> >> Vitaly
>> >
>> > It's great to know that you already have a solution for this. :)
>> >
>> > By the way, is there any requirement for the version of windows or
>> > windows updates for the new feature to work?
>>
>> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
>> expect WS2008 to ignore it completely (and thus SynIC will always be
>> disabling APICv for it).
>>
>
> Hi Vitaly,
> I tried your patchset and found it's not helpful to reduce the
> virtualization overhead.
> here are some perfdata with the same workload
>
> ===============================
> Analyze events for all VMs, all VCPUs:
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> MSR_WRITE 924045 89.96% 81.10% 0.42us
> 68.42us 1.26us ( +- 0.07% )
> DR_ACCESS 44669 4.35% 2.36% 0.32us
> 50.74us 0.76us ( +- 0.32% )
> EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
> 70.75us 3.10us ( +- 0.54% )
> VMCALL 17819 1.73% 5.21% 0.75us
> 15.64us 4.20us ( +- 0.33%
>
> Total Samples:1027227, Total events handled time:1436343.94us.
> ===============================
>
> The result shows the overhead increased. enable the apicv can help to
> reduce the vm-exit
> caused by interrupt injection, but on the other side, there are a lot
> of vm-exit caused by APIC_EOI.
>
> When turning off the hyper-v and using the kvm apicv, there is no such
> overhead.
I think I know what's happening. We've asked Windows to use synthetic
MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be
accelerated in hardware.
Could you please try the following hack (KVM):
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index c8f2592ccc99..66ee85a83e9a 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
vcpu->arch.ia32_misc_enable_msr &
MSR_IA32_MISC_ENABLE_MWAIT);
}
+
+ /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! */
+ best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0);
+ if (best) {
+ best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
+ best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED;
+ }
}
EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime);
> It seems turning on hyper V related features is not always the best
> choice for a windows guest.
Generally it is, we'll just need to make QEMU smarter when setting
'recommendation' bits.
--
Vitaly
