Hi Roman:
Thanks for your review.
> From: Roman Kagan <rkagan@xxxxxxxxxxxxx>
> Sent: Wednesday, October 23, 2019 4:14 AM
> To: Paolo Bonzini <pbonzini@xxxxxxxxxx>
> Cc: lantianyu1986@xxxxxxxxx; rth@xxxxxxxxxxx; ehabkost@xxxxxxxxxx;
> mtosatti@xxxxxxxxxx; vkuznets <vkuznets@xxxxxxxxxx>; Tianyu Lan
> <Tianyu.Lan@xxxxxxxxxxxxx>; qemu-devel@xxxxxxxxxx;
> kvm@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH V3 2/2] target/i386/kvm: Add Hyper-V direct tlb flush
> support
>
> On Tue, Oct 22, 2019 at 07:04:11PM +0200, Paolo Bonzini wrote:
> > On 16/10/19 15:07, lantianyu1986@xxxxxxxxx wrote:
>
> Somehow this patch never got through to me so I'll reply here.
>
> > > From: Tianyu Lan <Tianyu.Lan@xxxxxxxxxxxxx>
> > >
> > > Hyper-V direct tlb flush targets KVM on Hyper-V guest.
> > > Enable direct TLB flush for its guests meaning that TLB flush
> > > hypercalls are handled by Level 0 hypervisor (Hyper-V) bypassing KVM
> > > in Level 1. Due to the different ABI for hypercall parameters
> > > between Hyper-V and KVM, KVM capabilities should be hidden when
> > > enable Hyper-V direct tlb flush otherwise KVM hypercalls may be
> > > intercepted by Hyper-V. Add new parameter "hv-direct-tlbflush".
> > > Check expose_kvm and Hyper-V tlb flush capability status before
> > > enabling the feature.
> > >
> > > Signed-off-by: Tianyu Lan <Tianyu.Lan@xxxxxxxxxxxxx>
> > > ---
> > > Change sicne v2:
> > > - Update new feature description and name.
> > > - Change failure print log.
> > >
> > > Change since v1:
> > > - Add direct tlb flush's Hyper-V property and use
> > > hv_cpuid_check_and_set() to check the dependency of tlbflush
> > > feature.
> > > - Make new feature work with Hyper-V passthrough mode.
> > > ---
> > > docs/hyperv.txt | 10 ++++++++++
> > > target/i386/cpu.c | 2 ++
> > > target/i386/cpu.h | 1 +
> > > target/i386/kvm.c | 24 ++++++++++++++++++++++++
> > > 4 files changed, 37 insertions(+)
> > >
> > > diff --git a/docs/hyperv.txt b/docs/hyperv.txt index
> > > 8fdf25c829..140a5c7e44 100644
> > > --- a/docs/hyperv.txt
> > > +++ b/docs/hyperv.txt
> > > @@ -184,6 +184,16 @@ enabled.
> > >
> > > Requires: hv-vpindex, hv-synic, hv-time, hv-stimer
> > >
> > > +3.18. hv-direct-tlbflush
> > > +=======================
> > > +Enable direct TLB flush for KVM when it is running as a nested
> > > +hypervisor on top Hyper-V. When enabled, TLB flush hypercalls from
> > > +L2 guests are being passed through to L0 (Hyper-V) for handling.
> > > +Due to ABI differences between Hyper-V and KVM hypercalls, L2
> > > +guests will not be able to issue KVM hypercalls (as those could be
> > > +mishanled by L0 Hyper-V), this requires KVM hypervisor signature to be
> hidden.
> > > +
> > > +Requires: hv-tlbflush, -kvm
> > >
> > > 4. Development features
> > > ========================
> > > diff --git a/target/i386/cpu.c b/target/i386/cpu.c index
> > > 44f1bbdcac..7bc7fee512 100644
> > > --- a/target/i386/cpu.c
> > > +++ b/target/i386/cpu.c
> > > @@ -6156,6 +6156,8 @@ static Property x86_cpu_properties[] = {
> > > HYPERV_FEAT_IPI, 0),
> > > DEFINE_PROP_BIT64("hv-stimer-direct", X86CPU, hyperv_features,
> > > HYPERV_FEAT_STIMER_DIRECT, 0),
> > > + DEFINE_PROP_BIT64("hv-direct-tlbflush", X86CPU, hyperv_features,
> > > + HYPERV_FEAT_DIRECT_TLBFLUSH, 0),
> > > DEFINE_PROP_BOOL("hv-passthrough", X86CPU, hyperv_passthrough,
> > > false),
> > >
> > > DEFINE_PROP_BOOL("check", X86CPU, check_cpuid, true), diff
> > > --git a/target/i386/cpu.h b/target/i386/cpu.h index
> > > eaa5395aa5..3cb105f7d6 100644
> > > --- a/target/i386/cpu.h
> > > +++ b/target/i386/cpu.h
> > > @@ -907,6 +907,7 @@ typedef uint64_t
> FeatureWordArray[FEATURE_WORDS];
> > > #define HYPERV_FEAT_EVMCS 12
> > > #define HYPERV_FEAT_IPI 13
> > > #define HYPERV_FEAT_STIMER_DIRECT 14
> > > +#define HYPERV_FEAT_DIRECT_TLBFLUSH 15
> > >
> > > #ifndef HYPERV_SPINLOCK_NEVER_RETRY
> > > #define HYPERV_SPINLOCK_NEVER_RETRY 0xFFFFFFFF
> > > diff --git a/target/i386/kvm.c b/target/i386/kvm.c index
> > > 11b9c854b5..043b66ab22 100644
> > > --- a/target/i386/kvm.c
> > > +++ b/target/i386/kvm.c
> > > @@ -900,6 +900,10 @@ static struct {
> > > },
> > > .dependencies = BIT(HYPERV_FEAT_STIMER)
> > > },
> > > + [HYPERV_FEAT_DIRECT_TLBFLUSH] = {
> > > + .desc = "direct paravirtualized TLB flush (hv-direct-tlbflush)",
> > > + .dependencies = BIT(HYPERV_FEAT_TLBFLUSH)
> > > + },
> > > };
> > >
> > > static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
> > > @@ -1224,6 +1228,7 @@ static int hyperv_handle_properties(CPUState
> *cs,
> > > r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_EVMCS);
> > > r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_IPI);
> > > r |= hv_cpuid_check_and_set(cs, cpuid,
> > > HYPERV_FEAT_STIMER_DIRECT);
> > > + r |= hv_cpuid_check_and_set(cs, cpuid,
> > > + HYPERV_FEAT_DIRECT_TLBFLUSH);
>
> AFAICS this will turn HYPERV_FEAT_DIRECT_TLBFLUSH on if
> hyperv_passthrough is on, so ...
Yes.
>
> > >
> > > /* Additional dependencies not covered by kvm_hyperv_properties[]
> */
> > > if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) && @@ -1243,6
> > > +1248,25 @@ static int hyperv_handle_properties(CPUState *cs,
> > > goto free;
> > > }
> > >
> > > + if (hyperv_feat_enabled(cpu, HYPERV_FEAT_DIRECT_TLBFLUSH) ||
> > > + cpu->hyperv_passthrough) {
>
> ... the test for ->hyperv_passthrough is redundant, and ...
>
> > > + if (!cpu->expose_kvm) {
> > > + r = kvm_vcpu_enable_cap(cs,
> KVM_CAP_HYPERV_DIRECT_TLBFLUSH, 0, 0);
> > > + if (hyperv_feat_enabled(cpu,
> > > + HYPERV_FEAT_DIRECT_TLBFLUSH) && r) {
>
> ... , more importantly, this will abort QEMU if
> HYPERV_FEAT_DIRECT_TLBFLUSH wasn't requested explicitly, but was
> activated by ->hyperv_passthrough, and setting the capability failed. I think
> the meaning of hyperv_passthrough is "enable all hyperv features supported
> by the KVM", so in this case it looks more correct to just clear the feature bit
> and go ahead.
>
> > > + fprintf(stderr,
> > > + "Hyper-V %s is not supported by kernel\n",
> > > +
> kvm_hyperv_properties[HYPERV_FEAT_DIRECT_TLBFLUSH].desc);
> > > + return -ENOSYS;
> > > + }
> > > + } else if (!cpu->hyperv_passthrough) {
> > > + fprintf(stderr,
> > > + "Hyper-V %s requires KVM hypervisor signature "
> > > + "to be hidden (-kvm).\n",
> > > +
> kvm_hyperv_properties[HYPERV_FEAT_DIRECT_TLBFLUSH].desc);
> > > + return -ENOSYS;
> > > + }
>
> You reach here if ->expose_kvm && ->hyperv_passthrough, and no
> capability is activated, and you go ahead with the feature bit set.
> This doesn't look right either.
>
> So in general it should probably look like
>
> if (hyperv_feat_enabled(HYPERV_FEAT_DIRECT_TLBFLUSH)) {
> if (kvm_vcpu_enable_cap(KVM_CAP_HYPERV_DIRECT_TLBFLUSH)) {
> if (!cpu->hyperv_passthrough) {
> ... report feature unsupported by kernel ...
> return -ENOSYS;
> }
> cpu->hyperv_features &= ~BIT(HYPERV_FEAT_DIRECT_TLBFLUSH);
> } else if (cpu->expose_kvm) {
> ... report conflict ...
> return -ENOSYS;
> }
> }
>
> [Yes, hyperv_passthrough hurts, but you've been warned ;)]
Yes, you are right, I ignore HYPERV_FEAT_DIRECT_TLBFLUSH bit was set in the hv_cpuid_check_and_set(().
Will update in the next version. Thanks.
>
> Thanks,
> Roman.
