On Wed, 15 Jun 2022 22:47:57 +0300
Alexander Mikhalitsyn <alexander@xxxxxxxxxxxxx> wrote:
> Dear Sean,
>
> Thanks a lot for your answer!
>
> On Wed, Jun 15, 2022 at 6:00 PM Sean Christopherson <seanjc@xxxxxxxxxx> wrote:
> >
> > On Wed, Jun 15, 2022, Alexander Mikhalitsyn wrote:
> > > Dear friends,
> > >
> > > I'm sorry for disturbing you but I've getting stuck with debugging KVM
> > > problem and looking for an advice. I'm working mostly on kernel
> > > containers/CRIU and am newbie with KVM so, I believe that I'm missing
> > > something very simple.
> > >
> > > My case:
> > > - AMD EPYC 7443P 24-Core Processor (Milan family processor)
> > > - OpenVZ kernel (based on RHEL7 3.10.0-1160.53.1) on the Host Node (HN)
> > > - Qemu/KVM VM (8 vCPU assigned) with many different kernels from 3.10.0-1160 RHEL7 to mainline 5.18
> > >
> > > Reproducer (run inside VM):
> > > echo 0 > /sys/devices/system/cpu/cpu3/online
> > > echo 1 > /sys/devices/system/cpu/cpu3/online <- got reset here
> > >
> > > *Not* reproducible on:
> > > - any Intel which we tried
> > > - AMD EPYC 7261 (Rome family)
> >
> > Hmm, given that Milan is problematic but Rome isn't, that implies the bug is related
> > to a feature that's new in Milan. PCID is the one that comes to mind, and IIRC there
> > were issues with PCID (or INVCPID?) in various kernels when running on Milan.
> >
> > Can you try hiding PCID and INVPCID from the guest?
>
> Yep, I've tried to disable PCID and INVPCID features by nopcid and
> noinvpcid kernel cmdline flags.
> noinvpcid not effects on the problem, but nopcid does! Fantastic!
>
> Of course, masking CPU feature from qemu side is also works:
> <cpu mode='host-model' check='partial'>
> <feature policy='disable' name='pcid'/>
> </cpu>
>
> Now, thanks to your advice, I will try to understand why the PCID
> feature breaks VMs. I see
> that we've some support for this feature in our host kernel (based on
> RHEL7 3.10.0-1160.53.1), probably
> We have some bugs or are not handling something PCID-related from the KVM side.
>
> Thanks again, I couldn't have pulled this off without your advice, Sean.
>
> >
> > > - without KVM (on Host)
> >
> > ...
> >
> > > ==== trace-cmd record -b 20000 -e kvm:kvm_cr -e kvm:kvm_userspace_exit -e probe:* =====
> > >
> > > CPU-1834 [003] 69194.833364: kvm_userspace_exit: reason KVM_EXIT_IO (2)
> > > CPU-1838 [000] 69194.834177: kvm_multiple_exception_L9: (ffffffff814313c6) vcpu=0xffff93ee9a528000
> > > CPU-1838 [000] 69194.834180: kvm_multiple_exception_L41: (ffffffff81431493) vcpu=0xffff93ee9a528000 exception=0xd000001 has_error=0x0 nr=0xd error_code=0x0 has_payload=0x0
> > > CPU-1838 [000] 69194.834195: kvm_multiple_exception_L9: (ffffffff814313c6) vcpu=0xffff93ee9a528000
> > > CPU-1838 [000] 69194.834196: kvm_multiple_exception_L41: (ffffffff81431493) vcpu=0xffff93ee9a528000 exception=0x8000100 has_error=0x0 nr=0x8 error_code=0x0 has_payload=0x0
> > > CPU-1838 [000] 69194.834200: shutdown_interception_L8: (ffffffff8146e4a0)
> >
> > If you can modify the host kernel, throwing a WARN in kvm_multiple_exception() should
> > pinpoint the source of the #GP. Though you may get unlucky and find that KVM is just
> > reflecting an intercepted a #GP that was first "injected" by hardware. Note that this
> > could spam the log if KVM is injecting a large number of #GPs.
> >
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index 9cea051ca62e..19d959bf97cc 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -612,6 +612,8 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu,
> > u32 prev_nr;
> > int class1, class2;
> >
> > + WARN_ON(nr == GP_VECTOR);
> > +
> > kvm_make_request(KVM_REQ_EVENT, vcpu);
> >
> > if (!vcpu->arch.exception.pending && !vcpu->arch.exception.injected) {
> >
>
> Thanks! I'll try to play with that.
>
> Best regards,
> Alex
Just in case if someone will meet the same issue with CPU hotplug on RHEL7 with AMD Milan.
This patch (authored by Babu Moger) helps:
KVM: SVM: Clear the CR4 register on reset
https://patchwork.kernel.org/project/kvm/patch/161471109108.30811.6392805173629704166.stgit@bmoger-ubuntu/
It's easy to apply it to RHEL7 kernel code:
https://lists.openvz.org/pipermail/devel/2022-June/079762.html
It fixes problem because cpu hotplug triggers the same codepath as early start of kexec'ed kernel.
Huge thanks to Sean Christopherson for pointing me out to PCID and Babu Moger for the fix :-)
Regards,
Alex
