On Mon, Apr 15, 2024, Xiong Y Zhang wrote:
> On 4/13/2024 2:32 AM, Sean Christopherson wrote:
> > On Fri, Apr 12, 2024, Xiong Y Zhang wrote:
> >>>> 2. NMI watchdog
> >>>> the perf event for NMI watchdog is a system wide cpu pinned event, it
> >>>> will be stopped also during vm running, but it doesn't have
> >>>> attr.exclude_guest=1, we add it in this RFC. But this still means NMI
> >>>> watchdog loses function during VM running.
> >>>>
> >>>> Two candidates exist for replacing perf event of NMI watchdog:
> >>>> a. Buddy hardlock detector[3] may be not reliable to replace perf event.
> >>>> b. HPET-based hardlock detector [4] isn't in the upstream kernel.
> >>>
> >>> I think the simplest solution is to allow mediated PMU usage if and only if
> >>> the NMI watchdog is disabled. Then whether or not the host replaces the NMI
> >>> watchdog with something else becomes an orthogonal discussion, i.e. not KVM's
> >>> problem to solve.
> >> Make sense. KVM should not affect host high priority work.
> >> NMI watchdog is a client of perf and is a system wide perf event, perf can't
> >> distinguish a system wide perf event is NMI watchdog or others, so how about
> >> we extend this suggestion to all the system wide perf events ? mediated PMU
> >> is only allowed when all system wide perf events are disabled or non-exist at
> >> vm creation.
> >
> > What other kernel-driven system wide perf events are there?
> does "kernel-driven" mean perf events created through
> perf_event_create_kernel_counter() like nmi_watchdog and kvm perf events ?
By kernel-driven I meant events that aren't tied to a single userspace process
or action.
E.g. KVM creates events, but those events are effectively user-driven because
they will go away if the associated VM terminates.
> User can create system wide perf event through "perf record -e {} -a" also, I
> call it as user-driven system wide perf events. Perf subsystem doesn't
> distinguish "kernel-driven" and "user-driven" system wide perf events.
Right, but us humans can build a list, even if it's only for documentation, e.g.
to provide help for someone to run KVM guests with mediated PMUs, but can't
because there are active !exclude_guest events.
> >> but NMI watchdog is usually enabled, this will limit mediated PMU usage.
> >
> > I don't think it is at all unreasonable to require users that want optimal PMU
> > virtualization to adjust their environment. And we can and should document the
> > tradeoffs and alternatives, e.g. so that users that want better PMU results don't
> > need to re-discover all the "gotchas" on their own.
> >
> > This would even be one of the rare times where I would be ok with a dmesg log.
> > E.g. if KVM is loaded with enable_mediated_pmu=true, but there are system wide
> > perf events, pr_warn() to explain the conflict and direct the user at documentation
> > explaining how to make their system compatible with mediate PMU usage.>
> >>>> 3. Dedicated kvm_pmi_vector
> >>>> In emulated vPMU, host PMI handler notify KVM to inject a virtual
> >>>> PMI into guest when physical PMI belongs to guest counter. If the
> >>>> same mechanism is used in passthrough vPMU and PMI skid exists
> >>>> which cause physical PMI belonging to guest happens after VM-exit,
> >>>> then the host PMI handler couldn't identify this PMI belongs to
> >>>> host or guest.
> >>>> So this RFC uses a dedicated kvm_pmi_vector, PMI belonging to guest
> >>>> has this vector only. The PMI belonging to host still has an NMI
> >>>> vector.
> >>>>
> >>>> Without considering PMI skid especially for AMD, the host NMI vector
> >>>> could be used for guest PMI also, this method is simpler and doesn't
> >>>
> >>> I don't see how multiplexing NMIs between guest and host is simpler. At best,
> >>> the complexity is a wash, just in different locations, and I highly doubt it's
> >>> a wash. AFAIK, there is no way to precisely know that an NMI came in via the
> >>> LVTPC.
> >> when kvm_intel.pt_mode=PT_MODE_HOST_GUEST, guest PT's PMI is a multiplexing
> >> NMI between guest and host, we could extend guest PT's PMI framework to
> >> mediated PMU. so I think this is simpler.
> >
> > Heh, what do you mean by "this"? Using a dedicated IRQ vector, or extending the
> > PT framework of multiplexing NMI?
> here "this" means "extending the PT framework of multiplexing NMI".
The PT framework's multiplexing is just as crude as regular PMIs though. Perf
basically just asks KVM: is this yours? And KVM simply checks that the callback
occurred while KVM_HANDLING_NMI is set.
E.g. prior to commit 11df586d774f ("KVM: VMX: Handle NMI VM-Exits in noinstr region"),
nothing would prevent perf from miscontruing a host PMI as a guest PMI, because
KVM re-enabled host PT prior to servicing guest NMIs, i.e. host PT would be active
while KVM_HANDLING_NMI is set.
And conversely, if a guest PMI skids past VM-Exit, as things currently stand, the
NMI will always be treated as host PMI, because KVM will not be in KVM_HANDLING_NMI.
KVM's emulated PMI can (and should) eliminate false positives for host PMIs by
precisely checking exclude_guest, but that doesn't help with false negatives for
guest PMIs, nor does it help with NMIs that aren't perf related, i.e. didn't come
from the LVTPC.
Is a naive implementation simpler? Maybe. But IMO, multiplexing NMI and getting
all the edge cases right is more complex than using a dedicated vector for guest
PMIs, as the latter provides a "hard" boundary and allows the kernel to _know_ that
an interrupt is for a guest PMI.
