On 2024-04-25 11:12 p.m., Mingwei Zhang wrote: >>>> the perf_guest_enter(). The stale information is saved in the KVM. Perf >>>> will schedule the event in the next perf_guest_exit(). KVM will not know it. >>> Ya, the creation of an event on a CPU that currently has guest PMU state loaded >>> is what I had in mind when I suggested a callback in my sketch: >>> >>> : D. Add a perf callback that is invoked from IRQ context when perf wants to >>> : configure a new PMU-based events, *before* actually programming the MSRs, >>> : and have KVM's callback put the guest PMU state >> >> when host creates a perf event with exclude_guest attribute which is >> used to profile KVM/VMM user space, the vCPU process could work at three >> places. >> >> 1. in guest state (non-root mode) >> >> 2. inside vcpu-loop >> >> 3. outside vcpu-loop >> >> Since the PMU state has already been switched to host state, we don't >> need to consider the case 3 and only care about cases 1 and 2. >> >> when host creates a perf event with exclude_guest attribute to profile >> KVM/VMM user space, an IPI is triggered to enable the perf event >> eventually like the following code shows. >> >> event_function_call(event, __perf_event_enable, NULL); >> >> For case 1, a vm-exit is triggered and KVM starts to process the >> vm-exit and then run IPI irq handler, exactly speaking >> __perf_event_enable() to enable the perf event. >> >> For case 2, the IPI irq handler would preempt the vcpu-loop and call >> __perf_event_enable() to enable the perf event. >> >> So IMO KVM just needs to provide a callback to switch guest/host PMU >> state, and __perf_event_enable() calls this callback before really >> touching PMU MSRs. > ok, in this case, do we still need KVM to query perf if there are > active exclude_guest events? yes? Because there is an ordering issue. > The above suggests that the host-level perf profiling comes when a VM > is already running, there is an IPI that can invoke the callback and > trigger preemption. In this case, KVM should switch the context from > guest to host. What if it is the other way around, ie., host-level > profiling runs first and then VM runs? > > In this case, just before entering the vcpu loop, kvm should check > whether there is an active host event and save that into a pmu data > structure. KVM doesn't need to save/restore the host state. Host perf has the information and will reload the values whenever the host events are rescheduled. But I think KVM should clear the registers used by the host to prevent the value leaks to the guest. > If none, do the context switch early (so that KVM saves a > huge amount of unnecessary PMU context switches in the future). > Otherwise, keep the host PMU context until vm-enter. At the time of > vm-exit, do the check again using the data stored in pmu structure. If > there is an active event do the context switch to the host PMU, > otherwise defer that until exiting the vcpu loop. Of course, in the > meantime, if there is any perf profiling started causing the IPI, the > irq handler calls the callback, preempting the guest PMU context. If > that happens, at the time of exiting the vcpu boundary, PMU context > switch is skipped since it is already done. Of course, note that the > irq could come at any time, so the PMU context switch in all 4 > locations need to check the state flag (and skip the context switch if > needed). > > So this requires vcpu->pmu has two pieces of state information: 1) the > flag similar to TIF_NEED_FPU_LOAD; 2) host perf context info (phase #1 > just a boolean; phase #2, bitmap of occupied counters). > > This is a non-trivial optimization on the PMU context switch. I am > thinking about splitting them into the following phases: > > 1) lazy PMU context switch, i.e., wait until the guest touches PMU MSR > for the 1st time. > 2) fast PMU context switch on KVM side, i.e., KVM checking event > selector value (enable/disable) and selectively switch PMU state > (reducing rd/wr msrs) > 3) dynamic PMU context boundary, ie., KVM can dynamically choose PMU > context switch boundary depending on existing active host-level > events. > 3.1) more accurate dynamic PMU context switch, ie., KVM checking > host-level counter position and further reduces the number of msr > accesses. > 4) guest PMU context preemption, i.e., any new host-level perf > profiling can immediately preempt the guest PMU in the vcpu loop > (instead of waiting for the next PMU context switch in KVM). I'm not quit sure about the 4. The new host-level perf must be an exclude_guest event. It should not be scheduled when a guest is using the PMU. Why do we want to preempt the guest PMU? The current implementation in perf doesn't schedule any exclude_guest events when a guest is running. Thanks, Kan
