On 29-Sep-23 5:23 PM, Peter Zijlstra wrote: > On Wed, Sep 27, 2023 at 10:27:07AM -0700, Sean Christopherson wrote: > >> I don't think it does work, at least not without a very, very carefully crafted >> setup and a host userspace that knows it must not use certain aspects of perf. >> E.g. for PEBS, if the guest virtual counters don't map 1:1 to the "real" counters >> in hardware, KVM+perf simply disables the counter. > > I have distinct memories of there being patches to rewrite the PEBS > buffer, but I really can't remember what we ended up doing. Like I said, > I can't operate KVM in any meaningful way -- it's a monster :-( > >> And for top-down slots, getting anything remotely accurate requires pinning vCPUs >> 1:1 with pCPUs and enumerating an accurate toplogy to the guest: >> >> The count is distributed among unhalted logical processors (hyper-threads) who >> share the same physical core, in processors that support Intel Hyper-Threading >> Technology. > > So IIRC slots is per logical CPU, it counts the actual pipeline stages > going towards that logical CPU, this is required to make it work on SMT > at all -- even for native. > > But it's been a long while since that was explained -- and because it > was a call, I can't very well read it back, god how I hate calls :-( > >> Jumping the gun a bit (we're in the *super* early stages of scraping together a >> rough PoC), but I think we should effectively put KVM's current vPMU support into >> maintenance-only mode, i.e. stop adding new features unless they are *very* simple >> to enable, and instead pursue an implementation that (a) lets userspace (and/or >> the kernel builder) completely disable host perf (or possibly just host perf usage >> of the hardware PMU) and (b) let KVM passthrough the entire hardware PMU when it >> has been turned off in the host. > > I don't think you need to go that far, host can use PMU just fine as > long as it doesn't overlap with a vCPU. Basically, if you force > perf_attr::exclude_guest on everything your vCPU can haz the full thing. > >> Hardware vendors are pushing us in the direction whether we like it or not, e.g. >> SNP and TDX want to disallow profiling the guest from the host, > > Yeah, sekjoerity model etc.. bah. > >> ARM has an upcoming PMU model where (IIUC) it can't be virtualized >> without a passthrough approach, > > :-( > >> Intel's hybrid CPUs are a complete trainwreck unless vCPUs are pinned, > > Anybodies hybrid things are a clusterfuck, hybrid vs virt doesn't work > sanely on ARM either AFAIU. > > I intensely dislike hybrid (and virt ofc), but alas we get to live with > that mess :/ And it's only going to get worse I fear.. > > At least (for now) AMD hybrid is committed to identical ISA, including > PMUs with their Zen4+Zen4c things. We'll have to wait and see how > that'll end up. > >> and virtualizing things like top-down slots, PEBS, and LBRs in the shared model >> requires an absurd amount of complexity throughout the kernel and userspace. > > I'm not sure about top-down, the other two, for sure. > > My main beef with top-down is the ludicrously bad hardware interface we > have on big cores, I like the atom interface a *ton* better. > >> Note, a similar idea was floated and rejected in the past[*], but that failed >> proposal tried to retain host perf+PMU functionality by making the behavior dynamic, >> which I agree would create an awful ABI for the host. If we make the "knob" a >> Kconfig > > Must not be Kconfig, distros would have no sane choice. > >> or kernel param, i.e. require the platform owner to opt-out of using perf >> no later than at boot time, then I think we can provide a sane ABI, keep the >> implementation simple, all without breaking existing users that utilize perf in >> the host to profile guests. > > It's a shit choice to have to make. At the same time I'm not sure I have > a better proposal. How about keying off based on PMU specific KVM module parameter? Something like what Manali has proposed for AMD VIBS? Please see solution 1.1: https://lore.kernel.org/r/3a6c693e-1ef4-6542-bc90-d4468773b97d@xxxxxxx > It does mean a host cannot profile one guest and have pass-through on the > other. Eg. have a development and production guest on the same box. This > is pretty crap. > > Making it a guest-boot-option would allow that, but then the host gets > complicated again. I think I can make it trivially work for per-task > events, simply error the creation of events without exclude_guest for > affected vCPU tasks. But the CPU events are tricky. > > > I will firmly reject anything that takes the PMU away from the host > entirely through. > > Also, NMI watchdog needs a solution.. Ideally hardware grows a second > per-CPU timer we can program to NMI. Thanks, Ravi
