Hi Colton, On Monday 25 Mar 2024 at 20:12:04 (+0000), Colton Lewis wrote: > Thanks for the feedback. > > Quentin Perret <qperret@xxxxxxxxxx> writes: > > > On Friday 22 Mar 2024 at 14:24:35 (+0000), Quentin Perret wrote: > > > On Tuesday 19 Mar 2024 at 16:43:41 (+0000), Colton Lewis wrote: > > > > Add a KVM_CAP to control WFx (WFI or WFE) trapping based on scheduler > > > > runqueue depth. This is so they can be passed through if the runqueue > > > > is shallow or the CPU has support for direct interrupt injection. They > > > > may be always trapped by setting this value to 0. Technically this > > > > means traps will be cleared when the runqueue depth is 0, but that > > > > implies nothing is running anyway so there is no reason to care. The > > > > default value is 1 to preserve previous behavior before adding this > > > > option. > > > > I recently discovered that this was enabled by default, but it's not > > > obvious to me everyone will want this enabled, so I'm in favour of > > > figuring out a way to turn it off (in fact we might want to make this > > > feature opt in as the status quo used to be to always trap). > > Setting the introduced threshold to zero will cause it to trap whenever > something is running. Is there a problem with doing it that way? No problem per se, I was simply hoping we could set the default to zero to revert to the old behaviour. I don't think removing WFx traps was a universally desirable behaviour, so it prob should have been opt-in from the start. > I'd also be interested to get more input before changing the current > default behavior. Ack, that is my personal opinion. > > > There are a few potential issues I see with having this enabled: > > > > - a lone vcpu thread on a CPU will completely screw up the host > > > scheduler's load tracking metrics if the vCPU actually spends a > > > significant amount of time in WFI (the PELT signal will no longer > > > be a good proxy for "how much CPU time does this task need"); > > > > - the scheduler's decision will impact massively the behaviour of the > > > vcpu task itself. Co-scheduling a task with a vcpu task (or not) will > > > impact massively the perceived behaviour of the vcpu task in a way > > > that is entirely unpredictable to the scheduler; > > > > - while the above problems might be OK for some users, I don't think > > > this will always be true, e.g. when running on big.LITTLE systems the > > > above sounds nightmare-ish; > > > > - the guest spending long periods of time in WFI prevents the host from > > > being able to enter deeper idle states, which will impact power very > > > negatively; > > > > And probably a whole bunch of other things. > > > > > Think about his option as a threshold. The instruction will be trapped > > > > if the runqueue depth is higher than the threshold. > > > > So talking about the exact interface, I'm not sure exposing this to > > > userspace is really appropriate. The current rq depth is next to > > > impossible for userspace to control well. > > Using runqueue depth is going off of a suggestion from Oliver [1], who I've > also talked to internally at Google a few times about this. > > But hearing your comment makes me lean more towards having some > enumeration of behaviors like TRAP_ALWAYS, TRAP_NEVER, > TRAP_IF_MULTIPLE_TASKS. Do you guys really expect to set this TRAP_IF_MULTIPLE_TASKS? Again, the rq depth is quite hard to reason about from userspace, so not sure anybody will really want that? A simpler on/off thing might be simpler. > > > My gut feeling tells me we might want to gate all of this on > > > PREEMPT_FULL instead, since PREEMPT_FULL is pretty much a way to say > > > "I'm willing to give up scheduler tracking accuracy to gain throughput > > > when I've got a task running alone on a CPU". Thoughts? > > > And obviously I meant s/PREEMPT_FULL/NOHZ_FULL, but hopefully that was > > clear :-) > > Sounds good to me but I've not touched anything scheduling related before. Do you guys use NOHZ_FULL in prod? If not that idea might very well be a non-starter, because switching to NOHZ_FULL would be a big ask. So, yeah, I'm curious :) Thanks, Quentin
