On Mon, Apr 08, 2024 at 10:16:24AM -0700, Sean Christopherson wrote: > On Fri, Apr 05, 2024, Paul E. McKenney wrote: > > On Fri, Apr 05, 2024 at 07:42:35AM -0700, Sean Christopherson wrote: > > > On Fri, Apr 05, 2024, Marcelo Tosatti wrote: > > > > rcuc wakes up (which might exceed the allowed latency threshold > > > > for certain realtime apps). > > > > > > Isn't that a false negative? (RCU doesn't detect that a CPU is about to (re)enter > > > a guest) I was trying to ask about the case where RCU thinks a CPU is about to > > > enter a guest, but the CPU never does (at least, not in the immediate future). > > > > > > Or am I just not understanding how RCU's kthreads work? > > > > It is quite possible that the current rcu_pending() code needs help, > > given the possibility of vCPU preemption. I have heard of people doing > > nested KVM virtualization -- or is that no longer a thing? > > Nested virtualization is still very much a thing, but I don't see how it is at > all unique with respect to RCU grace periods and quiescent states. More below. > > > But the help might well involve RCU telling the hypervisor that a given > > vCPU needs to run. Not sure how that would go over, though it has been > > prototyped a couple times in the context of RCU priority boosting. > > > > > > > > 3 - It checks if the guest exit happened over than 1 second ago. This 1 > > > > > > second value was copied from rcu_nohz_full_cpu() which checks if the > > > > > > grace period started over than a second ago. If this value is bad, > > > > > > I have no issue changing it. > > > > > > > > > > IMO, checking if a CPU "recently" ran a KVM vCPU is a suboptimal heuristic regardless > > > > > of what magic time threshold is used. > > > > > > > > Why? It works for this particular purpose. > > > > > > Because maintaining magic numbers is no fun, AFAICT the heurisitic doesn't guard > > > against edge cases, and I'm pretty sure we can do better with about the same amount > > > of effort/churn. > > > > Beyond a certain point, we have no choice. How long should RCU let > > a CPU run with preemption disabled before complaining? We choose 21 > > seconds in mainline and some distros choose 60 seconds. Android chooses > > 20 milliseconds for synchronize_rcu_expedited() grace periods. > > Issuing a warning based on an arbitrary time limit is wildly different than using > an arbitrary time window to make functional decisions. My objection to the "assume > the CPU will enter a quiescent state if it exited a KVM guest in the last second" > is that there are plenty of scenarios where that assumption falls apart, i.e. where > _that_ physical CPU will not re-enter the guest. > > Off the top of my head: > > - If the vCPU is migrated to a different physical CPU (pCPU), the *old* pCPU > will get false positives, and the *new* pCPU will get false negatives (though > the false negatives aren't all that problematic since the pCPU will enter a > quiescent state on the next VM-Enter. > > - If the vCPU halts, in which case KVM will schedule out the vCPU/task, i.e. > won't re-enter the guest. And so the pCPU will get false positives until the > vCPU gets a wake event or the 1 second window expires. > > - If the VM terminates, the pCPU will get false positives until the 1 second > window expires. > > The false positives are solvable problems, by hooking vcpu_put() to reset > kvm_last_guest_exit. And to help with the false negatives when a vCPU task is > scheduled in on a different pCPU, KVM would hook vcpu_load(). Sean, It seems that fixing the problems you pointed out above is a way to go.
