On 2/6/2024 12:51 PM, Andrea Righi wrote: > - stress-ng --matrix seems quite unpredictable to be used a benchmarks > in this scenario (the bogo-ops/s are very susceptible to any kind of > interference, so even if in the long runs NO_HZ_FULL still seems to > provide some benefits looking at the average, we also need to > consider that there might be a significant error in the measurements, > standard deviation was pretty high) > Ack on the bogo-ops disclaimers as also mentioned in the stress-ng docs. Agreed a better metric for perf is helpful. I am assuming you also have RCU_NOCB enabled for this test? > - fio doing short writes (in page cache) seems to perform like 2x > better in terms of iops with nohz_full, respect to the other cases > and it performs 2x slower with large IO writes (not sure why... need > to investigate more) This is interesting, it could be worth counting how many kernel entries/exits occur for large IO vs small IO. I'd imagine for large IO we have fewer syscalls and hence lower entry/exit overhead. But if there more interrupts for whatever reason with large IO, then that also implies more kernel entries/exits. As Frederic was saying, NOHZ_FULL has higher overhead on kernel entry/exit. > > - with lazy RCU enabled hrtimer_interrupt() takes like 2x more to > return, respect to the other cases (is this expected?) It depends on which hrtimer_interrupt() instance? There must be several in the trace due to unrelated timers. Are you saying the worst case or it is always 2x more? We do queue a timer for Lazy RCU to flush the RCU work but it is set to 10 seconds and should be canceled most of the time (Its just a guard rail). It is possible there is lock contention on ->nocb_gp_lock which is causing the timer handler execution to be slow. We have several trace_rcu_nocb* trace points, including for the timer. Perhaps you could enable those and we dig deeper? Further, it is interesting to see if it is only the hrtimer_interrupt() instance that also results in a call to do_nocb_deferred_wakeup_timer() via say function tracing. That will confirm that it is the lazy timer that is slow for you. The actual number of callbacks should not be causing specifically the hrtimer_interrupt() to take too long to run, AFAICS. But RCU's lazy feature does increase the number of timer interrupts. Further still, it depends on how much hrtimer_interrupt() takes with lazy RCU to call it a problem IMO. Some numbers with units will be nice. thanks, - Joel
