On 2/8/2024 9:51 AM, Uladzislau Rezki wrote: > On Thu, Feb 08, 2024 at 01:53:58PM +0100, Uladzislau Rezki wrote: >> On Thu, Feb 08, 2024 at 07:55:37AM +0100, Andrea Righi wrote: >>> On Thu, Feb 08, 2024 at 12:54:58AM -0500, Joel Fernandes wrote: >>> ... >>>>>> Slightly related, but one of the things we are wondering also is how >>>>>> much of the overhead for your nohz-full and lazy-RCU test (on top of >>>>>> baseline - that is just CONFIG_HZ=1000 without nohz-full or nocbs) is >>>>>> because of just using NOCB. Uladsizlau mentioned he might run a test >>>>>> for comparing along those lines as well. >>>>> >>>>> Just to clarify, "lazy rcu on" results are just with rcu_nocb=all and >>>>> lazy RCUs enabled (and HZ=1000), so without nohz_full. >>>>> >>>>> If I enable only nohz_full=all (without rcu_nocb) I see something like >>>>> this: >>>> >>>> Ok. I did want to mention nohz_full implies rcu_nocb on the same CPUs as well. >>>> >>>> Its also mentioned in the boot param docs on the last line of the description: >>>> >>>> nohz_full= [KNL,BOOT,SMP,ISOL] >>>> The argument is a cpu list, as described above. >>>> In kernels built with CONFIG_NO_HZ_FULL=y, set >>>> the specified list of CPUs whose tick will be stopped >>>> whenever possible. The boot CPU will be forced outside >>>> the range to maintain the timekeeping. Any CPUs >>>> in this list will have their RCU callbacks offloaded, >>>> just as if they had also been called out in the >>>> rcu_nocbs= boot parameter. >>> >>> Ah I didn't realize that, it definitely makes sense, thanks for >>> clarifying it. >>> >>> Then basically in the results that I posted the difference is >>> "nohz_full=all+rcu_nocb=all" vs "rcu_nocb=all+lazy_RCU=on". >>> >> So, you say that a hrtimer_interrupt() handler takes more time in case >> of lazy + nocb + rcu_nocb=all and for nohz_full + rcu_nocb=all it faster? >> Could you please clarify this? I will try to measure from my side! >> >> I have done some basic research about hrtimer_interrupt() latency on my >> HW with latest Linux kernel. I have compared below cases: >> >> case a: 1000HZ + lazy + nocb_all_cpus >> case b: 1000HZ + nocb_all_cpus >> >> I used "ftrace" to measure time(in microseconds). Steps: >> >> echo 0 > tracing_on >> echo function_graph > current_tracer >> echo funcgraph-proc > trace_options >> echo funcgraph-abstime > trace_options >> echo hrtimer_interrupt > set_ftrace_filter >> >> fio --rw=write --bs=1M --size=1G --numjobs=8 --name=worker --time_based --runtime=50& >> >> echo 1 > tracing_on; sleep 10; echo 0 > tracing_on >> >> data is based on 10 seconds collection: >> >> <case a> >> 6 2102 ############################################################ >> 8 2079 ############################################################ >> 10 1464 ########################################## >> 7 897 ########################## So first column is microseconds and second one is count? >> 9 625 ################## >> 12 490 ############## >> 13 479 ############## >> 11 289 ######### >> 5 249 ######## >> 14 124 #### >> 15 72 ### >> 16 41 ## >> 17 24 # >> 4 22 # >> 18 12 # >> 22 2 # >> 19 1 # >> <case a> >> >> <case b> >> 9 1658 ############################################################ >> 13 1308 ################################################ >> 12 1224 ############################################# Assuming that, it does seem the "best" case is off by 3 microseconds (9 vs 6), still would not warrant being regarded a bug and possibly just in the noise. >> 10 972 #################################### >> 8 703 ########################## >> 14 595 ###################### >> 15 571 ##################### >> 11 525 ################### >> 17 350 ############# >> 16 235 ######### >> 7 214 ######## >> 4 73 ### >> 5 68 ### >> 6 54 ## >> 20 9 # >> 18 9 # >> 19 6 # >> 33 1 # >> 3 1 # >> 28 1 # >> 27 1 # >> 25 1 # >> 22 1 # >> 21 1 # >> <case b> >> >> I do not see the difference, there is a nose of 1/2/3 microseconds diff. >> > Let me further have a look at what we use for lazy in terms on hrtimer though. Thanks for tracing it. Yeah it would be nice to count how many counts of do_nocb_deferred_wakeup() does the fio test trigger. If it is few, then maybe the problem with hrtimer_interrupt() is something else. - Joel
