On 2/7/2024 12:05 PM, Andrea Righi wrote:
> On Wed, Feb 07, 2024 at 11:52:35AM -0500, Joel Fernandes wrote:
>> On Wed, Feb 7, 2024 at 11:31 AM Andrea Righi <andrea.righi@xxxxxxxxxxxxx> wrote:
>>
>>>> The actual number of callbacks should not be causing specifically the
>>>> hrtimer_interrupt() to take too long to run, AFAICS. But RCU's lazy feature does
>>>> increase the number of timer interrupts.
>>>>
>>>> Further still, it depends on how much hrtimer_interrupt() takes with lazy RCU to
>>>> call it a problem IMO. Some numbers with units will be nice.
>>>
>>> This is what I see (this is a single run, but the other runs are
>>> similar), unit is nanosec, with lazy RCU enabled hrtimer_interrupt()
>>> takes around 4K-16K ns, with lazy RCU off most of the times it takes
>>> 2K-4K ns:
>>>
>>> - lazy rcu off:
>>>
>>> [1K, 2K) 88307 |@@@@@@@@@@@@ |
>>> [2K, 4K) 380695 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
>>> [4K, 8K) 194 | |
>>>
>>> - lazy rcu on:
>>>
>>> [2K, 4K) 3094 | |
>>> [4K, 8K) 265763 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
>>> [8K, 16K) 182341 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
>>> [16K, 32K) 3422 | |
>>>
>>> Again, I'm not sure if this is really a problem or not, or if it is even
>>> a relevant metric for the overall performance, I was just curious to
>>> understand why it is different.
>>
>> This is an interesting find, the number of timer interrupt executions
>> looks roughly the same in this histogram so it might not be missed
>> cancellations or such, so it is not clear to me. But it is worth
>> debugging and we'll try to reproduce your results.
>>
>> Some more theories from our internal RCU discussion:
>> - Could it be another user of RCU (call_rcu) from an unrelated hrtimer
>> interrupt callback that is causing a "flush" of lazy callbacks?
>> - What does the distribution look like for
>> do_nocb_deferred_wakeup_timer ? That will have to probably be made
>> non-static to be picked up by bpftrace (If you could try that real
>> quick, appreciate!).
>
> Sure, I'll repeat the test tracing do_nocb_deferred_wakeup_timer.
Thanks, that'll be great.
>> Slightly related, but one of the things we are wondering also is how
>> much of the overhead for your nohz-full and lazy-RCU test (on top of
>> baseline - that is just CONFIG_HZ=1000 without nohz-full or nocbs) is
>> because of just using NOCB. Uladsizlau mentioned he might run a test
>> for comparing along those lines as well.
>
> Just to clarify, "lazy rcu on" results are just with rcu_nocb=all and
> lazy RCUs enabled (and HZ=1000), so without nohz_full.
>
> If I enable only nohz_full=all (without rcu_nocb) I see something like
> this:
Ok. I did want to mention nohz_full implies rcu_nocb on the same CPUs as well.
Its also mentioned in the boot param docs on the last line of the description:
nohz_full= [KNL,BOOT,SMP,ISOL]
The argument is a cpu list, as described above.
In kernels built with CONFIG_NO_HZ_FULL=y, set
the specified list of CPUs whose tick will be stopped
whenever possible. The boot CPU will be forced outside
the range to maintain the timekeeping. Any CPUs
in this list will have their RCU callbacks offloaded,
just as if they had also been called out in the
rcu_nocbs= boot parameter.
(btw, I am a bit slow to respond to email due to being ill).
thanks,
- Joel
