On Mon, Feb 14, 2011 at 5:10 PM, Peter LaDow <petela@xxxxxxxxxxxxxxx> wrote:
> On Mon, Feb 14, 2011 at 3:30 PM, Frank Rowand <frank.rowand@xxxxxxxxx> wrote:
>> On Mon, Feb 14, 2011 at 2:31 PM, Peter LaDow <petela@xxxxxxxxxxxxxxx> wrote:
>>> How is the scheduling of the hrtimers softirq thread handled?
>>>
>>> When querying the RT priority of the hrtimer softirq, I get a priority
>>> of 50. But when running a priority 99 thread, we still seem to be
>>> getting interrupted. Shouldn't the hrtimer softirq be put off until
>>> the CPU is idle or a lower priority task is running?
>>
>> Is the hrtimer softirq executing when the priority 99 thread is spinning
>> in it's for loop? Your "jitter Due to Large Number of Timers" email
>> said that the lower priority tasks don't seem to be interrupting the
>> priority 99 thread.
>
> Did I? Hmm, well I mean the lower priority task with 100 threads. At
> least I think so. It is hard to tell.
>
> It seems to me that the softirq thread is the source of the problem.
> Since the tight loop is getting such a variety of times (400us of
> jitter only while the other process is running) that it does seem that
> the loop is getting interrupt.
Just so we are speaking with a common definition of jitter, your first email
said that the duration of the priority 99 thread loop increased by
around 350us (average and maximum) when the lower priority task
timers were added to the system.
>
>> The hardware timer interupts will interrupt the priority 99 thread. The
>> cost of these interrupts and the resultant calls to try_to_wake_up()
>> of the hrtimer softirq might be quite large considering the rate of
>> timer expires you mentioned in your first email.
>
> Sure, we expect the timer interrupt to interfere. But as we
So what is the overhead of the timer interrupt?
1) All 100 of the the test100 timers pop at the same time:
- 1 hardware interrupt
- Chasing through an hrtimer list of size 100
- call try_to_wake_up() for the hrtimer softirq
(I don't remember whether try_to_wake_up() will be called just once,
or once per timer. But even if called 100 times, the first call is
"expensive" and the other 99 will be very cheap).
2) Each of the 100 test100 timers pop at a separate, unique time:
- 100 hardware interrupts
- For each interrupt, chase through an hrtimer list of size 1
- For each interrupt, call try_to_wake_up(). The first call is
"expensive", for the other 99 interrupts the call will be cheap.
3) Then every other possible combination of clumps of timers popping
at the same time.
4) Just for completeness, not all of the test100 timers has to pop
during each iteration of the priority 99 thread loop, but that does not
impact the analysis of the worst case scenario, so we can just ignore
that.
I would expect scenario 1 to have the lowest overhead, scenario 2 to
have the highest overhead, and scenario 3 to be in the middle.
For a 533 Mhz PPC, I would not be surprised if the overhead of these
three scenarios is as large as 350 us.
> understand it, the softirq is what schedules the task switch. The top
> half only schedules the bottom half. But since the bottom half is
> priority 50, there shouldn't be any interruption of the priority 99
> expect to handle the low level IRQ.
You can verify whether any other process is executing through a variety
of tools. LTT if you have it in your kernel. I think 2.6.29 had ftrace
"Trace process context switches" (CONFIG_CONTEXT_SWITCH_TRACER),
I don't think there was a perf option for context switches in 2.6.29.
>
>> Out of curiosity, is the system UP or SMP?
>
> UP. Just a single MPC5349.
>
> pete
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
