On 11/03/2008, Hiroshi Shimamoto <h-shimamoto@xxxxxxxxxxxxx> wrote: > Peter Zijlstra wrote: > > On Mon, 2008-03-10 at 19:12 -0700, Hiroshi Shimamoto wrote: > >> Peter Zijlstra wrote: > >>> On Mon, 2008-03-10 at 13:01 -0700, Hiroshi Shimamoto wrote: > >>> > >>>> thanks, your patch looks nice to me. > >>>> I had focused setprio, on_rq=0 and running=1 situation, it makes me to > >>>> fix these functions. > >>>> But one point, I've just noticed. I'm not sure on same situation against > >>>> sched_rt. I think the pre_schedule() of rt has chance to drop rq lock. > >>>> Is it OK? > >>> Ah, you are quite right, that'll teach me to rush out a patch just > >>> because dinner is ready :-). > >>> > >>> How about we submit the following patch for mainline and CC -stable to > >>> fix .23 and .24: > >>> > >> Unfortunately, I encountered similar panic with this patch on -rt. > >> I'll look into this, again. I might have missed something... > >> > >> Unable to handle kernel NULL pointer dereference at 0000000000000128 RIP: > >> [<ffffffff802297f5>] pick_next_task_fair+0x2d/0x42 > > > > :-( > > > > OK, so that means I'm not getting it. > > > > So what does your patch do that mine doesn't? > > > > It removes the dependency of running (=task_current()) from on_rq > > (p->se.on_rq). > > > > So how can a current task not be on the runqueue? > > > > Only sched.c:dequeue_task() and sched_fair.c:account_entity_dequeue() > > set on_rq to 0, the only one changing rq->curr is schedule(). > > > > So the only scheme I can come up with is that we did dequeue p (on_rq == > > 0), but we didn't yet schedule so rq->curr == p. > > > > Is this how you ended up with your previuos analysis that it must be due > > to a hole introduced by double_lock_balance()? > > > > Because now we can seemingly call deactivate_task() and put_prev_task() > > in non-atomic fashion, but by placing the put_prev_task() before the > > load balance calls we should avoid doing that. > > > > So what else is going on... /me puzzled > > > thanks for taking time. > I've tested it with debug tracer, it took several hours to half day to > reproduce it on my box. And I got the possible scenario. > > Before begin, I can tell that se->on_rq is changed at enqueue_task() or > dequeue_task() in sched.c. > > Here is the flow to panic which I got; > > CPU0 CPU1 > | schedule() > | ->deactivate_task() > > | -->dequeue_task() > | * on_rq=0 > | ->put_prev_task_fair() > > | ->idle_balance() > | -->load_balance_newidle() > > (a wakeup function) | > > | --->double_lock_balance() > *get lock *rel lock > > * wake up target is CPU1's curr | > ->enqueue_task() | > * on_rq=1 | > ->rt_mutex_setprio() | > * on_rq=1, ruuning=1 | > -->dequeue_task()!! | > -->put_prev_task_fair()!! | humm... this one should have caused the problem. ->put_prev_task() has been previously done in schedule() so we get 2 consequent ->put_prev_task() without set_curr_task/pick_next_task() being called in between [ as a result, __enqueue_entitty() is called twice for CPU1's curr and that definitely corrupts an rb-tree ] your initial patch doesn't have this problem. humm... logically-wise, it looks like a change of the 'current' which can be expressed by a pair : (1) put_prev_task() + (2) pick_next_task() or set_curr_task() (both end up calling set_next_entity()) has to be 'atomic' wrt the rq->lock. For schedule() that also involves a change of rq->curr. Your initial patch seems to qualify for this rule... but I'm still thinking whether the current scheme is a bit 'hairy' :-/ > > Hiroshi Shimamoto > -- Best regards, Dmitry Adamushko -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
