* Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx> wrote: > On Tue, May 19, 2009 at 02:44:36PM +0200, Ingo Molnar wrote: > > > > * Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx> wrote: > > > > > On Tue, May 19, 2009 at 10:58:25AM +0200, Ingo Molnar wrote: > > > > > > > > * Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx> wrote: > > > > > > > > > On Mon, May 18, 2009 at 05:42:41PM +0200, Ingo Molnar wrote: > > > > > > > > > > > > * Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx> wrote: > > > > > > > > > > > > > > i might be missing something fundamental here, but why not just > > > > > > > > have per CPU helper threads, all on the same waitqueue, and wake > > > > > > > > them up via a single wake_up() call? That would remove the SMP > > > > > > > > cross call (wakeups do immediate cross-calls already). > > > > > > > > > > > > > > My concern with this is that the cache misses accessing all the > > > > > > > processes on this single waitqueue would be serialized, slowing > > > > > > > things down. In contrast, the bitmask that smp_call_function() > > > > > > > traverses delivers on the order of a thousand CPUs' worth of bits > > > > > > > per cache miss. I will give it a try, though. > > > > > > > > > > > > At least if you go via the migration threads, you can queue up > > > > > > requests to them locally. But there's going to be cachemisses > > > > > > _anyway_, since you have to access them all from a single CPU, > > > > > > and then they have to fetch details about what to do, and then > > > > > > have to notify the originator about completion. > > > > > > > > > > Ah, so you are suggesting that I use smp_call_function() to run > > > > > code on each CPU that wakes up that CPU's migration thread? I > > > > > will take a look at this. > > > > > > > > My suggestion was to queue up a dummy 'struct migration_req' up with > > > > it (change migration_req::task == NULL to mean 'nothing') and simply > > > > wake it up using wake_up_process(). > > > > > > OK. I was thinking of just using wake_up_process() without the > > > migration_req structure, and unconditionally setting a per-CPU > > > variable from within migration_thread() just before the list_empty() > > > check. In your approach we would need a NULL-pointer check just > > > before the call to __migrate_task(). > > > > > > > That will force a quiescent state, without the need for any extra > > > > information, right? > > > > > > Yep! > > > > > > > This is what the scheduler code does, roughly: > > > > > > > > wake_up_process(rq->migration_thread); > > > > wait_for_completion(&req.done); > > > > > > > > and this will always have to perform well. The 'req' could be put > > > > into PER_CPU, and a loop could be done like this: > > > > > > > > for_each_online_cpu(cpu) > > > > wake_up_process(cpu_rq(cpu)->migration_thread); > > > > > > > > for_each_online_cpu(cpu) > > > > wait_for_completion(&per_cpu(req, cpu).done); > > > > > > > > hm? > > > > > > My concern is the linear slowdown for large systems, but this > > > should be OK for modest systems (a few 10s of CPUs). However, I > > > will try it out -- it does not need to be a long-term solution, > > > after all. > > > > I think there is going to be a linear slowdown no matter what - > > because sending that many IPIs is going to be linear. (there are > > no 'broadcast to all' IPIs anymore - on x86 we only have them if > > all physical APIC IDs are 7 or smaller.) > > With the current code, agreed. One could imagine making an IPI > tree, so that a given CPU IPIs (say) eight subordinates. Making > this work nice with CPU hotplug would be entertaining, to say the > least. Certainly! :-) As a general note, unrelated to your patches: i think our CPU-hotplug related complexity seems to be a bit too much. This is really just a gut feeling - from having seen many patches that also have hotplug notifiers. I'm wondering whether this is because it's structured in a suboptimal way, or because i'm (intuitively) under-estimating the complexity of what it takes to express what happens when a CPU is offlined and then onlined? Ingo -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
