Andi Kleen wrote:
On Wed, Dec 10, 2008 at 10:42:28AM -0800, Vasu Dev wrote:
It had load balancing issue but now it is fixed, related latest
submitted code with its updated comment is:-
/*
* The incoming frame exchange id(oxid) is ANDed with num of online
* cpu bits to get cpu_idx and then this cpu_idx is used for
selecting
* a per cpu kernel thread from fcoe_percpu. In case the cpu is
* offline or no kernel thread for derived cpu_idx then cpu_idx is
* initialize to first online cpu index.
*/
cpu_idx = oxid & (num_online_cpus() - 1);
First note that num_online_cpus() is not guaranteed to be a power of two,
- 1 is not guaranteed to give a suitable mask. So you might actually lose
random bits.
Correct, this will work best for only power of 2 online cpus and that
would be the most common typical use case. I agree it won't load balance
better in non power of 2 cpus case.
Also your load balancing scheme is unusual to say at least.
e.g. when you're just talking to a single frame exchange you would always
transfer data between CPUs instead of keeping it all on the CPU that
processes the interrupt.
Normally the rule of thumb is to use local
data as much as possible. Or when you distribute like this at least
stay in the same socket.
We cannot control what cpu to get interrupted for a FC frame in a
typical generic NIC, so we may end up receiving mostly all FC frames on
a single same cpu though system might have several other cpus available.
In this scenario if frame are passed up to same cpu as suggested above
then that won't do any load balancing, therefore some sort of load
balancing is required based on some FC frame attributes here.
As I said in my last response that "performance tuning is yet to be
done" but you bring up some good related points now of cross socket
frame migration and balancing on non power of 2 cpus system. These
should be considered during pending performance tuning but for now I can
add additional check to select cpu within same socket but not sure how
to do that, any kernel call for this ? This might cause more locking
contentions on libfc structs so really we have to experiment these thing
during performance tuning. Thanks Andi for these hints on performance
consideration.
Vasu
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
