On 10/28/14 19:32, Sagi Grimberg wrote:
On 10/21/2014 12:10 PM, Sagi Grimberg wrote:
On 10/20/2014 3:56 PM, Bart Van Assche wrote:
On 10/19/14 19:36, Sagi Grimberg wrote:
On 10/7/2014 4:07 PM, Bart Van Assche wrote:
* comp_vector, a number in the range 0..n-1 specifying the
- MSI-X completion vector. Some HCA's allocate multiple (n)
- MSI-X vectors per HCA port. If the IRQ affinity masks of
- these interrupts have been configured such that each MSI-X
- interrupt is handled by a different CPU then the
comp_vector
- parameter can be used to spread the SRP completion workload
- over multiple CPU's.
+ MSI-X completion vector of the first RDMA channel. Some
+ HCA's allocate multiple (n) MSI-X vectors per HCA port. If
+ the IRQ affinity masks of these interrupts have been
+ configured such that each MSI-X interrupt is handled by a
+ different CPU then the comp_vector parameter can be used to
+ spread the SRP completion workload over multiple CPU's.
This is fairly not trivial for the user...
Aren't we requesting a bit too much awareness here?
Can't we just "make it work"? The user hands out ch_count - why can't
you do some least-used logic here?
Maybe we can even go with per-cpu QPs and discard comp_vector argument?
this would probably bring the best performance, wouldn't it?
(fallback to least-used logic in case HW support less vectors)
The only reason the comp_vector parameter is still supported is because
of backwards compatibility. What I expect is that users will set the
ch_count parameter but not the comp_vector parameter.
Another wander I have with this. Say you have 8 cores on a single numa
node. First connection will attach to vectors 0-3 (ch_count=4) and so
are all the connections. Don't we want to spread that a little?
If we are not going per-cpu, why aren't we trying to spread vectors
around to try and reduce the interference?
Hello Sagi,
Sorry but your question is not entirely clear to me. Are you referring
to spreading the workload over CPU's or over completion vectors ? If a
user wants to spread the completion workload maximally by using all
completion vectors that can be achieved by setting ch_count to a value
that is equal to or larger than the number of completion vectors.
As mentioned in the commit message, spreading the completion workload
over CPU's is not entirely under control of the SRP initiator driver. It
is assumed that a user assigns IRQ affinity such that the interrupts
associated with different completion vectors are processed by different
CPU threads. If there are more RDMA channels than completion vectors,
the SRP initiator driver associates RDMA channels with completion
vectors such that the workload is spread evenly over the completion vectors.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
