On 12/7/2014 12:20 PM, Sagi Grimberg wrote:
On 12/4/2014 9:41 PM, Shirley Ma wrote:
On 12/04/2014 10:43 AM, Bart Van Assche wrote:
On 12/04/14 17:47, Shirley Ma wrote:
What's the history of this patch?
http://lists.openfabrics.org/pipermail/general/2008-May/050813.html
I am working on multiple QPs workload. And I created a similar approach
with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small
I/O
performance. I think this CQ_VECTOR loading balance should be
maintained
in provider not the caller. I didn't see this patch was submitted to
mainline kernel, wonder any reason behind?
My interpretation is that an approach similar to
IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but
suboptimal on multi-socket systems. Hence the code for associating CQ
sets with CPU sockets in the SRP initiator. These changes have been
queued for kernel 3.19. See also branch drivers-for-3.19 in git repo
git://git.infradead.org/users/hch/scsi-queue.git.
What I did is that I manually controlled IRQ and working thread on the
same socket. The CQ is created when mounting the file system in
NFS/RDMA, but the workload thread might start from different socket,
so per-cpu based implementation might not apply. I will look at SRP
implementation.
Hey Shirley,
Bart is correct, in general the LEAST_ATTACHED approach might not be
optimal in the NUMA case. The thread <-> QP/CQ/CPU assignment is
addressed by the multi-channel approach which to my understanding won't
be implemented in NFSoRDMA in the near future (right Chuck?)
However, the LEAST_ATTACH vector hint will revive again in the future
as there is a need to spread applications on different interrupt
vectors (especially for user-space).
CC'ing Matan who is working on this, perhaps he can comment on this as
well.
Sagi.
Hi,
I'm not sure LEAST_ATTACHED is the best practice here. Applications
might want to create a CQ on n different cores. You can't guarantee that
with LEAST_ATTACHED policy. Anything smarter would probably require an
API change or some tricks.
We might, for example, add an API like "give me the least attached CQ
vector which isn't in the following list {a, b, c....}". Another option
might be that when several CQs are registered with LEAST_ATTACHED on the
same PD, we'll try to give them different vectors.
Anyway, these are some rough ideas. We should really think about that
thoroughly before implementing.
Regards,
Matan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
