On Tue, 7 Aug 2018 13:30:39 -0600
Jason Gunthorpe <jgg@xxxxxxxx> wrote:
> But isn't it a bit strange overall that CQ processing works globally
> single threaded? If mad cares I bet everything else does too..
>
> Is there some reason why we need to have a bound work queue at all?
>From Sagi's commit:
commit b7363e67b23e04c23c2a99437feefac7292a88bc
Author: Sagi Grimberg <sagi@xxxxxxxxxxx>
Date: Wed Mar 8 22:03:17 2017 +0200
IB/device: Convert ib-comp-wq to be CPU-bound
This workqueue is used by our storage target mode ULPs
via the new CQ API. Recent observations when working
with very high-end flash storage devices reveal that
UNBOUND workqueue threads can migrate between cpu cores
and even numa nodes (although some numa locality is accounted
for).
While this attribute can be useful in some workloads,
it does not fit in very nicely with the normal
run-to-completion model we usually use in our target-mode
ULPs and the block-mq irq<->cpu affinity facilities.
The whole block-mq concept is that the completion will
land on the same cpu where the submission was performed.
The fact that our submitter thread is migrating cpus
can break this locality.
We assume that as a target mode ULP, we will serve multiple
initiators/clients and we can spread the load enough without
having to use unbound kworkers.
Also, while we're at it, expose this workqueue via sysfs which
is harmless and can be useful for debug.
Signed-off-by: Sagi Grimberg <sagi@xxxxxxxxxxx>
Reviewed-by: Bart Van Assche <bart.vanassche@xxxxxxxxxxx>--
Signed-off-by: Doug Ledford <dledford@xxxxxxxxxx>
Unfortunately, rather than simply add a bound WQ type (and leave the
"default" IB_POLL_WORKQUEUE unbound), they chose to change the default
type to bound. Now, any users of the CQ api who want an unbound WQ need
to explicitly request IB_POLL_UNBOUND_WQ provided by this patch.
Oh, well.
Anyhow, the only users of IB_POLL_WQ are given below (I searched under
directory drivers/infiniband):
core/cq.c:177: case IB_POLL_WORKQUEUE:
core/cq.c:215: case IB_POLL_WORKQUEUE:
core/mad.c:3172: IB_POLL_WORKQUEUE);
ulp/srpt/ib_srpt.c:1733: 0 /* XXX: spread CQs */,IB_POLL_WORKQUEUE);
ulp/isert/ib_isert.c:281: IB_POLL_WORKQUEUE);
Thus, at least for now, the only users of the new CQ api are the
storage ULPs and MADs.
Does an iser target also require a bound WQ?
-Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
