On 3/18/2020 10:03 AM, Leon Romanovsky wrote:
On Wed, Mar 18, 2020 at 09:52:27AM +0200, Yishai Hadas wrote:
On 3/17/2020 9:51 PM, Jason Gunthorpe wrote:
On Tue, Mar 17, 2020 at 11:10:15AM -0400, Andrew Boyer wrote:
On Mar 17, 2020, at 11:00 AM, Leon Romanovsky <leonro@xxxxxxxxxxxx> wrote:
On Tue, Mar 17, 2020 at 10:45:08AM -0400, Andrew Boyer wrote:
Hello Leon,
I understand that we are not to create new providers that use environment variables to control locking behavior. The ‘new’ way to do it is to use thread domains and parent domains.
However, even mlx5 still uses the env var exclusively to control lockless behavior for CQs. Do you have anything in mind for how to extend thread_domains or some other part of the API to cover the CQ case?
Which parameter did you have in mind?
I would say that all those parameters are coming from pre-rdma-core era.
Doesn't this commit do what you are asking?
https://github.com/linux-rdma/rdma-core/commit/0dbde57c59d2983e848c3dbd9ae93eaf8e7b9405
Thanks
Thank you,
Andrew
You are right - I got thrown off by this:
if (mlx5_spinlock_init(&cq->lock, !mlx5_single_threaded))
goto err;
If IBV_CREATE_CQ_ATTR_SINGLE_THREADED is set, it passes an argument
to the polling function to skip the lock calls entirely. So it
doesn’t matter that they are still enabled internally.
Hmm, a thread domain should probably force
IBV_CREATE_CQ_ATTR_SINGLE_THREADED during greation with the new API..
Yishai?
We can really consider extending the functionality of a parent domain that
was just added for CQ in case it holds a thread domain for this purpose.
However, current code enforces creating a dedicated BF as part of thread
domain unless we extend ibv_alloc_td() to ask only for marking the single
thread functionality.
The existing alternative is or to use the legacy ENV that mentioned above or
to use the ibv_cq_ex functionality which upon its creation gets an explicit
flag for single threaded one (i.e. IBV_CREATE_CQ_ATTR_SINGLE_THREADED).
"dedicated BF", isn't this Mellanox internal implementation?
Yes, however from API point of view we can consider extending
ibv_alloc_td() to ask for this specific usage that may prevent this
resource allocation, unless we are fine with having it always which may
give a benefit also for a QP that will use it down the road.
As pointed above, there are already few alternatives that may be used in
the meantime.
Yishai
