On Thu, Sep 21, 2017 at 4:41 AM, Jason Gunthorpe
<jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote:
> But, as you say, you'd have to solve the MR sharing problem at least
> to make PD workable. Perhaps some kind of 'child' PD that was only for
> threading..
OK, this definitely sounds like an interesting approach.
// alloc a child ibv_pd, which inherits the parent properties and adds
some limitations (e.g. thread domain, memory domain)
Two options I'm thinking in this spirit:
1) Extend the original alloc PD API (stay with a single PD alloc entry point):
struct ibv_pd *ibv_alloc_pd_ex(
struct ibv_context *my_ctx,
struct ibv_pd_init_attr {
ibv_pd* parent_pd = my_pd, // use NULL to alloc a
parent PD, or use a parent pd to alloc a child PD
int flags = IBV_PD_PARENT | IBV_PD_INIT_ATTR_THREAD_DOMAIN)
} );
2) New dedicated alloc Child PD API:
struct ibv_pd *ibv_alloc_child_pd(struct ibv_pd *pd, int flags =
IBV_PD_INIT_ATTR_THREAD_DOMAIN);
> That goes both ways, a single 'max' does not make sense if there are
> multiple different sorts of objects being controlled (eg UAR and
> something else). It only works well in your context of having a single
> constrained UAR.
I agree.
> Probably the better option is to introduce something like a 'struct
> verbs_locking_domain *' input which is a proper object and can have a
> range of operators.
I did not understand how the 'struct verbs_locking_domain' comes into play here.
In regards to the child PD, I'm assuming the 'struct ibv_pd' will be
encap in a 'struct verbs_pd', and/or maybe encap in a 'struct
provider_pd'. Any linkage and dedicated resource should be stored in
the internal structures.
> I would say each creating locking domain objects is just strided
> across the various limited resources in the device.
With the above verbs API for child PD on dedicated thread, user gets a
best effort allocation of thread resource. I agree that striding
across resources sounds good.
> 'dv' accessors, eg mlxdv_set_locking_uar() can be provided to directly
> assign specific limited resources under the control of the
> application. 'dv' query functions can be used to directly learn the
> specific limits for each specific thing.
int mlx5dv_set_pd_uar_index(struct ibv_pd* pd, int uar_index);
So with DV API, application can selectively choose a UAR index for
Mellanox HW for a child (or parent) PD.
All objects created with this PD will use the chosen UAR index or
other thread options.
max_uar_index will be expose by the DV device caps via
'mlx5dv_query_device()'. If user exceeds the max_uar_index, the above
function call will fail.
Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
