On Fri, Jun 07, 2019 at 02:51:11PM -0300, Jason Gunthorpe wrote:
> On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote:
> > This patch set complete the cleanup done in the driver/verbs/uverbs
> > where the dependency of the code in the ib_x uobject pointer was
> > removed.
> >
> > The uobject pointer is removed from the ib_pd as last step
> > before I can start adding the pd sharing code.
> >
> > The rdma/netlink code now don't have dependency in the ib_pd
> > uobject pointer and can report multiple context id that point
> > to same ib_pd.
> >
> > Using iproute2 I can test the modified rdma/netlink:
> > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0
> > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core]
> > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core]
> > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib]
> > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib]
> > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp]
> > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt]
> > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw
> > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw
> >
> > Changelog:
> >
> > v1->v2
> > * 1 patch from v1 applied (Jason)
> > * Fix uobj_get_obj_read macro (Jason)
> > * Do not allocate memory when fixing uobj_get_obj_read (Jason)
> > * Fix uobj_get_obj_read macro (Jason)
> > * rdma/netlink can now work as before (Leon)
> >
> > v2->v3:
> > * rdma/netlink nest multiple context ids of same ib_pd (Leon)
> >
> > Shamir Rabinovitch (4):
> > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject
> > RDMA/uverbs: uobj_put_obj_read macro should be removed
> > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext
> > IB/{core,hw}: ib_pd should not have ib_uobject pointer
>
> I have to think it is rather a clunky approach..
>
> I keep moving the _cmd flow closer to the attrs flow - so to do that
> we should be recording the uobj 'get' in the uverbs_attr_bundle and
> then putting them when the handler exits automatically. Just like the
> attr flow does.
>
> This requires a bit of moving code around so that the write path has
> the bundle_priv which records all of this.
>
> What do you think?
>
> Jason
Hi Jason, sorry for the delay,
The above make perfect sense to me and can help to solve figure where
uobjects are left as uobj_get_read. It do have to solve the cases where
some code is doing a multiple uobj_get_read in same uberbs write call
(as we had in ib_uverbs_post_send). It can be done after the shared pd
patch set is adopted if it is acceptable by you.
For the shared pd I'd rather to suggest the follow path to make things
faster and get Yuval's help:
Next patch-set:
kernel:
- ib_mr->uobject cleanup from (Yuval)
- Add generic sharing support for ib_x object destroy in IB/core
- Add export_to_fd verb
- Add generic 'clone' for ib_x objects
- Enable export of ib_pd
- Add clone to ib_pd
- Add clone of ib_pd to mlx4, mlx5, rxe
- Add import_pd verb
rdma-core:
- Add export_to_fd verb
- Add import_pd verb
Follow patch-set:
Kernel:
- Enable export of ib_mr (Yuval)
- Add clone to ib_mr (Yuval)
- Add clone of ib_mr to mlx4, mlx5, rxe (Yuval)
- Add import_mr verb (Yuval)
rdma-core:
- Add import_mr verb (Yuval)
Next patch-set:
rdma-core:
- Add shpd pingpong sample app based on the new verbs (Yuval/me)
I have some time constraint and I appreciate the time you take to review
and progress this feature.
Thanks
