On Wed, Jun 05, 2019 at 05:46:51PM +0300, Leon Romanovsky wrote:
> On Wed, Jun 05, 2019 at 04:02:45PM +0300, Shamir Rabinovitch wrote:
> > On Wed, Jun 05, 2019 at 11:25:49AM +0300, Leon Romanovsky wrote:
> > > On Wed, Jun 05, 2019 at 10:21:26AM +0300, Shamir Rabinovitch wrote:
> > > > On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote:
> > > > > This patch set complete the cleanup done in the driver/verbs/uverbs
> > > > > where the dependency of the code in the ib_x uobject pointer was
> > > > > removed.
> > > > >
> > > > > The uobject pointer is removed from the ib_pd as last step
> > > > > before I can start adding the pd sharing code.
> > > > >
> > > > > The rdma/netlink code now don't have dependency in the ib_pd
> > > > > uobject pointer and can report multiple context id that point
> > > > > to same ib_pd.
> > > > >
> > > > > Using iproute2 I can test the modified rdma/netlink:
> > > > > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0
> > > > > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core]
> > > > > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core]
> > > > > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib]
> > > > > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib]
> > > > > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp]
> > > > > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt]
> > > > > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw
> > > > > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw
> > > > >
> > > > > Changelog:
> > > > >
> > > > > v1->v2
> > > > > * 1 patch from v1 applied (Jason)
> > > > > * Fix uobj_get_obj_read macro (Jason)
> > > > > * Do not allocate memory when fixing uobj_get_obj_read (Jason)
> > > > > * Fix uobj_get_obj_read macro (Jason)
> > > > > * rdma/netlink can now work as before (Leon)
> > > > >
> > > > > v2->v3:
> > > > > * rdma/netlink nest multiple context ids of same ib_pd (Leon)
> > > > >
> > > > > Shamir Rabinovitch (4):
> > > > > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject
> > > > > RDMA/uverbs: uobj_put_obj_read macro should be removed
> > > > > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext
> > > > > IB/{core,hw}: ib_pd should not have ib_uobject pointer
> > > > >
> > > > > drivers/infiniband/core/nldev.c | 129 +++++++++++-
> > > > > drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++--------
> > > > > drivers/infiniband/core/verbs.c | 1 -
> > > > > drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 -
> > > > > drivers/infiniband/hw/mlx5/main.c | 1 -
> > > > > drivers/infiniband/hw/mthca/mthca_qp.c | 3 +-
> > > > > include/rdma/ib_verbs.h | 1 -
> > > > > include/rdma/uverbs_std_types.h | 11 +-
> > > > > include/uapi/rdma/rdma_netlink.h | 3 +
> > > > > 9 files changed, 273 insertions(+), 95 deletions(-)
> > > > >
> > > > > --
> > > > > 2.20.1
> > > > >
> > > >
> > > > Jason, Leon, can you please review this patch set ?
> > >
> > > I'm sorry for the delay.
> > >
> > > >
> > > > Anything missing from my side here?
> > >
> > > Can you please post rdmatool output for shared PD?
> > > In such case, all those shared PD need to have same PDN.
> >
> > Leon I do not have all the pieces in place for this yes.
> >
> > I only tested that current rdmatool can cope with netlink message that
> > has nested context ids as it will happen in shared pd case.
>
> Thanks, multiple PDNs worried me and I was afraid that they come from
> shared PDs. You will still need to mock something to see that you
> print them correctly.
I tried to check this by iterating twice on the code in
'fill_res_pd_entry' for user pd.
kernel confirm that we have duplicate lines for each pd:
[ 4707.051279] fill_res_pd_entry: iter 0
[ 4707.052371] fill_res_pd_entry: iter 1
[ 4707.053426] fill_res_pd_entry: iter 0
[ 4707.054529] fill_res_pd_entry: iter 1
But the rdmatool prints remain as-is:
[root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0
dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core]
dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core]
dev mlx4_0 pdn 2 local_dma_lkey 0x0 users 2 ctxn 0 pid 12918 comm ib_send_bw
dev mlx4_0 pdn 3 local_dma_lkey 0x0 users 2 ctxn 1 pid 12919 comm ib_send_bw
Does this make sense to you ?
Since we do not have any crash in the rdmatool in this case, can we
postpone this test till we have proper shared pd ?
Can you please review the series ?
Thanks
>
> >
> > I'll do that once we have the export_to_fd & import_pd & import_mr verbs
> > in place.
> >
> > >
> > > Thanks
