> -----Original Message----- > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma-owner@xxxxxxxxxxxxxxx] On Behalf Of Sagi Grimberg > Sent: Monday, July 06, 2015 11:18 AM > To: Steve Wise; dledford@xxxxxxxxxx > Cc: sagig@xxxxxxxxxxxx; ogerlitz@xxxxxxxxxxxx; roid@xxxxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; eli@xxxxxxxxxxxx; target- > devel@xxxxxxxxxxxxxxx; linux-nfs@xxxxxxxxxxxxxxx; trond.myklebust@xxxxxxxxxxxxxxx; bfields@xxxxxxxxxxxx > Subject: Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags > > On 7/6/2015 5:37 PM, Steve Wise wrote: > > > > > >> -----Original Message----- > >> From: Sagi Grimberg [mailto:sagig@xxxxxxxxxxxxxxxxxx] > >> Sent: Monday, July 06, 2015 2:54 AM > >> To: Steve Wise; dledford@xxxxxxxxxx > >> Cc: sagig@xxxxxxxxxxxx; ogerlitz@xxxxxxxxxxxx; roid@xxxxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; eli@xxxxxxxxxxxx; target- > >> devel@xxxxxxxxxxxxxxx; linux-nfs@xxxxxxxxxxxxxxx; trond.myklebust@xxxxxxxxxxxxxxx; bfields@xxxxxxxxxxxx > >> Subject: Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags > >> > >> On 7/6/2015 2:22 AM, Steve Wise wrote: > >>> The semantics for MR access flags are not consistent across RDMA > >>> protocols. So rather than have applications try and glean what they > >>> need, have them pass in the intended roles and attributes for the MR to > >>> be allocated and let the RDMA core select the appropriate access flags > >>> given the roles, attributes, and device capabilities. > >>> > >>> We introduce rdma_mr_roles and rdma_mr_attributes that enumerate the > >>> possible roles and attributes for a MR. These are documented in the > >>> enums themselves. > >>> > >>> New services exported: > >>> > >>> rdma_device_access_flags() - given the intended MR roles and attributes > >>> passed in, return the ib_access_flags bits for the device. > >>> > >>> rdma_get_dma_mr() - allocate a dma mr using the applications intended > >>> MR roles and MR attributes. This uses rdma_device_access_flags(). > >>> > >>> rdma_fast_reg_access_flags() - return the ib_access_flags bits needed > >>> for a fast register WR given the applications intended MR roles and > >>> MR attributes. This uses rdma_device_access_flags(). > >>> > >>> Signed-off-by: Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> > >>> --- > >>> > >>> drivers/infiniband/core/verbs.c | 30 +++++++++++ > >>> include/rdma/ib_verbs.h | 106 +++++++++++++++++++++++++++++++++++++++ > >>> 2 files changed, 136 insertions(+), 0 deletions(-) > >>> > >>> diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c > >>> index bac3fb4..f42595c 100644 > >>> --- a/drivers/infiniband/core/verbs.c > >>> +++ b/drivers/infiniband/core/verbs.c > >>> @@ -1144,6 +1144,36 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int mr_access_flags) > >>> } > >>> EXPORT_SYMBOL(ib_get_dma_mr); > >>> > >>> +int rdma_device_access_flags(struct ib_pd *pd, int roles, int attrs) > >>> +{ > >>> + int access_flags = attrs; > >>> + > >>> + if (roles & RDMA_MRR_RECV) > >>> + access_flags |= IB_ACCESS_LOCAL_WRITE; > >>> + > >>> + if (roles & RDMA_MRR_WRITE_DEST) > >>> + access_flags |= IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE; > >>> + > >>> + if (roles & RDMA_MRR_READ_DEST) { > >>> + access_flags |= IB_ACCESS_LOCAL_WRITE; > >>> + if (rdma_protocol_iwarp(pd->device, > >>> + rdma_start_port(pd->device))) > >>> + access_flags |= IB_ACCESS_REMOTE_WRITE; > >>> + } > >>> + > >>> + if (roles & RDMA_MRR_READ_SOURCE) > >>> + access_flags |= IB_ACCESS_REMOTE_READ; > >>> + > >>> + if (roles & RDMA_MRR_ATOMIC_DEST) > >>> + access_flags |= IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_ATOMIC; > >>> + > >>> + if (roles & RDMA_MRR_MW_BIND) > >>> + access_flags |= IB_ACCESS_MW_BIND; > >>> + > >>> + return access_flags; > >>> +} > >>> +EXPORT_SYMBOL(rdma_device_access_flags); > >>> + > >>> struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd, > >>> struct ib_phys_buf *phys_buf_array, > >>> int num_phys_buf, > >>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h > >>> index 986fddb..da1eadf 100644 > >>> --- a/include/rdma/ib_verbs.h > >>> +++ b/include/rdma/ib_verbs.h > >>> @@ -2494,6 +2494,112 @@ static inline int ib_req_ncomp_notif(struct ib_cq *cq, int wc_cnt) > >>> struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int mr_access_flags); > >>> > >>> /** > >>> + * rdma_mr_roles - possible roles an RDMA MR will be used for > >>> + * > >>> + * This allows a transport independent RDMA application to > >>> + * create MRs that are usable for all the desired roles w/o > >>> + * having to understand which access rights are needed. > >>> + */ > >>> +enum { > >>> + > >>> + /* lkey used in a ib_recv_wr sge */ > >>> + RDMA_MRR_RECV = 1, > >>> + > >>> + /* lkey used for IB_WR_SEND* ops in the ib_send_wr sge */ > >>> + RDMA_MRR_SEND = (1<<1), > >>> + > >>> + /* rkey used for IB_WR_RDMA_READ* ops in ib_send_wr wr.rdma.rkey */ > >>> + RDMA_MRR_READ_SOURCE = (1<<2), > >>> + > >>> + /* lkey used for IB_WR_RDMA_READ* ops in the ib_send_wr sge */ > >>> + RDMA_MRR_READ_DEST = (1<<3), > >>> + > >>> + /* lkey used for IB_WR_RDMA_WRITE* ops in the ib_send_wr sge */ > >>> + RDMA_MRR_WRITE_SOURCE = (1<<4), > >>> + > >>> + /* rkey used for IB_WR_RDMA_WRITE* ops in ib_send_wr wr.rdma.rkey */ > >>> + RDMA_MRR_WRITE_DEST = (1<<5), > >>> + > >>> + /* > >>> + * lkey used for IB_WR_ATOMIC/MASKED_ATOMIC* ops in the ib_send_wr sge > >>> + */ > >>> + RDMA_MRR_ATOMIC_SOURCE = (1<<6), > >>> + > >>> + /* > >>> + * rkey used for IB_WR_ATOMIC/MASKED_ATOMIC* ops in ib_send_wr > >>> + * wr.atomic.rkey > >>> + */ > >>> + RDMA_MRR_ATOMIC_DEST = (1<<7), > >>> + > >>> + /* MR used for a IB_WR_MW_BIND in ib_send_wr wr.bind_mw.bind_info.mr */ > >>> + RDMA_MRR_MW_BIND = (1<<8), > >>> +}; > >>> + > >>> +/** > >>> + * rdma_mr_attributes - attributes for rdma memory regions > >>> + */ > >>> +enum { > >>> + RDMA_MRA_ZERO_BASED = IB_ZERO_BASED, > >>> + RDMA_MRA_ACCESS_ON_DEMAND = IB_ACCESS_ON_DEMAND, > >>> +}; > >>> + > >>> +/** > >>> + * rdma_device_access_flags - Returns the device-specific MR access flags. > >>> + * @pd: The protection domain associated with the memory region. > >>> + * @roles: The intended roles of the MR > >>> + * @attrs: The desired attributes of the MR > >>> + * > >>> + * Use the intended roles from @roles along with @attrs and the device > >>> + * capabilities to generate the needed access rights. > >>> + * > >>> + * Return: the ib_access_flags value needed to allocate the MR. > >>> + */ > >>> +int rdma_device_access_flags(struct ib_pd *pd, int roles, int attrs); > >>> + > >>> +/** > >>> + * rdma_get_dma_mr - Returns a memory region for system memory that is > >>> + * usable for DMA. > >>> + * @pd: The protection domain associated with the memory region. > >>> + * @roles: The intended roles of the MR > >>> + * @attrs: The desired attributes of the MR > >>> + * > >>> + * Use the intended roles from @roles along with @attrs and the device > >>> + * capabilities to define the needed access rights, and call > >>> + * ib_get_dma_mr() to allocate the MR. > >>> + * > >>> + * Note that the ib_dma_*() functions defined below must be used > >>> + * to create/destroy addresses used with the Lkey or Rkey returned > >>> + * by ib_get_dma_mr(). > >>> + * > >>> + * Return: struct ib_mr pointer upon success, or a pointer encoded errno upon > >>> + * failure. > >>> + */ > >>> +static inline struct ib_mr *rdma_get_dma_mr(struct ib_pd *pd, int roles, > >>> + int attrs) > >>> +{ > >>> + return ib_get_dma_mr(pd, rdma_device_access_flags(pd, roles, attrs)); > >>> +} > >>> + > >> > >> I still think this wrapper should go away... > > > > Ok. I'll remove all uses of ib_get_dma_mr()... > > > > > > I meant that rdma_get_dma_mr can go away. I'd prefer to get the > needed access_flags and just call existing verb. > So just export rdma_device_access_flags(rd, roles, attrs), and then change the ULPs to call this to obtain the access flags. That is ok with me. What do others think? -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
