On Jul 20, 2015, at 6:41 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > On Mon, Jul 20, 2015 at 06:31:11PM -0400, Chuck Lever wrote: >> >> On Jul 20, 2015, at 6:26 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: >> >>> On Mon, Jul 20, 2015 at 03:03:11PM -0400, Chuck Lever wrote: >>>> + iov->length = size; >>>> + iov->lkey = ia->ri_have_dma_lkey ? >>>> + ia->ri_dma_lkey : ia->ri_bind_mem->lkey; >>>> + rb->rg_size = size; >>>> + rb->rg_owner = NULL; >>>> return rb; >>> >>> There is something odd looking about this.. >>> >>> ri_bind_mem is only setup in the RPCRDMA_ALLPHYSICAL and >>> RPCRDMA_MTHCAFMR cases. >>> >>> RPCRDMA_FRMR doesn't set it up. >>> >>> So this code in rpcrdma_alloc_regbuf is never called for the FRMR >>> case? >>> >>> If yes, then, how is FRMR working? There is absolutely no reason to >>> use FRMR to register local send buffers, just use the global all >>> memory lkey... >>> >>> If no, then that is an oops? >> >> I’ve tested this code, no oops. >> >> FRWR always uses the DMA lkey. xprtrdma does not use FRWR if >> IB_DEVICE_LOCAL_DMA_LKEY is not asserted. > > Ah, I see. Ok. > > Is there a reason to link FRWR and LOCAL_DMA_LKEY together? Tom would know. > Just use > the code you have for fmr with frwr to get the lkey, probably move it > to rpcrdma_ia_open .. Physical MR should create a 2nd MR dedicated for > rkey use. > > That will work really well with the series I'm working on: > > https://github.com/jgunthorpe/linux/tree/remove-ib_get_dma_mr > > To just drop ib_get_dma_mr entirely. Yes, I sorted this out so it would be obvious how to deal with xprtrdma when the time comes. All three registration modes can just use pd->local_dma_lkey for the SEND/RECV buffers. PHYSICAL would still need ib_get_dma_mr() (or something equivalent) for MRs needing remote access, agreed. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
