On 7/20/2015 5:34 PM, Chuck Lever wrote:
On Jul 20, 2015, at 8:11 PM, Tom Talpey <tom@xxxxxxxxxx> wrote:
On 7/20/2015 4:36 PM, Chuck Lever wrote:
On Jul 20, 2015, at 6:41 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote:
On Mon, Jul 20, 2015 at 06:31:11PM -0400, Chuck Lever wrote:
On Jul 20, 2015, at 6:26 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote:
On Mon, Jul 20, 2015 at 03:03:11PM -0400, Chuck Lever wrote:
+ iov->length = size;
+ iov->lkey = ia->ri_have_dma_lkey ?
+ ia->ri_dma_lkey : ia->ri_bind_mem->lkey;
+ rb->rg_size = size;
+ rb->rg_owner = NULL;
return rb;
There is something odd looking about this..
ri_bind_mem is only setup in the RPCRDMA_ALLPHYSICAL and
RPCRDMA_MTHCAFMR cases.
RPCRDMA_FRMR doesn't set it up.
So this code in rpcrdma_alloc_regbuf is never called for the FRMR
case?
If yes, then, how is FRMR working? There is absolutely no reason to
use FRMR to register local send buffers, just use the global all
memory lkey...
If no, then that is an oops?
I’ve tested this code, no oops.
FRWR always uses the DMA lkey. xprtrdma does not use FRWR if
IB_DEVICE_LOCAL_DMA_LKEY is not asserted.
Ah, I see. Ok.
Is there a reason to link FRWR and LOCAL_DMA_LKEY together?
Tom would know.
Tom Talpey or Tom Tucker? I don't think I used a dma_lkey at all
in the original work, but there were some changes later.
The commit which adds FRWR support to xprtrdma introduced a check in
rpcrdma_ia_open:
+ case RPCRDMA_FRMR:
+ /* Requires both frmr reg and local dma lkey */
+ if ((devattr.device_cap_flags &
+ (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) !=
+ (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) {
This seems to be the only place the local DMA lkey requirement is
expressed or enforced. But yeah, the local DMA lkey doesn’t seem
to be used anywhere in the FRWR-specific parts of the code.
I now vaguely remember someone telling me that both attributes were
required, but a) I may have misunderstood and b) I'm sure something
has changed. FRMR was brand new at the time.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
