On Jul 20, 2015, at 8:11 PM, Tom Talpey <tom@xxxxxxxxxx> wrote:
> On 7/20/2015 4:36 PM, Chuck Lever wrote:
>>
>> On Jul 20, 2015, at 6:41 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote:
>>
>>> On Mon, Jul 20, 2015 at 06:31:11PM -0400, Chuck Lever wrote:
>>>>
>>>> On Jul 20, 2015, at 6:26 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>>> On Mon, Jul 20, 2015 at 03:03:11PM -0400, Chuck Lever wrote:
>>>>>> + iov->length = size;
>>>>>> + iov->lkey = ia->ri_have_dma_lkey ?
>>>>>> + ia->ri_dma_lkey : ia->ri_bind_mem->lkey;
>>>>>> + rb->rg_size = size;
>>>>>> + rb->rg_owner = NULL;
>>>>>> return rb;
>>>>>
>>>>> There is something odd looking about this..
>>>>>
>>>>> ri_bind_mem is only setup in the RPCRDMA_ALLPHYSICAL and
>>>>> RPCRDMA_MTHCAFMR cases.
>>>>>
>>>>> RPCRDMA_FRMR doesn't set it up.
>>>>>
>>>>> So this code in rpcrdma_alloc_regbuf is never called for the FRMR
>>>>> case?
>>>>>
>>>>> If yes, then, how is FRMR working? There is absolutely no reason to
>>>>> use FRMR to register local send buffers, just use the global all
>>>>> memory lkey...
>>>>>
>>>>> If no, then that is an oops?
>>>>
>>>> I’ve tested this code, no oops.
>>>>
>>>> FRWR always uses the DMA lkey. xprtrdma does not use FRWR if
>>>> IB_DEVICE_LOCAL_DMA_LKEY is not asserted.
>>>
>>> Ah, I see. Ok.
>>>
>>> Is there a reason to link FRWR and LOCAL_DMA_LKEY together?
>>
>> Tom would know.
>
> Tom Talpey or Tom Tucker? I don't think I used a dma_lkey at all
> in the original work, but there were some changes later.
The commit which adds FRWR support to xprtrdma introduced a check in
rpcrdma_ia_open:
+ case RPCRDMA_FRMR:
+ /* Requires both frmr reg and local dma lkey */
+ if ((devattr.device_cap_flags &
+ (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) !=
+ (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) {
This seems to be the only place the local DMA lkey requirement is
expressed or enforced. But yeah, the local DMA lkey doesn’t seem
to be used anywhere in the FRWR-specific parts of the code.
--
Chuck Lever
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
