On 7/20/2015 3:41 PM, Steve Wise wrote:
-----Original Message-----
From: Tom Talpey [mailto:tom@xxxxxxxxxx]
Sent: Monday, July 20, 2015 5:04 PM
To: Steve Wise; 'Jason Gunthorpe'
Cc: 'Chuck Lever'; linux-rdma@xxxxxxxxxxxxxxx; linux-nfs@xxxxxxxxxxxxxxx
Subject: Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site
On 7/20/2015 2:16 PM, Steve Wise wrote:
-----Original Message-----
From: linux-nfs-owner@xxxxxxxxxxxxxxx [mailto:linux-nfs-owner@xxxxxxxxxxxxxxx] On Behalf Of Jason Gunthorpe
Sent: Monday, July 20, 2015 4:06 PM
To: Tom Talpey; Steve Wise
Cc: Chuck Lever; linux-rdma@xxxxxxxxxxxxxxx; linux-nfs@xxxxxxxxxxxxxxx
Subject: Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site
On Mon, Jul 20, 2015 at 01:34:16PM -0700, Tom Talpey wrote:
On 7/20/2015 12:03 PM, Chuck Lever wrote:
All HCA providers have an ib_get_dma_mr() verb. Thus
rpcrdma_ia_open() will either grab the device's local_dma_key if one
is available, or it will call ib_get_dma_mr() which is a 100%
guaranteed fallback.
I recall that in the past, some providers did not support mapping
all of the machine's potential physical memory with a single dma_mr.
If an rnic did/does not support 44-ish bits of length per region,
for example.
Looks like you are right, but the standard in kernel is to require
ib_get_dma_mr, if the HCA can't do that, then it cannot be used on a
big memory machine with kernel ULPs.
Looking deeper, both amso1100 and cxgb3 seem limited to 32 bits of
physical memory, and silently break all kernel ULPs if they are used
on a modern machine with > 4G.
Is that right Steve?
Yes.
Based on that, should we remove the cxgb3 driver as well? Or at least
can you fix it up to at least fail get_dma_mr if there is too much
ram?
I would like to keep cxgb3 around. I can add code to fail if the memory is > 32b. Do you know how I get the amount of
available
ram?
A) are you sure it's an unsigned length, i.e. is it really 31 bits?
yes.
B) why bother to check? Are machines with <4GB interesting, and worth
supporting a special optimization?
No, but cxgb3 is still interesting to user applications, and perhaps NFSRDMA using FRMRs.
I'm obviously not making myself clear. I am suggesting that cxgb3 fail
the ib_get_dma_mr() verb, regardless of installed memory.
I am not suggesting it fail to load, or fail other memreg requests. It
should work normally in all other respects.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
