> On Nov 10, 2015, at 4:30 PM, Tom Talpey <tom@xxxxxxxxxx> wrote: > > On 11/10/2015 4:17 PM, Jason Gunthorpe wrote: >> On Tue, Nov 10, 2015 at 04:00:43PM -0500, Tom Talpey wrote: >> >>> Hmm, agreed, but it must still be acceptable to perform a registration >>> instead of using the local_dma_lkey. As mentioned earlier, there are >>> scatter limits and other implications for all-physical addressing that >>> an upper layer may choose to avoid. >> >> It is always acceptable to use a lkey MR instead of the local dma >> lkey, but ULPs should prefer to use the local dma lkey if possible, >> for performance reasons. > > Sure, the key words are "if possible". For example, a single 1MB > RDMA Read is unlikely to be possible with the dma lkey, assuming > worst-case physical page scatter it would need 256 SGEs. But 1MB > can be registered easily. > > In any case, my point was that the ULP gets to choose, so, it looks > like we agree. I’d like to see our NFS server use the local DMA lkey where it makes sense, to avoid the cost of registering and invalidating memory. I have to agree with Tom that once the device’s s/g limit is exceeded, the server has to post an RDMA Read WR every few pages, and appears to get expensive for large NFS requests. The current mechanism of statically choosing either FRMR or local DMA depending on the device is probably not adequate. Maybe we could post all the Read WRs via a single chain? Or stick with FRMR when the amount of data to read is significant. I’ve also tested Christoph’s patch. The logic currently in rdma_read_chunk_lcl does not seem up to the task. Once the device’s s/g limit is exceeded, the server starts throwing local length exceptions, and the client workload hangs. None of this is impossible to fix, but there is some work to do here. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
