On 1/29/2014 4:13 PM, Bart Van Assche wrote:
On 01/28/14 22:02, Or Gerlitz wrote:
Roland, ping! the signature patches were posted > three months ago.
I have a question about RDMA and T10-DIF support. The Linux block layer,
the Linux SCSI core, Linux filesystems and block drivers all support
discontiguous buffers. These are buffers that cannot be mapped onto a
single contiguous range of virtual memory addresses. I think the RDMA
FMR and FR memory registration mechanisms only support mapping a range
of addresses that can be mapped onto a contiguous region of virtual
addresses. This means that either multiple RDMA memory regions are
needed to map a discontiguous buffer or that the buffer contents has to
be copied before associating it with a memory region. What I understood
from the mlx5 T10-DIF patch series is that the API introduced in that
patch series is such that only a single memory region per data transfer
operation is supported. This made me wonder how to support T10-DIF for
discontiguous buffers.
Hey Bart, Thanks for the observation,
You are correct, the T10-PI RDMA offload API requires a single memory
region that describes the data
and a single memory region that describes the protection information.
Non-contiguous buffers are a well known problem/challenge for RDMA
transports, each handles them
differently: iSER uses bounce buffers, SRP registers multiple contiguous
regions. The T10-PI offload API does
not impose any other constraint on this situation, each ULP will act the
same: iSER will copy to contiguous
bounce buffers and SRP will register multiple "Signature" memory regions
(each for a contiguous chunk of data)
and send the rkeys to the target.
Would one of the options below perhaps be an
appropriate solution ?
- Add a flag in struct request_queue that tells the block layer to use a
bounce buffer (data copying) for I/O requests with a discontiguous
data buffer. The block layer already supports copying data buffers for
which e.g. DMA alignment requirements are not met. Rework patch
"IB/iser: Introduce fast memory registration model" (commit
5587856c9659ac2d6ab201141aa8a5c2ff3be4cd) such that it takes advantage
of that new flag. See also blk_rq_map_user_iov() for an example of
how the block layer decides when copying a data buffer is necessary.
Didn't understand why should it matter where the copy is done (iser/block)?
- Modify the patches that add T10-DIF such that discontiguous buffers
are supported.
I thought about adding some kind of logic to T10-PI RDMA API to try and
solve the alignment
issue at the same time, but the fact is that the two are unrelated, and
I figured it is not
such a good idea. Non-contiguous memory registration is separate problem
(hint: under
development) and should be addressed separately, T10-PI RDMA offload
should play well
with it.
Sorry that I had not realized this before.
Bart.
Hops this helps,
Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
