On 2/1/2023 10:45 PM, Bob Pearson wrote:
On 2/1/23 09:38, Tom Talpey wrote:
On 2/1/2023 6:06 AM, Daisuke Matsuda (Fujitsu) wrote:
On Sat, Jan 28, 2023 6:10 AM Bob Pearson wrote:
Currently the rxe driver does not handle all cases of zero length
rdma operations correctly. The client does not have to provide an
rkey for zero length RDMA operations so the rkey provided may be
invalid and should not be used to lookup an mr.
This patch corrects the driver to ignore the provided rkey if the
reth length is zero and make sure to set the mr to NULL. In read_reply()
if length is zero rxe_recheck_mr() is not called. Warnings are added in
the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.
Signed-off-by: Bob Pearson <rpearsonhpe@xxxxxxxxx>
---
When I applied this change, a testcase in rdma-core failed as shown below:
======================================================================
ERROR: test_qp_ex_rc_flush (tests.test_qpex.QpExTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/rdma-core/tests/test_qpex.py", line 258, in test_qp_ex_rc_flush
raise PyverbsError(f'Unexpected {wc_status_to_str(wcs[0].status)}')
pyverbs.pyverbs_error.PyverbsError: Unexpected Remote access error
----------------------------------------------------------------------
In my opinion, your change makes sense within the range of traditional
RDMA operations, but conflicts with the new RDMA FLUSH operation.
Responder cannot access the target MR because of invalid rkey. The
root cause is written in IBA Annex A19, especially 'oA19-2'.
We thus cannot set qp->resp.rkey to 0 in qp_resp_from_reth().
Do you have anything to say about this? > Li Zhijian
Thanks,
Daisuke Matsuda
I'm confused too, Bob can you point to the section of the spec
that allows the rkey to be zero? It's my understanding that
a zero-length RDMA Read must always check for access, even
though no data is actually fetched. That would not be possible
without an rkey.
Tom.
Tom, Daisuke,
C9-88: For an HCA responder using Reliable Connection service, for
each zero-length RDMA READ or WRITE request, the R_Key shall not be
validated, even if the request includes Immediate data.
Interesting, thanks. I'm pretty certain I recall HCAs not allowing it
in the past. Also note that the iWARP protocol does not have this
exception, and always requires validation.
Further I have seen the pyverbs test suite sending a totally bogus rkey on a zero length rdma read. That was the impetus for me looking at this.
Daisuke has a different issue since flush is a different operation than read or write.
I need to look into what a zero length flush means.
Yes, RDMA Flush has a flag which allows flushing the entire region,
and the length is ignored in that case. oA19-13 always requires
validation of the region.
The IBTA Link Working Group may need to clarify this language.
I suspect the existing text in 9.7.4.1.5 may be correct strictly
speaking, but the RDMA Flush and RDMA Verify operations aren't
mentioned there. And the Annex A19 processing text is different.
I'll bring it up for discussion in an upcoming LWG meeting.
Thanks!
Tom.
