Re: [PATCH RFC] xprtrdma: Move initial Receive posting

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




> On Aug 17, 2021, at 9:32 PM, Tom Talpey <tom@xxxxxxxxxx> wrote:
> 
>> The first available moment after cm_send_req where xprtrdma can post
>> Receives is when the RDMA core reports the QP connection has been
>> established.

For reference, I should have included a link to Håkon's recent
remarks about this issue:

https://lore.kernel.org/linux-rdma/A3297641-63BA-4DF1-886A-3620E2A40BA3@xxxxxxxxxx/


> This has significant implications to upper layer protocols. It
> means that the listener cannot safely send the first message on
> a new connection. This constrains the upper layer protocol to
> be client/server style, with the active-side ULP first to send.

The current situation is that the consumer doesn't get control
again until it gets RDMA_CM_EVENT_ESTABLISHED, which is a bit
late, I agree. We could plumb a callback that is invoked at
the "Send REQ" step for the purpose of populating the Receive
Queue on the active side.

Alternately, xprtrdma's connect worker can explicitly move a
new QP to INIT state if it wants to post Receives early. I
don't see a hard requirement for rdma_create_qp() to do that
for us.


> If the CM is changed in the way described here, peer-to-peer
> protocols will be rendered unusable, as the passive side cannot
> reliably send the first message since the connection will have
> no receive posted. The MPA Extensions in RFC6581 discuss this,
> and support peer-to-peer connection models with the "A" flag
> (section 9.2) enabling passive-first ULPs.

At the moment, possibly the only passive-first ULP in the
Linux kernel is RDS.


> Posting receives and awakening client processing as proposed below
> does not close this race. A passive-side-first protocol will have
> already begun to send, regardless of this rearrangement. It's an
> inherent race and will not interoperate reliably.

Sure, but as far as I can tell, RPC-over-RDMA v1 does not
have an issue here?


> Why change the CM API? The IB spec is not authoritative on this,
> and there currently is no bug, right?

The minor bug is described in the link above.


--
Chuck Lever







[Index of Archives]     [Linux Filesystem Development]     [Linux USB Development]     [Linux Media Development]     [Video for Linux]     [Linux NILFS]     [Linux Audio Users]     [Yosemite Info]     [Linux SCSI]

  Powered by Linux