> On Aug 17, 2021, at 9:32 PM, Tom Talpey <tom@xxxxxxxxxx> wrote: > >> The first available moment after cm_send_req where xprtrdma can post >> Receives is when the RDMA core reports the QP connection has been >> established. For reference, I should have included a link to Håkon's recent remarks about this issue: https://lore.kernel.org/linux-rdma/A3297641-63BA-4DF1-886A-3620E2A40BA3@xxxxxxxxxx/ > This has significant implications to upper layer protocols. It > means that the listener cannot safely send the first message on > a new connection. This constrains the upper layer protocol to > be client/server style, with the active-side ULP first to send. The current situation is that the consumer doesn't get control again until it gets RDMA_CM_EVENT_ESTABLISHED, which is a bit late, I agree. We could plumb a callback that is invoked at the "Send REQ" step for the purpose of populating the Receive Queue on the active side. Alternately, xprtrdma's connect worker can explicitly move a new QP to INIT state if it wants to post Receives early. I don't see a hard requirement for rdma_create_qp() to do that for us. > If the CM is changed in the way described here, peer-to-peer > protocols will be rendered unusable, as the passive side cannot > reliably send the first message since the connection will have > no receive posted. The MPA Extensions in RFC6581 discuss this, > and support peer-to-peer connection models with the "A" flag > (section 9.2) enabling passive-first ULPs. At the moment, possibly the only passive-first ULP in the Linux kernel is RDS. > Posting receives and awakening client processing as proposed below > does not close this race. A passive-side-first protocol will have > already begun to send, regardless of this rearrangement. It's an > inherent race and will not interoperate reliably. Sure, but as far as I can tell, RPC-over-RDMA v1 does not have an issue here? > Why change the CM API? The IB spec is not authoritative on this, > and there currently is no bug, right? The minor bug is described in the link above. -- Chuck Lever
