On Tue, Jun 20, 2017 at 01:01:39PM -0400, Chuck Lever wrote: > >> Shouldn't this be protected somehow by the device? > >> Can someone explain why the above cannot happen? Jason? Liran? Anyone? > >> Say host register MR (a) and send (1) from that MR to a target, > >> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE > >> on MR (a) and the host HCA process it, then host HCA timeout on send (1) > >> so it retries, but ehh, its already invalidated. I'm not sure I understand the example.. but... If you pass a MR key to a send, then that MR must remain valid until the send completion is implied by an observation on the CQ. The HCA is free to re-execute the SEND against the MR at any time up until the completion reaches the CQ. As I've explained before, a ULP must not use 'implied completion', eg a receive that could only have happened if the far side got the send. In particular this means it cannot use an incoming SEND_INV/etc to invalidate an MR associated with a local SEND, as that is a form of 'implied completion' For sanity a MR associated with a local send should not be remote accessible at all, and shouldn't even have a 'rkey', just a 'lkey'. Similarly, you cannot use a MR with SEND and remote access sanely, as the far end could corrupt or invalidate the MR while the local HCA is still using it. > So on occasion there is a Remote Access Error. That would > trigger connection loss, and the retransmitted Send request > is discarded (if there was externally exposed memory involved > with the original transaction that is now invalid). Once you get a connection loss I would think the state of all the MRs need to be resync'd. Running through the CQ should indicate which ones are invalidate and which ones are still good. > NFS has a duplicate replay cache. If it sees a repeated RPC > XID it will send a cached reply. I guess the trick there is > to squelch remote invalidation for such retransmits to avoid > spurious Remote Access Errors. Should be rare, though. .. and because of the above if a RPC is re-issued it must be re-issued with corrected, now-valid rkeys, and the sender must somehow detect that the far side dropped it for replay and tear down the MRs. > RPC-over-RDMA uses persistent registration for its inline > buffers. The problem there is avoiding buffer reuse to soon. > Otherwise a garbled inline message is presented on retransmit. > Those would probably not be caught by the DRC. We've had this discussion on the list before. You can *never* re-use a SEND, or RDMA WRITE buffer until you observe the HCA is done with it via a CQ poll. > But the real problem is preventing retransmitted Sends from > causing a ULP request to be executed multiple times. IB RC guarentees single delivery for SEND, so that doesn't seem possible unless the ULP re-transmits the SEND on a new QP. > > Signalling all send completions and also finishing I/Os only after > > we got them will add latency, and that sucks... There is no choice, you *MUST* see the send completion before reclamining any resources associated with the send. Only the completion guarentees that the HCA will not resend the packet or otherwise continue to use the resources. > With FRWR, won't subsequent WRs be delayed until the HCA is > done with the Send? I don't think a signal is necessary in > every case. Send Queue accounting currently relies on that. No. The SQ side is asynchronous to the CQ side, the HCA will pipeline send packets on the wire up to some internal limit. Only the local state changed by FRWR related op codes happens sequentially with other SQ work. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
