On Wed, Oct 28, 2015 at 05:30:17PM -0400, Chuck Lever wrote: > IBTA spec states: > > > MW access operations (i.e. RDMA Write, RDMA Reads, and Atom- > > ics) are only allowed if the Type 2B MW is in the Valid state and the > > QP Number (QPN) and PD of the QP performing the MW access op- > > eration matches the QPN and PD associated with the Bound Type 2B > > MW. > > Once the QP is out of RTS, there can be no incoming RDMA > requests that match the R_key, QPN, PD tuple. I think you > are saying that the QP state change has the same problem > as not waiting for an invalidation to complete. MW (Memory Window) is something different from a MR. MR's do not match on the QPN. > > If there was one PD per QP then the above would be true, since the MR > > is linked to the PD. > > There is a per-connection struct rpcrdma_ia that contains > both a PD and a QP. Therefore there is one PD and only one > QP (on the client) per connection. Oh, that is great then > > FWIW, the same is true on the send side too, if the RPC had send > > buffers and gets canceled, you have to block until a CQ linked to that > > send is seen. > > By “you have to block” you mean the send buffer cannot be reused > until the Send WR is known to have completed, and new Send WRs > cannot be posted until it is known that enough send queue resources > are available. Yes > I’m not certain we are careful to ensure > the hardware has truly relinquished the send buffer before it is > made available for re-use. A known issue. This is the issue I was thinking of, yes. Ideally the CPU would not touch the send buffer until the HW is done with it under any situation. This is less serious than having a rouge writable R_Key however. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
