> > > iWARP spec too. This is in regard to completion ordering though. The local > > invalidate send WR must have the IB_SEND_FENCE flag set if you want it to only > > be executed after all prior send WRs are executed. > > Oh right, yes, you need to have local invalidate fence to guarentee > that, but usually a local operation will not refer to a RKEY, so > execution ordering wont matter. > > But this is a good general point, doesn't a ULP need to set FENCE on > SEND, eg: > > RDMA READ (rkey) > RDMA READ (rkey) > SEND w/ FENCE (tell remote to invalidate rkey) > > Otherwise IBA Table 79 says RDMA READ can pass SEND and we have a > situation where the rkey has become invalidated when the remote is > still trying to use it. > > IBA actually explicitly calls this out (pg 538): > > RDMA Read operations may be executed at the target after subse- > quent Send and Invalidate operation already performed the invalida- > tion at the target. That may cause the RDMA Read operation to fail. > Setting the Fence Indicator on the subsequent operations guarantees > that the RDMA Read will fully complete before the invalidation takes > place. > > None of our ULPs use IB_SEND_FENCE - so this is a bug! > I think the nvmf target doesn't post the SEND_W_INVALIDATE (NVMF CQE) until the RDMA READ(s) (and back end BIO submission) complete. Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
