Hi Sean- On Jun 20, 2014, at 5:17 PM, Hefty, Sean <sean.hefty@xxxxxxxxx> wrote: >> During a remote transport disconnect, the QP leaves RTS. >> >> xprtrdma deals with this in a separate transport connect worker process, >> where it creates a new id and qp, and replaces the existing id and qp. >> >> Unfortunately there are parts of xprtrdma (namely FRMR deregistration) >> that are not easy to serialize with this reconnect logic. >> >> Re-using the QP would mean no serialization would be needed between >> transport reconnect and FRMR deregistration. >> >> If QP re-use is not supported, though, it's not worth considering any >> further. > > It may be possible to reuse the QP, just not the rdma_cm_id without additional code changes. Reuse of the rdma_cm_id may also require changes in the underlying IB/iWarp CMs. Steve Wise is helping me with a particular issue where QP re-use might be helpful. When an RPC/RDMA transport connection is dropped (for example, the NFS server crashes), xprtrdma destroys the transport's QP and creates a new one for the next connection. We’re not quite sure what IB_WC_WR_FLUSH_ERR means in that instance. Our theory is there is a gap when the old QP is destroyed: 1. If the HW reports a successful WR completion but the QP no longer exists, the provider substitutes an IB_WC_WR_FLUSH_ERR completion 2. If the WR is dropped before the HW even saw it, the provider inserts an IB_WC_WR_FLUSH_ERR completion So if xprtrdma is trying to submit a FAST_REG_MR WR and the completion gets flushed, xprtrdma has no way to know whether the rkey was bumped in the adapter. Thus it has no certainty which rkey to use to invalidate that FRMR. I was idly wondering whether re-using the QP during connection loss would provide a guarantee that xprtrdma would never see case 1 above. Then IB_WC_WR_FLUSH_ERR on a FAST_REG_MR WR would be a more certain indication that the HW still has the old rkey. I suppose that xprtrdma can “hang onto” the QP without re-using it by simply not destroying it until all WRs scheduled on the old QP are completed. Is reference counting the QP the usual design pattern to deal with this case? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
