> On Mar 10, 2016, at 10:31 AM, Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> wrote: > >>> On Mar 10, 2016, at 10:04 AM, Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> >> wrote: >>> >>>>>> Moving the QP into error state right after with rdma_disconnect >>>>>> you are not sure that none of the subset of the invalidations >>>>>> that _were_ posted completed and you get the corresponding MRs >>>>>> in a bogus state... >>>>> >>>>> Moving the QP to error state and then draining the CQs means >>>>> that all LOCAL_INV WRs that managed to get posted will get >>>>> completed or flushed. That's already handled today. >>>>> >>>>> It's the WRs that didn't get posted that I'm worried about >>>>> in this patch. >>>>> >>>>> Are there RDMA consumers in the kernel that use that third >>>>> argument to recover when LOCAL_INV WRs cannot be posted? >>>> >>>> None :) >>>> >>>>>>> I suppose I could reset these MRs instead (that is, >>>>>>> pass them to ib_dereg_mr). >>>>>> >>>>>> Or, just wait for a completion for those that were posted >>>>>> and then all the MRs are in a consistent state. >>>>> >>>>> When a LOCAL_INV completes with IB_WC_SUCCESS, the associated >>>>> MR is in a known state (ie, invalid). >>>>> >>>>> The WRs that flush mean the associated MRs are not in a known >>>>> state. Sometimes the MR state is different than the hardware >>>>> state, for example. Trying to do anything with one of these >>>>> inconsistent MRs results in IB_WC_BIND_MW_ERR until the thing >>>>> is deregistered. >>>> >>>> Correct. >>>> >>> >>> It is legal to invalidate an MR that is not in the valid state. So you > don't >>> have to deregister it, you can assume it is valid and post another LINV WR. >> >> I've tried that. Once the MR is inconsistent, even LOCAL_INV >> does not work. >> > > Maybe IB Verbs don't mandate that invalidating an invalid MR must be allowed? > (looking at the verbs spec now). If the MR is truly invalid, then there is no issue, and the second LOCAL_INV completes successfully. The problem is after a flushed LOCAL_INV, the MR state sometimes does not match the hardware state. The MR is neither registered or invalid. A flushed LOCAL_INV tells you nothing more than that the LOCAL_INV didn't complete. The MR state at that point is unknown. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
