On Jul 24, 2015, at 1:46 PM, Chuck Lever <chuck.lever@xxxxxxxxxx> wrote: > On Jul 24, 2015, at 12:26 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > >> On Fri, Jul 24, 2015 at 10:36:07AM -0400, Chuck Lever wrote: >> >>> Unfinished, but operational: >>> >>> http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=shortlog;h=refs/heads/nfs-rdma-future >> >> Nice.. >> >> Can you spend some time and reflect on how some of this could be >> lowered into the core code? > > The point of the prototype is to start thinking about this with > actual data. :-) So I’m with you. > > >> The FMR and FRWR side have many >> similarities now.. IMO ib_unmap_fmr is a very different animal from LOCAL_INV WR. ib_unmap_fmr is synchronous, provides no ordering guarantees with send queue operations, and does not depend on a connected QP to be available. You could emulate asynchronicity with a work queue but that still does not provide SQ ordering. There are few if any failure modes for ib_unmap_fmr. LOCAL_INV WR is asynchronous, provides strong ordering with other send queue operations, but does require a non-NULL QP in RTS to work. The failure modes are complex: without a QP in RTS, the post_send fails. If the QP leaves RTS while LOCAL_INV is in flight, the LINV flushes. MRs can be left in a state where the MR's rkey is not in sync with the HW, in which case a synchronous operation may be required to recover the MR. These are the reasons I elected to employ a synchronous invalidation model in the RPC reply handler. This model can be made to work adequately for both FMR and FRWR, provides proper DMA unmap ordering guarantees for both, and hides wonky transport disconnect recovery mechanics. The only downside is the performance cost. A generic MR invalidation API that buries underlying verb activity and guarantees proper DMA unmap ordering I think would have to be synchronous. In the long run, two things will change: first, FMR will eventually be deprecated; and second, ULPs will likely adopt SEND_WITH_INV. The complexion of MR invalidation could be vastly different in a few years: handled entirely by the target-side, and only verified by the initiator. Verification doesn't need to sleep, and the slow path (the target failed to invalidate) can be deferred. All that would be necessary at that point would be a synchronous invalidation API (synchronous in the sense that the invalidate is complete if the API returns without error). -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
