> -----Original Message----- > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma- > owner@xxxxxxxxxxxxxxx] On Behalf Of Jason Gunthorpe > Sent: Thursday, November 02, 2017 10:13 AM > To: Sagi Grimberg > Cc: idanb; Max Gurtovoy; linux-rdma@xxxxxxxxxxxxxxx; linux- > nvme@xxxxxxxxxxxxxxxxxxx; Christoph Hellwig > Subject: Re: [PATCH 0/3] Fix request completion holes > > On Thu, Nov 02, 2017 at 10:06:30AM +0200, Sagi Grimberg wrote: > > > > >>the default mode utilize remote invalidation, so no, its not valid. > > > > > >Well, usually the ULP design should allow some things to be reaped > > >async, and latency senstive things to be sync. > > > > > >Eg if you have a SEND using a local buffer with a RKEY, then you'd > > >declare the RKEY data completed when SEND_WITH_INVALIDATE is returned. > > > > > >Recycling the lkey command buffer is an async process and can wait > > >unsignaled until something signalled comes to push its completion > > >through. > > > > Not when using inline data with the send, which is the main issue > > here. if we inline data to the command, we will use the local > > dma lkey, which does not even have a local invalidate following it. > > Does nvme use inline data send and RKEY transfer in the same SEND? > Then it would need to signal the SEND if remote invalidate is used, > otherwise it only needs to signal the local invalidate for the RKEY.. > > > >Local invalidate is defined to always be ordered by the spec, so it > > >is required to guarentee that the SEND is completed. > > > So local invalidate completion is guaranteed to come after all the > > completions prior to it in the send queue? > > IBA spec says so.. > iWARP spec too. This is in regard to completion ordering though. The local invalidate send WR must have the IB_SEND_FENCE flag set if you want it to only be executed after all prior send WRs are executed. Either way, the completions are always inserted into the cq in-sq-submission-order and a signaled completion implies completions of all prior unsignaled WRs. Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
