Hi Tom, > -----Original Message----- > From: Tom Talpey [mailto:tom@xxxxxxxxxx] > Sent: Monday, June 12, 2017 4:09 PM > To: Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx>; Parav Pandit > <parav@xxxxxxxxxxxx> > Cc: Bart Van Assche <Bart.VanAssche@xxxxxxxxxxx>; leon@xxxxxxxxxx; > dledford@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; Idan Burstein > <idanb@xxxxxxxxxxxx> > Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement > > On 6/12/2017 12:55 PM, Jason Gunthorpe wrote: > > On Mon, Jun 12, 2017 at 04:53:28PM +0000, Parav Pandit wrote: > >> > >> > >>> From: Jason Gunthorpe [mailto:jgunthorpe@xxxxxxxxxxxxxxxxxxxx] > >>> Sent: Monday, June 12, 2017 11:44 AM > >>> To: Parav Pandit <parav@xxxxxxxxxxxx> > >>> Cc: Bart Van Assche <Bart.VanAssche@xxxxxxxxxxx>; leon@xxxxxxxxxx; > >>> dledford@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; Idan Burstein > >>> <idanb@xxxxxxxxxxxx> > >>> Subject: Re: [PATCH rdma-next 0/3] Support out of order data > >>> placement > >>> > >>> On Mon, Jun 12, 2017 at 04:42:27PM +0000, Parav Pandit wrote: > >>> > >>>>> I would suggest at least using the inverted sense like Bart > >>>>> describes in the kernel - every kernel ULP is safe. > >>> > >>>> I don't see a need to use inverted sense in code. I can surely make > >>> documentation more descriptive as Bart suggested. > >>> > >>> If this is 'better' then it should be on as much as possible, and I > >>> certianly don't want to see kernel ULPs query caps and other > >>> pointless things when they already, necessarily, deal with out of order. > >> > >> Sure. Kernel ULPs and any other user ULPs can skip query caps. > > > > No, they can't because only mlx5 accepts the new flag. > > > > This is why inverting the flag in the kernel makes much more sense. > > Absolutely. All iWARP adapters support multiple RDMA Writes in flight, and > their placement is not ordered. It's a quirk of the MLX devices, and in turn > their implementation of the IB protocol, that they place writes in-order. I'm > happy to see Mellanox become aware of this, it limits write performance on > networks with any significant latencies. > > I agree with Jason, the bit should be 1 by default, if defined as you propose. > Out-of-order is the norm, not the exception, for ULPs. > Honestly, I think you should perhaps consider making it the default on your > devices, and allowing only MLX-aware ULPs to turn it off. > There can be cases in deployment where responder has support for receiving out-of-order, but requester doesn't. Read responses arriving out of order for different messages at requester side would break. In other case if its supported on requester side, but not supported on responder side, would result into more retransmission because responder cannot handle them and triggers PSN sequence error. Requester and responder both needs to know when they enable out-of-order on a given QP. So we shouldn’t enable it by default on all QPs. > The Windows NDK API has a similar concept, by the way, it flips your > meaning and defaults to 0. Also, it's not settable by the ULP. The flag isn't > used for much, as a result. > > NDK_ADAPTER_FLAG_IN_ORDER_DMA_SUPPORTED > > https://msdn.microsoft.com/en- > us/library/windows/hardware/hh439851(v=vs.85).aspx > > > Tom. ��.n��������+%������w��{.n�����{���fk��ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
