Hi Jason, > -----Original Message----- > From: Jason Gunthorpe [mailto:jgunthorpe@xxxxxxxxxxxxxxxxxxxx] > Sent: Monday, June 12, 2017 12:15 PM > To: Parav Pandit <parav@xxxxxxxxxxxx> > Cc: Bart Van Assche <Bart.VanAssche@xxxxxxxxxxx>; leon@xxxxxxxxxx; > dledford@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; Idan Burstein > <idanb@xxxxxxxxxxxx> > Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement > > On Mon, Jun 12, 2017 at 05:11:09PM +0000, Parav Pandit wrote: > > > > This is why inverting the flag in the kernel makes much more sense. > > When you say invert the flag, which flag do you mean? > > Capability flag or QP attribute? > > QP attribute. > > > Let's not mix PCIe write ordering with RDMA level out-of-order > > processing. IB spec clearly says, as I reiterated in the > > Documentation/out_of_order.txt - that application should not depend on > > the write data until write_with_imm arrives or subsequent rdma send > > arrives at responder side. If some MPI or other application have > > taken shortcut based on some platform, they have to help themselves. > > This path is not intent to address such issue. > > Exactly - all spec conformant ULPs are compatible with enabling this new > function of mlx5. > This per QP attribute is for read and write both. So responder can receive out-of-order read responses. And HCA's QP need to be told to accept it that way, which by default doesn't. > There is no reason to add a flag at all, except to retain compatibility with > certain non-confomant ULPs. We have none of those in the kernel, so kernel > QPs should always have the flag on. > ULP can be conformant, but it may not be the same HCA on both requester and responder side which has this feature. So requester or responder cannot assume what other side have enabled. Therefore the need to be explicit enable flag. As I explained, by default HCA doesn't process out-of-order packets. User need to tell him to do so. For example adapter is capable of 16 incoming read requests, but when modify_qp() happens, user need to tell how many incoming outstanding read requests it should accept. Same for outgoing read requests. Etc. > Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
