Hi Jason, > -----Original Message----- > From: Jason Gunthorpe [mailto:jgunthorpe@xxxxxxxxxxxxxxxxxxxx] > Sent: Saturday, July 22, 2017 11:10 AM > To: Tom Talpey <tom@xxxxxxxxxx> > Cc: Parav Pandit <parav@xxxxxxxxxxxx>; Bart Van Assche > <Bart.VanAssche@xxxxxxxxxxx>; leon@xxxxxxxxxx; dledford@xxxxxxxxxx; > linux-rdma@xxxxxxxxxxxxxxx; Idan Burstein <idanb@xxxxxxxxxxxx> > Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement > > On Sat, Jul 22, 2017 at 07:51:33AM -0700, Tom Talpey wrote: > > Well, if the broken applications won't use the extension, and the > > existing storage protocols and applications will have to change both > > their implementation and their protocol to use it, who do you envision > > actually doing so? > > The apps are not broken, it appears that this extension allows the receiver HCA > to process inbound packets without seeing earlier packets. > Correct. > This means it can really start doing things out of order, subject only to the ability > of the transmitter to halt sending (eg fence) until acks are seen. > > Since there is no way to predict what is in the data the HCA didn't see, this would > seem to allow quite a lot of out of order execution inside the HCA. > > Eg > > RDMA-W VA=0 Data=1 > RDMA-R VA=0 > RDMA-W VA=0 Data=2 > > What does the read return? Spec says 1, but it sounds like this relaxed ordering > could return 2. > Spec says Data=1 on RDMA-R only if Fence is set on read operation in Table-76. Otherwise duplicate read request after executing RDMA-W2 of Data=2 can return Data=2 on read request. > Whta about > > RDMA-W VA=0 Data=1 > SEND WITH INVALIDATE VA=0 > RDMA-W VA=0 Data=2 > > Spec says the second RDMA-W must fail, Right. > but it sounds like this relaxed ordering would allow it to happen. No. Table 76 is followed in this case. 1st operation Write. 2nd operation send. There is implicit fence defined by '#' in Table, which is followed. So 2nd RDMA-W continue to fail. > > So it cannot be turned on by default.. But a traditional well behaved storage ULP > will only do single access to a single VA and should be able to safely turn > something like this on. > > All of these is independent to observability at the CPU.. > > All this means it needs to be negotiated via RDMA-CM, and making it a common > verbs option would make sense - is there a plan to do that? I completely understand and agree that storage protocols who depend on RDMA-CM would like to have this. But again, unavailability of this bit in RDMA-CM is not a blocker for user land apps and let them start using it. Once RDMA-CM has it, storage kernel ULPs can also be enabled. I am not the right person for commenting on RDMA-CM plan. Idan, Do you know? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
