On Sat, Jul 22, 2017 at 05:32:05PM +0000, Parav Pandit wrote: > > > > Eg > > > > RDMA-W VA=0 Data=1 > > RDMA-R VA=0 > > RDMA-W VA=0 Data=2 > > > > What does the read return? Spec says 1, but it sounds like this relaxed ordering > > could return 2. > > > Spec says Data=1 on RDMA-R only if Fence is set on read operation in Table-76. > Otherwise duplicate read request after executing RDMA-W2 of Data=2 can return Data=2 on read request. Erm, I should have written it like this Initial Condition VA=0 Data = 0 RDMA-W VA=0 Data=1 RDMA-R VA=0 Spec says 1 must be returned, but sounds like this relaxed version could return 0. So RDMA Write -> RDMA Read degrades to a F Similarly, RDMA-W VA=0 Data=1 RDMA-W VA=0 Data=2 SEND Sounds like with the relaxed version the app could see 1 at SEND CQ time. So RDMA-W -> RDMA-W degrades to a F > > Whta about > > > > RDMA-W VA=0 Data=1 > > SEND WITH INVALIDATE VA=0 > > RDMA-W VA=0 Data=2 > > > > Spec says the second RDMA-W must fail, > Right. > > > but it sounds like this relaxed ordering would allow it to happen. > No. Table 76 is followed in this case. > 1st operation Write. > 2nd operation send. > There is implicit fence defined by '#' in Table, which is followed. > So 2nd RDMA-W continue to fail. So, I expect what is happening here is that the SEND RCQ is delayed until the sequence numbers catch up, eg guarenteeing that all packets prior to the SEND have been seen and committed to memory. Which is what table 76 is primarily talking about. However, SEND WITH INVALIDATE is a special cases that impacts the processing of work itself, not just the CPU observation, which is a bit outside what table 76 is talking about. I'd advocate for allowing this to be out of order (but documented as such), as impliclty fencing SEND WITH INVALIDATE is not acceptable for performance and most workloads using that feature do not care about this strict ordering. The requirement is really that by the time the SEND RCQ is seen that the INVALIDATE has taken effect. Atomic are basically similar, sounds like Atomic Op -> RDMA Read should degrade to a F as well. I'd say that is desired as well. The point is we want a definition for this feature that is broad enough to allow future hardware optimization, and not just some some narrow defintion that follows what mlx5 happened to implement. > I completely understand and agree that storage protocols who depend > on RDMA-CM would like to have this. But again, unavailability of > this bit in RDMA-CM is not a blocker for user land apps and let them > start using it. Once RDMA-CM has it, storage kernel ULPs can also > be enabled. If there is no path to get this into the RDMA CM then it is just another vendor feature and it does not belong in the common API. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
