On 9/7/21 11:39 AM, Jason Gunthorpe wrote: > On Tue, Sep 07, 2021 at 11:35:17AM -0500, Bob Pearson wrote: > >> Interesting. But if that is the case the bigger problem is the ib_map_mr_sg() call which updates the >> mapping. rxe definitely does look at the mr->rkey value but we could fix that. It also looks at the >> mapping which is updated by ib_map_mr_sg(). My impression is that HW also uses this mapping or does >> HW also copy all the FMRs into SRAM? > > Yes, real HW has a copy of the DMA list. The sg in the mr struct is > for CPU use only. > > It is not OK to use the CPU SG list inside the MR for DMA by HW, it > has to be synchronized with the WR. > >> There seems to be an assumption that users will be looking at CQE. > > Yes, the kernel has to be driven by CQE, not only for data transfer > but the DMA unmap of the SGL cannot be until after the invalidation > CQE is observed. > > Ie the CPU should have two DMA lists active during the invalidation > cycle. > > Jason > OK. Not 100% sure what that implies for SRP. SRP does *not* look at the CQE for invalidate and register WQEs. I can fix the rkey and DMA list semantics, making a copy of the list which is installed by the register WQE.
