On Tue, Sep 07, 2021 at 11:35:17AM -0500, Bob Pearson wrote: > Interesting. But if that is the case the bigger problem is the ib_map_mr_sg() call which updates the > mapping. rxe definitely does look at the mr->rkey value but we could fix that. It also looks at the > mapping which is updated by ib_map_mr_sg(). My impression is that HW also uses this mapping or does > HW also copy all the FMRs into SRAM? Yes, real HW has a copy of the DMA list. The sg in the mr struct is for CPU use only. It is not OK to use the CPU SG list inside the MR for DMA by HW, it has to be synchronized with the WR. > There seems to be an assumption that users will be looking at CQE. Yes, the kernel has to be driven by CQE, not only for data transfer but the DMA unmap of the SGL cannot be until after the invalidation CQE is observed. Ie the CPU should have two DMA lists active during the invalidation cycle. Jason
