On Thu, Dec 01, 2022 at 11:11:10AM -0600, Bob Pearson wrote:
> On 12/1/22 09:39, Jason Gunthorpe wrote:
> > On Thu, Dec 01, 2022 at 09:38:10AM -0600, Bob Pearson wrote:
> >
> >> Further, looking at ipoib as an example, it builds sge lists using the lkey from get_dma_mr()
> >> and sets the sge->addr to a kernel virtual memory address after previously calling
> >> ib_dma_map_single() so the addresses are mapped for dma access and visible before use.
> >> They are unmapped after the read/write operation completes. What is the point of kmapping
> >> the address after dma mapping them?
> >
> > Because not everything is ipoib, and things like block will map sgls
> > with struct pages, not kva.
> >
> > Jason
>
> OK it's working now but there is a bug in your rxe_mr_fill_pages_from_sgt() routine.
> You have a
>
> if (xas_xa_index && WARN_ON(sg_iter.sg_pgoffset % PAGE_SIZE)) {...}
>
> which seems to assume that sg_pgoffset contains the byte offset in the current page.
> But looking at __sg_page_iter_next() it appears that it is the number of pages offset
> in the current sg entry which results in a splat when I run
> ib_send_bw.
The pgoffset depends on the creator of the SGL, I guess something must
be pointing the SGL ad a folio and using the sg_pgoffset as a sub
folio index?
If so then the calculation of the kmap has to be adjusted to extract
the number of pages from the sg_pgoffset
Jason
