On Wed, Nov 04, 2020 at 03:01:08PM +0100, Christoph Hellwig wrote:
> > Sigh. I think the proper fix is to replace addr/length with a
> > scatterlist pointer in the struct ib_sge, then have SW drivers
> > directly use the page pointer properly.
>
> The proper fix is to move the DMA mapping into the RDMA core, yes.
> And as you said it will be hard. But I don't think scatterlists
> are the right interface. IMHO we can keep re-use the existing
> struct ib_sge:
>
> struct ib_ge {
> u64 addr;
> u32 length;
> u32 lkey;
> };
Gah, right, this is all about local_dma_lkey..
> with the difference that if lkey is not a MR, addr is the physical
> address of the memory, not a dma_addr_t or virtual address.
It could work, I think a resonable ULP API would be to have some
rdma_fill_ib_sge_from_sgl()
rdma_map_sge_single()
etc etc
ie instead of wrappering the DMA API as-is we have a new API that
directly builds the ib_sge. It always fills the local_dma_lkey from
the pd, so it knows it is doing DMA from local kernel memory.
Logically SW devices then have a local_dma_lkey MR that has an IOVA of
the CPU physical address space, not the DMA address space as HW
devices have. The ib_sge builders can know this detail and fill in
addr from either a cpu phyical or a dma map.
The SW device has to translate the addr/length in CPU space to
something else. It actually makes reasonable sense architecturally.
This is actually much less horrible than I thought..
Convert all ULPs to one of these new APIs, searching for
local_dma_lkey will find all places. This will replace a whole lot of
calls to ib DMA API wrapper functions. Searching for local_dma_lkey
will find all users. Drivers already work with sge.addr == CPU
address, so no driver change
Then to kill the dma_ops wrappers the remaining users should all be
connected to map_mr_sg. In this case we want a no-op dma map and fix
the three map_mr_sg's to use the page side of the sgl, not the DMA
side
Not as horrible as I imagined at first, actually..
Jason
