Currently the only efficient way to map a complex memory description through
the DMA API is by using the scatterlist APIs. The SG APIs are unique in that
they efficiently combine the two fundamental operations of sizing and allocating
a large IOVA window from the IOMMU and processing all the per-address
swiotlb/flushing/p2p/map details.
This uniqueness has been a long standing pain point as the scatterlist API
is mandatory, but expensive to use. It prevents any kind of optimization or
feature improvement (such as avoiding struct page for P2P) due to the
impossibility of improving the scatterlist.
Several approaches have been explored to expand the DMA API with additional
scatterlist-like structures (BIO, rlist), instead split up the DMA API
to allow callers to bring their own data structure [1].
The API is split up into parts:
- Allocate IOVA space:
To do any pre-allocation required. This is done based on the caller
supplying some details about how much IOMMU address space it would need
in worst case.
- Map and unmap relevant structures to pre-allocated IOVA space:
Perform the actual mapping into the pre-allocated IOVA. This is very
similar to dma_map_page().
In this topic, I would like to present existing DMA API abuses and present path
to move forward with the help of new DMA API.
In this topic I will briefly present the new API and have a forward
looking discussion about how such a significant change is expected to
impact the kernel.
Particularly how this API fits with Matthew's phyr sketch, and where
we might see this go in the storage layer (David's proposal for iter [2]).
In addition, we will discuss the roadmap of converting DMABUF and RDMA to
SG-free world (Jasons's vision [3]):
1) The new DMA API lands
2) We improve the new DMA API to be fully struct page free, including
setting up P2P
3) VFIO provides a dmbuf exporter using the new DMA API's P2P
support. We'd have to continue with the scatterlist hacks for now.
VFIO would be a move_notify exporter. This should work with RDMA
4) RDMA works to drop scatterlist from its internal flows and use the
new DMA API instead.
5) VFIO/RDMA implement a new non-scatterlist DMABUF op to
demonstrate the post-scatterlist world and deprecate the scatterlist
hacks.
6) We add revoke semantics to dmabuf, and VFIO/RDMA implements them
7) iommufd can import a pinnable revokable dmabuf using CPU pfns
through the non-scatterlist op.
8) Relevant GPU drivers implement the non-scatterlist op and RDMA
removes support for the deprecated scatterlist hacks.
[1] https://lore.kernel.org/all/cover.1737106761.git.leon@xxxxxxxxxx
[2] https://lore.kernel.org/all/886959.1737148612@xxxxxxxxxxxxxxxxxxxxxx/
[3] https://lore.kernel.org/all/20250120175901.GP5556@xxxxxxxxxx
----------------------------------------------------------------------------
LWN coverage:
Dancing the DMA two-step - https://lwn.net/Articles/997563/
----------------------------------------------------------------------------
Thanks
