On Mon, Jan 10, 2022 at 08:41:26PM -0400, Jason Gunthorpe wrote: > > Finally, it may be possible to stop using scatterlist to describe the > > input to the DMA-mapping operation. We may be able to get struct > > scatterlist down to just dma_address and dma_length, with chaining > > handled through an enclosing struct. > > Can you talk about this some more? IMHO one of the key properties of > the scatterlist is that it can hold huge amounts of pages without > having to do any kind of special allocation due to the chaining. > > The same will be true of the phyr idea right? No special allocations as in no vmalloc? The chaining still has to allocate memory using a mempool. Anyway, to explain my idea which is very similar but not identical to the one willy has: - on the input side to dma mapping the bio_vecs (or phyrs) are chained as bios or whatever the containing structure is. These already exist and have infrastructure at least in the block layer - on the output side I plan for two options: 1) we have a sane IOMMU and everyting will be coalesced into a single dma_range. This requires setting the block layer merge boundary to match the IOMMU page size, but that is a very good thing to do anyway. 2) we have no IOMMU (or a weird one) and get one output dma_range per input bio_vec. We'd eithe have to support chaining or use vmalloc or huge numbers of entries. > If you limit to that scenario then we can be more optimal because > things like byte granular offsets and size in the interior pages don't > need to exist. Every interior chunk is always aligned to its order and > we only need to record the order. The block layer does not small offsets. Direct I/O can often be 512 byte aligned, and some other passthrough commands can have even smaller alignment, although I don't think we ever go below 4-byte alignment anywhere in the block layer. > IMHO storage density here is quite important, we end up having to keep > this stuff around for a long time. If we play these tricks it won't be general purpose enough to get rid of the existing scatterlist usage.
