On Wed, May 17, 2023 at 08:56:53AM +0200, Christoph Hellwig wrote: > Just thinking out loud: > > - what if we always way overallocate the swiotlb buffer > - and then mark the second half / two thirds / <pull some number out > of the thin air> slots as used, and make that region available > through a special CMA mechanism as ZONE_MOVABLE (but not allowing > other CMA allocations to dip into it). > > This allows us to have a single slot management for the entire > area, but allow reclaiming from it. We'd probably also need to make > this CMA variant irq safe. I think this could work. It doesn't need to be ZONE_MOVABLE (and we actually need this buffer in ZONE_DMA). But we can introduce a new migrate type, MIGRATE_SWIOTLB, and movable page allocations can use this range. The CMA allocations go to free_list[MIGRATE_CMA], so they won't overlap. One of the downsides is that migrating movable pages still needs a sleep-able context. Another potential confusion is is_swiotlb_buffer() for pages in this range allocated through the normal page allocator. We may need to check the slots as well rather than just the buffer boundaries. (we are actually looking at a MIGRATE_METADATA type for the arm64 memory tagging extension which uses a 3% carveout of the RAM for storing the tags and people want that reused somehow; we have some WIP patches but we'll post them later this summer) > This could still be combined with more aggressive use of per-device > swiotlb area, which is probably a good idea based on some hints. > E.g. device could hint an amount of inflight DMA to the DMA layer, > and if there are addressing limitations and the amout is large enough > that could cause the allocation of a per-device swiotlb area. If we go for one large-ish per-device buffer for specific cases, maybe something similar to the rmem_swiotlb_setup() but which can be dynamically allocated at run-time and may live alongside the default swiotlb. The advantage is that it uses a similar slot tracking to the default swiotlb, no need to invent another. This per-device buffer could also be allocated from the MIGRATE_SWIOTLB range if we make it large enough at boot. It would be seen just a local accelerator for devices that use bouncing frequently or from irq context. -- Catalin
