From: Christoph Hellwig <hch@xxxxxxxxxxxxx> Date: Tue, 18 Apr 2023 22:16:53 -0700 > On Mon, Apr 17, 2023 at 11:19:47PM -0700, Jakub Kicinski wrote: >>> You can't just do dma mapping outside the driver, because there are >>> drivers that do not require DMA mapping at all. virtio is an example, >>> but all the classic s390 drivers and some other odd virtualization >>> ones are others. >> >> What bus are the classic s390 on (in terms of the device model)? > > I think most of them are based on struct ccw_device, but I'll let the > s390 maintainers fill in. > > Another interesting case that isn't really relevant for your networking > guys, but that caused as problems is RDMA. For hardware RDMA devices > it wants the ULPs to DMA map, but it turns out we have various software > drivers that map to network drivers that do their own DMA mapping > at a much lower layer and after potentially splitting packets or > even mangling them. > >> >>>> I don't think it's reasonable to be bubbling up custom per-subsystem >>>> DMA ops into all of them for the sake of virtio. >>> >>> dma addresses and thus dma mappings are completely driver specific. >>> Upper layers have no business looking at them. Here it's not an "upper layer". XSk core doesn't look at them or pass them between several drivers. It maps DMA solely via the struct device passed from the driver and then just gets-sets addresses for this driver only. Just like Page Pool does for regular Rx buffers. This got moved to the XSk core to not repeat the same code pattern in each driver. >> >> Damn, that's unfortunate. Thinking aloud -- that means that if we want >> to continue to pull memory management out of networking drivers to >> improve it for all, cross-optimize with the rest of the stack and >> allow various upcoming forms of zero copy -- then we need to add an >> equivalent of dma_ops and DMA API locally in networking? Managing DMA addresses is totally fine as long as you don't try to pass mapped addresses between different drivers :D Page Pool already does that and I don't see a problem in that in general. > > Can you explain what the actual use case is? > >>From the original patchset I suspect it is dma mapping something very > long term and then maybe doing syncs on it as needed? As I mentioned, XSk provides some handy wrappers to map DMA for drivers. Previously, XSk was supported by real hardware drivers only, but here the developer tries to add support to virtio-net. I suspect he needs to use DMA mapping functions different from which the regular driver use. So this is far from dma_map_ops, the author picked wrong name :D And correct, for XSk we map one big piece of memory only once and then reuse it for buffers, no inflight map/unmap on hotpath (only syncs when needed). So this mapping is longterm and is stored in XSk core structure assigned to the driver which this mapping was done for. I think Jakub thinks of something similar, but for the "regular" Rx/Tx, not only XDP sockets :) Thanks, Olek
