On Wed, 19 Apr 2023 15:14:48 +0200, Alexander Lobakin <aleksander.lobakin@xxxxxxxxx> wrote: > From: Christoph Hellwig <hch@xxxxxxxxxxxxx> > Date: Tue, 18 Apr 2023 22:16:53 -0700 > > > On Mon, Apr 17, 2023 at 11:19:47PM -0700, Jakub Kicinski wrote: > >>> You can't just do dma mapping outside the driver, because there are > >>> drivers that do not require DMA mapping at all. virtio is an example, > >>> but all the classic s390 drivers and some other odd virtualization > >>> ones are others. > >> > >> What bus are the classic s390 on (in terms of the device model)? > > > > I think most of them are based on struct ccw_device, but I'll let the > > s390 maintainers fill in. > > > > Another interesting case that isn't really relevant for your networking > > guys, but that caused as problems is RDMA. For hardware RDMA devices > > it wants the ULPs to DMA map, but it turns out we have various software > > drivers that map to network drivers that do their own DMA mapping > > at a much lower layer and after potentially splitting packets or > > even mangling them. > > > >> > >>>> I don't think it's reasonable to be bubbling up custom per-subsystem > >>>> DMA ops into all of them for the sake of virtio. > >>> > >>> dma addresses and thus dma mappings are completely driver specific. > >>> Upper layers have no business looking at them. > > Here it's not an "upper layer". XSk core doesn't look at them or pass > them between several drivers. It maps DMA solely via the struct device > passed from the driver and then just gets-sets addresses for this driver > only. Just like Page Pool does for regular Rx buffers. This got moved to > the XSk core to not repeat the same code pattern in each driver. > > >> > >> Damn, that's unfortunate. Thinking aloud -- that means that if we want > >> to continue to pull memory management out of networking drivers to > >> improve it for all, cross-optimize with the rest of the stack and > >> allow various upcoming forms of zero copy -- then we need to add an > >> equivalent of dma_ops and DMA API locally in networking? > > Managing DMA addresses is totally fine as long as you don't try to pass > mapped addresses between different drivers :D Page Pool already does > that and I don't see a problem in that in general. > > > > > Can you explain what the actual use case is? > > > >>From the original patchset I suspect it is dma mapping something very > > long term and then maybe doing syncs on it as needed? > > As I mentioned, XSk provides some handy wrappers to map DMA for drivers. > Previously, XSk was supported by real hardware drivers only, but here > the developer tries to add support to virtio-net. I suspect he needs to > use DMA mapping functions different from which the regular driver use. > So this is far from dma_map_ops, the author picked wrong name :D Yes, dma_ops caused some misunderstandings, which is indeed a wrong name. Thanks. > And correct, for XSk we map one big piece of memory only once and then > reuse it for buffers, no inflight map/unmap on hotpath (only syncs when > needed). So this mapping is longterm and is stored in XSk core structure > assigned to the driver which this mapping was done for. > I think Jakub thinks of something similar, but for the "regular" Rx/Tx, > not only XDP sockets :) > > Thanks, > Olek
