Le mar., mars 29 2022 at 16:07:21 +0200, Daniel Vetter <daniel@xxxxxxxx> a écrit :
On Tue, Mar 29, 2022 at 10:47:23AM +0100, Paul Cercueil wrote:Hi Daniel,Le mar., mars 29 2022 at 10:54:43 +0200, Daniel Vetter <daniel@xxxxxxxx> aécrit : > On Mon, Feb 07, 2022 at 01:01:40PM +0000, Paul Cercueil wrote: > > Document the new DMABUF based API. > >> > v2: - Explicitly state that the new interface is optional and is> > not implemented by all drivers.> > - The IOCTLs can now only be called on the buffer FD returned by> > IIO_BUFFER_GET_FD_IOCTL.> > - Move the page up a bit in the index since it is core stuff> > and not > > driver-specific. > > > > Signed-off-by: Paul Cercueil <paul@xxxxxxxxxxxxxxx> > > --- > > Documentation/driver-api/dma-buf.rst | 2 + > > Documentation/iio/dmabuf_api.rst | 94 > > ++++++++++++++++++++++++++++ > > Documentation/iio/index.rst | 2 + > > 3 files changed, 98 insertions(+) > > create mode 100644 Documentation/iio/dmabuf_api.rst > > > > diff --git a/Documentation/driver-api/dma-buf.rst > > b/Documentation/driver-api/dma-buf.rst > > index 2cd7db82d9fe..d3c9b58d2706 100644 > > --- a/Documentation/driver-api/dma-buf.rst > > +++ b/Documentation/driver-api/dma-buf.rst > > @@ -1,3 +1,5 @@ > > +.. _dma-buf: > > + > > Buffer Sharing and Synchronization > > ================================== > > > > diff --git a/Documentation/iio/dmabuf_api.rst > > b/Documentation/iio/dmabuf_api.rst > > new file mode 100644 > > index 000000000000..43bb2c1b9fdc > > --- /dev/null > > +++ b/Documentation/iio/dmabuf_api.rst > > @@ -0,0 +1,94 @@ > > +=================================== > > +High-speed DMABUF interface for IIO > > +=================================== > > + > > +1. Overview > > +=========== > > +> > +The Industrial I/O subsystem supports access to buffers through a> > file-based> > +interface, with read() and write() access calls through the IIO> > device's dev > > +node. > > + > > +It additionally supports a DMABUF based interface, where the > > userspace> > +application can allocate and append DMABUF objects to the buffer's> > queue. > > +This interface is however optional and is not available in all > > drivers. > > + > > +The advantage of this DMABUF based interface vs. the read()> > +interface, is that it avoids an extra copy of the data between the > > +kernel and userspace. This is particularly useful for high-speed > > +devices which produce several megabytes or even gigabytes of data> > per > > +second. > > +> > +The data in this DMABUF interface is managed at the granularity of > > +DMABUF objects. Reducing the granularity from byte level to block> > level > > +is done to reduce the userspace-kernelspace synchronization > > overhead> > +since performing syscalls for each byte at a few Mbps is just not> > +feasible. > > + > > +This of course leads to a slightly increased latency. For this > > reason an> > +application can choose the size of the DMABUFs as well as how many> > it> > +allocates. E.g. two DMABUFs would be a traditional double buffering> > +scheme. But using a higher number might be necessary to avoid > > +underflow/overflow situations in the presence of scheduling > > latencies. > > So this reads a lot like reinventing io-uring with pre-registered > O_DIRECT > memory ranges. Except it's using dma-buf and hand-rolling a lot of > pieces > instead of io-uring and O_DIRECT.I don't see how io_uring would help us. It's an async I/O framework, does it allow us to access a kernel buffer without copying the data? Does it allowus to zero-copy the data to a network interface?With networking, do you mean rdma, or some other kind of networking?Anything else than rdma doesn't support dma-buf, and I don't think it will likely ever do so. Similar it's really tricky to glue dma-buf support intothe block layer.
By networking I mean standard sockets. If I'm not mistaken, Jonathan Lemon's work on zctap was to add dma-buf import/export support to standard sockets.
Wrt io_uring, yes it's async, but that's not the point. The point is thatwith io_uring you pre-register ranges for reads and writes to target,which in combination with O_DIRECT, makes it effectively (and efficient!) zero-copy. Plus it has full integration with both networking and normalfile io, which dma-buf just doesn't have. Like you _cannot_ do zero copy from a dma-buf into a normal file. You absolutely can do the same with io_uring.
I believe io_uring does zero-copy the same way as splice(), by duplicating/moving pages? Because that wouldn't work with DMA coherent memory, which is contiguous and not backed by pages.
> At least if the entire justification for dma-buf support is zero-copy > support between the driver and userspace it's _really_ not the right> tool> for the job. dma-buf is for zero-copy between devices, with cpu access > from userpace (or kernel fwiw) being very much the exception (and often> flat-out not supported at all). We want both. Using dma-bufs for the driver/userspace interface is aconvenience as we then have a unique API instead of two distinct ones.Why should CPU access from userspace be the exception? It works fine for IIO dma-bufs. You keep warning about this being a terrible design, but I simplydon't see it.It depends really on what you're trying to do, and there's extremely highchances it will simply not work.
Well it does work though. The userspace interface is stupidly simple here - one dma-buf, backed by DMA coherent memory, is enqueued for processing by the DMA. The userspace calling the "sync" ioctl on the dma-buf will block until the transfer is complete, and then userspace can access it again.
Unless you want to do zero copy with a gpu, or something which is in that ecosystem of accelerators and devices, then dma-buf is probably not whatyou're looking for. -Daniel
I want to do zero-copy between a IIO device and the network/USB, and right now there is absolutely nothing in place that allows me to do that. So I have to get creative.
Cheers, -Paul
> > + > > +2. User API > > +=========== > > +> > +``IIO_BUFFER_DMABUF_ALLOC_IOCTL(struct iio_dmabuf_alloc_req *)`` > > +----------------------------------------------------------------> > +> > +Each call will allocate a new DMABUF object. The return value (if> > not> > +a negative errno value as error) will be the file descriptor of> > the new > > +DMABUF. > > + > > +``IIO_BUFFER_DMABUF_ENQUEUE_IOCTL(struct iio_dmabuf *)`` > > +-------------------------------------------------------- > > + > > +Place the DMABUF object into the queue pending for hardware > > process. > > + > > +These two IOCTLs have to be performed on the IIO buffer's file> > +descriptor, obtained using the `IIO_BUFFER_GET_FD_IOCTL` ioctl.> > + > > +3. Usage > > +======== > > +> > +To access the data stored in a block by userspace the block must be > > +mapped to the process's memory. This is done by calling mmap() on> > the > > +DMABUF's file descriptor. > > + > > +Before accessing the data through the map, you must use the > > +DMA_BUF_IOCTL_SYNC(struct dma_buf_sync *) ioctl, with the> > +DMA_BUF_SYNC_START flag, to make sure that the data is available. > > +This call may block until the hardware is done with this block.> > Once> > +you are done reading or writing the data, you must use this ioctl> > again> > +with the DMA_BUF_SYNC_END flag, before enqueueing the DMABUF to the> > +kernel's queue. > > +> > +If you need to know when the hardware is done with a DMABUF, you> > can > > +poll its file descriptor for the EPOLLOUT event. > > +> > +Finally, to destroy a DMABUF object, simply call close() on its> > file > > +descriptor. > > + > > +For more information about manipulating DMABUF objects, see: > > :ref:`dma-buf`. > > + > > +A typical workflow for the new interface is: > > + > > + for block in blocks: > > + DMABUF_ALLOC block > > + mmap block > > + > > + enable buffer > > + > > + while !done > > + for block in blocks: > > + DMABUF_ENQUEUE block > > + > > + DMABUF_SYNC_START block > > + process data > > + DMABUF_SYNC_END block > > + > > + disable buffer > > + > > + for block in blocks: > > + close block > > diff --git a/Documentation/iio/index.rst > > b/Documentation/iio/index.rst > > index 58b7a4ebac51..669deb67ddee 100644 > > --- a/Documentation/iio/index.rst > > +++ b/Documentation/iio/index.rst > > @@ -9,4 +9,6 @@ Industrial I/O > > > > iio_configfs > > > > + dmabuf_api > > + > > ep93xx_adc > > -- > > 2.34.1 > > > > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch-- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
