Hi,
Le ven., août 13 2021 at 18:20:19 +0100, Pavel Begunkov
<asml.silence@xxxxxxxxx> a écrit :
Hi Paul,
On 8/13/21 12:41 PM, Paul Cercueil wrote:
Hi,
A few months ago we (ADI) tried to upstream the interface we use
with our high-speed ADCs and DACs. It is a system with custom ioctls
on the iio device node to dequeue and enqueue buffers (allocated
with dma_alloc_coherent), that can then be mmap'd by userspace
applications. Anyway, it was ultimately denied entry [1]; this API
was okay in ~2014 when it was designed but it feels like
re-inventing the wheel in 2021.
Back to the drawing table, and we'd like to design something that
we can actually upstream. This high-speed interface looks awfully
similar to DMABUF, so we may try to implement a DMABUF interface for
IIO, unless someone has a better idea.
Our first usecase is, we want userspace applications to be able to
dequeue buffers of samples (from ADCs), and/or enqueue buffers of
samples (for DACs), and to be able to manipulate them (mmapped
buffers). With a DMABUF interface, I guess the userspace application
would dequeue a dma buffer from the driver, mmap it, read/write the
data, unmap it, then enqueue it to the IIO driver again so that it
can be disposed of. Does that sound sane?
Our second usecase is - and that's where things get tricky - to be
able to stream the samples to another computer for processing, over
Ethernet or USB. Our typical setup is a high-speed ADC/DAC on a dev
board with a FPGA and a weak soft-core or low-power CPU; processing
the data in-situ is not an option. Copying the data from one buffer
to another is not an option either (way too slow), so we absolutely
want zero-copy.
Usual userspace zero-copy techniques (vmsplice+splice, MSG_ZEROCOPY
etc) don't really work with mmapped kernel buffers allocated for DMA
[2] and/or have a huge overhead, so the way I see it, we would also
need DMABUF support in both the Ethernet stack and USB (functionfs)
stack. However, as far as I understood, DMABUF is mostly a DRM/V4L2
thing, so I am really not sure we have the right idea here.
And finally, there is the new kid in town, io_uring. I am not very
literate about the topic, but it does not seem to be able to handle
DMA buffers (yet?). The idea that we could dequeue a buffer of
samples from the IIO device and send it over the network in one
single syscall is appealing, though.
You might be interested to look up zctap, previously a.k.a netgpu.
CCing Jonathan (Lemon) then.
Jonathan: Am I right that zctap supports importing/exporting dmabufs?
Because that would solve half of my problem.
Cheers,
-Paul
For io_uring, it's work in progress as well.
Any thoughts? Feedback would be greatly appreciated.
Cheers,
-Paul
[1]:
https://lore.kernel.org/linux-iio/20210217073638.21681-1-alexandru.ardelean@xxxxxxxxxx/T/#m6b853addb77959c55e078fbb06828db33d4bf3d7
[2]:
https://newbedev.com/zero-copy-user-space-tcp-send-of-dma-mmap-coherent-mapped-memory
--
Pavel Begunkov
