On 10/24/24 15:43, Jens Axboe wrote:
On 10/24/24 8:36 AM, Pavel Begunkov wrote:
On 10/23/24 17:07, Jens Axboe wrote:
Hi,
Normally a request can take a provided buffer, which means "pick a
buffer from group X and do IO to/from it", or it can use a registered
buffer, which means "use the buffer at index Y and do IO to/from it".
For things like O_DIRECT and network zero copy, registered buffers can
be used to speedup the operation, as they avoid repeated
get_user_pages() and page referencing calls for each IO operation.
Normal (non zero copy) send supports bundles, which is a way to pick
multiple provided buffers at once and send them. send zero copy only
supports registered buffers, and hence can only send a single buffer
That's not true, has never been, send[msg] zc work just fine with
normal (non-registered) buffers.
That's not what I'm saying, perhaps it isn't clear. What I'm trying to
say is that it only supports registered buffers, it does not support
provided buffers. It obviously does support regular user provided
buffers that aren't registered or provided, I figured that goes without
saying explicitly.
Normally goes without saying yes, but the confusion here is because
of a more or less explicit implication (or at least I read it so)
"it only supports registered buffers => selected buffer support
should support registered buffers, which it adds"
Does the series allows provided buffers with normal user memory?
at the time.
And that's covered by the posted series for vectored registered
buffers support.
Right, for sendmsg.
This patchset adds support for using a mix of provided and registered
buffers, where the provided buffers merely provide an index into which
registered buffers to use. This enables using provided buffers for
send zc in general, but also bundles where multiple buffers are picked.
This is done by changing how the provided buffers are intepreted.
Normally a provided buffer has an address, length, and buffer ID
associated with it. The address tells the kernel where the IO should
occur. If both fixed and provided buffers are asked for, the provided
buffer address field is instead an encoding of the registered buffer
index and the offset within that buffer. With that in place, using a
combination of the two can work.
What the series doesn't say is how it works with notifications and
what is the proposed user API in regard to it, it's the main if not
the only fundamental distinctive part of the SENDZC API.
Should not change that? You'll should get the usual two notifications on
send complete, and reuse safe.
Right you get a notification, but what is it supposed to mean to
the user? Like "the notification indicates that all buffers that
are consumed by this request can be reused". Multishot is not a
thing, but how the user has to track what buffers are consumed
by this request? I assume it posts a CQE per buffer completion,
right?
And let's say you have send heavy workload where the user pushes
more than the socket can take, i.e. it has to wait to send more
and there is always something to send. Does it poll-retry as it's
usually done for multishots? How notifications are paced? i.e.
it'll continue hooking more and more buffers onto the same
notification locking all the previously used buffers.
--
Pavel Begunkov
