On 2/28/24 10:28 AM, Jens Axboe wrote: > When I have some time I can do add the append case, or feel free to do > that yourself, and I can run some testing with that too. I did a quick hack to add the append mode, and by default we get roughly ring_size / 2 number of appended vecs, which I guess is expected. There's a few complications there: 1) We basically need a per-send data structure at this point. While that's not hard to do, at least for my case I'd need to add that just for this case and everything would now need to do it. Perhaps. You can perhaps steal a few low bits and only do it for sendmsg. But why? Because now you have multiple buffer IDs in a send completion, and you need to be able to unravel it. If we always receive and send in order, then it'd always been contiguous, which I took advantage of. Not a huge deal, just mentioning some of the steps I had to go through. 2) The iovec. Fine if you have the above data structure, as you can just alloc/realloc -> free when done. I just took the easy path and made the iovec big enough (eg ring size). Outside of that, didn't need to make a lot of changes: 1 file changed, 39 insertions(+), 17 deletions(-) Performance is great, because we get to pass in N (in my case ~65) packets per send. No more per packet locking. Which I do think highlights that we can do better on the multishot send/sendmsg by grabbing buffers upfront and passing them in in one go rather than simply loop around calling tcp_sendmsg_locked() for each one. Anyway, something to look into! -- Jens Axboe
