The third iteration of patches for zerocopy io_uring sends. I fixed all known issues since the previous version and reshuffled io_uring patches, but the net/ code didn't change much. I think it's ready and will send it as a non-RFC soon. All tests below are done using io_uring with all relevant performance options turned on. Numbers look good, send + flush per request, which is the worst case, is on par with non-zerocopy with the payload size lower than 600 bytes with dummy netdev and b/w 1200-1500 for NIC tests. Without "buffer-free" notification flushing at all it's on par with NIC at around 600 bytes. dummy: IO size | non-zc (tx/s) | zc (tx/s) | zc + flush (tx/s) 8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%) 600 | 2106794 | 2381527 (+13%) | 2195295 (+4%) NIC: IO size | non-zc (tx/s) | zc (tx/s) | zc + flush (tx/s) 4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%) 600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%) Apart from zerocopy, it also removes page referencing for reigstered buffers (used in all zc tests). I'm experimenting with notificaiton optimsation, which should improve the 3rd column, but that will go separately from this series. I've also seen good CPU usage reduction for TCP comparing to non-zc, but not posting numbers as had problems saturating CPU. Links: RFC v1: https://lore.kernel.org/io-uring/cover.1638282789.git.asml.silence@xxxxxxxxx/ RFC v2: https://lore.kernel.org/io-uring/cover.1640029579.git.asml.silence@xxxxxxxxx/ liburing (copy of the benchmark + some tests): https://github.com/isilence/liburing/tree/zc_v3 kernel repo: https://github.com/isilence/linux/tree/zc_v3 API design overview: First we take an internal zerocopy handler, aka struct ubuf_info, and let io_uring to pass it into the network layer in struct msghdr. io_uring stores them as wrapping into struct io_notif. It also has an array of so called notification slots, each keeps one and only one active notifier at a time, to which the userspace can bind requests by specifying the slot index. Then the userspace can request to flush a notifier, so when all buffers and requests used with this notifier complete/freed it'll post one CQE. The userspace can't bind new requests to a flushed notifier, however, it can use the slot as flushing automatically replaces the notifier with a new one. Changelog: RFC v2 -> RFC v3: TCP support accounting for normal (non-registered) buffers allow to combine reg and normal requests within a notifier notification flushing via IORING_OP_RSRC_UPDATE overriding io_uring notification tag/user_data add ubuf_info submmision side reference caching/batching fix buffer indexing fix io-wq ->uring_lock locking fix bugs when mixing with MSG_ZEROCOPY fix managed refs bugs in skbuff.c numerous cleanups RFC -> RFC v2: remove additional overhead for non-zc from skb_release_data() avoid msg propagation, hide extra bits of non-zc overhead task_work based "buffer free" notifications improve io_uring's notification refcounting added 5/19, (no pfmemalloc tracking) added 8/19 and 9/19 preventing small copies with zc misc small changes Pavel Begunkov (29): ipv4: avoid partial copy for zc ipv6: avoid partial copy for zc skbuff: add SKBFL_DONT_ORPHAN flag skbuff: carry external ubuf_info in msghdr net: bvec specific path in zerocopy_sg_from_iter net: optimise bvec-based zc page referencing net: don't track pfmemalloc for managed frags skbuff: don't mix ubuf_info of different types ipv4/udp: support zc with managed data ipv6/udp: support zc with managed data tcp: support zc with managed data tcp: kill extra io_uring's uarg refcounting net: let callers provide extra ubuf_info refs io_uring: opcode independent fixed buf import io_uring: add zc notification infrastructure io_uring: cache struct io_notif io_uring: complete notifiers in tw io_uring: add notification slot registration io_uring: rename IORING_OP_FILES_UPDATE io_uring: add zc notification flush requests io_uring: wire send zc request type io_uring: account locked pages for non-fixed zc io_uring: allow to pass addr into sendzc io_uring: add rsrc referencing for notifiers io_uring: sendzc with fixed buffers io_uring: flush notifiers after sendzc io_uring: allow to override zc tag on flush io_uring: batch submission notif referencing selftests/io_uring: test zerocopy send fs/io_uring.c | 566 +++++++++++++++- include/linux/skbuff.h | 59 +- include/linux/socket.h | 8 + include/uapi/linux/io_uring.h | 43 +- net/compat.c | 2 + net/core/datagram.c | 53 +- net/core/skbuff.c | 35 +- net/ipv4/ip_output.c | 66 +- net/ipv4/tcp.c | 56 +- net/ipv6/ip6_output.c | 65 +- net/socket.c | 6 + tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 14 files changed, 1613 insertions(+), 83 deletions(-) create mode 100644 tools/testing/selftests/net/io_uring_zerocopy_tx.c create mode 100755 tools/testing/selftests/net/io_uring_zerocopy_tx.sh -- 2.36.1
