On 9/26/24 13:18, Jens Axboe wrote:
On 9/26/24 4:27 AM, Ming Lei wrote:
Hello Pavel, Jens and Guys,
On Thu, Sep 12, 2024 at 06:49:20PM +0800, Ming Lei wrote:
Hello,
The 1st 3 patches are cleanup, and prepare for adding sqe group.
The 4th patch supports generic sqe group which is like link chain, but
allows each sqe in group to be issued in parallel and the group shares
same IO_LINK & IO_DRAIN boundary, so N:M dependency can be supported with
sqe group & io link together. sqe group changes nothing on
IOSQE_IO_LINK.
The 5th patch supports one variant of sqe group: allow members to depend
on group leader, so that kernel resource lifetime can be aligned with
group leader or group, then any kernel resource can be shared in this
sqe group, and can be used in generic device zero copy.
The 6th & 7th patches supports providing sqe group buffer via the sqe
group variant.
The 8th patch supports ublk zero copy based on io_uring providing sqe
group buffer.
Tests:
1) pass liburing test
- make runtests
2) write/pass sqe group test case and sqe provide buffer case:
https://github.com/axboe/liburing/compare/master...ming1:liburing:sqe_group_v3
https://github.com/ming1/liburing/tree/sqe_group_v3
- covers related sqe flags combination and linking groups, both nop and
one multi-destination file copy.
- cover failure handling test: fail leader IO or member IO in both single
group and linked groups, which is done in each sqe flags combination
test
- covers IORING_PROVIDE_GROUP_KBUF by adding ublk-loop-zc
3) ublksrv zero copy:
ublksrv userspace implements zero copy by sqe group & provide group
kbuf:
git clone https://github.com/ublk-org/ublksrv.git -b group-provide-buf_v3
make test T=loop/009:nbd/061 #ublk zc tests
When running 64KB/512KB block size test on ublk-loop('ublk add -t loop --buffered_io -f $backing'),
it is observed that perf is doubled.
V6:
- follow Pavel's suggestion to disallow IOSQE_CQE_SKIP_SUCCESS &
LINK_TIMEOUT
- kill __io_complete_group_member() (Pavel)
- simplify link failure handling (Pavel)
- move members' queuing out of completion lock (Pavel)
- cleanup group io complete handler
- add more comment
- add ublk zc into liburing test for covering
IOSQE_SQE_GROUP & IORING_PROVIDE_GROUP_KBUF
Any comments on V6? So that I may address them in next version since
v6 has small conflict with mainline.
It looks fine to me, don't know if Pavel has any comments. Maybe just
toss out a v7 so it applies cleanly? I'll kick off the 6.13 branch
pretty soon.
Impl is not that straightforwardand warrants some prudence in
reviewing. I was visiting conferences, but going to take a look
next week or earlier.
--
Pavel Begunkov
