On Wed, Jun 14, 2023 at 6:03 PM Jens Axboe <axboe@xxxxxxxxx> wrote: > > On 6/14/23 8:09?AM, Marek Majkowski wrote: > > Hi! > > > > I'm playing with io-uring, and I found the io-wrk thread situation confusing. > > > > (A) In one case, I have a SOCK_DGRAM socket (blocking), over which I > > do IORING_OP_RECVMSG. This works well, and unless I mark the sqe as > > IOSQE_ASYNC, it doesn't seem to start an io-wrk kernel thread. > > > > (B) However, the same can't be said of another situation. In the > > second case I have a tap file descriptor (blocking), which doesn't > > support "Socket operations on non-socket", so I must do > > IORING_OP_READV. This however seems to start a new io-wrk for each > > readv request: > > > > $ pstree -pt `pidof tapuring` > > tapuring(44932)???{iou-wrk-44932}(44937) > > ??{iou-wrk-44932}(44938) > > ??{iou-wrk-44932}(44939) > > ??{iou-wrk-44932}(44940) > > ??{iou-wrk-44932}(44941) > > ??{iou-wrk-44932}(44942) > > > > I would expect both situations to behave the same way. > > > > The manpage for IOSQE_ASYNC: > > > > IOSQE_ASYNC > > Normal operation for io_uring is to try and issue an sqe > > as non-blocking first, and if that fails, execute it in an > > async manner. To support more efficient overlapped > > operation of requests that the application knows/assumes > > will always (or most of the time) block, the application > > can ask for an sqe to be issued async from the start. Note > > that this flag immediately causes the SQE to be offloaded > > to an async helper thread with no initial non-blocking > > attempt. This may be less efficient and should not be > > used liberally or without understanding the performance > > and efficiency tradeoffs. > > > > This seems to cover the tap file descriptor case. It tries to readv > > and when that fails a new io-wrk is spawned. Fine. However, as I > > described it seems this is not true for sockets, as without > > IOSQE_ASYNC the io-wrk thread is _not_ spawned there? > > > > Is the behaviour different due to socket vs non-socket or readv vs > > recvmsg? > > What kernel are you using? tap just recently got FMODE_NOWAIT support, > which should trigger poll instead of needing to spawn an io worker. > > Also, as usual, a test case would be appreciated. Particularly if this > is on a current kernel where we would not expect to see io-wq activity > for a read of tap. After two days I think I finally have some repro. Let's track this particular tap io-wrk issue under https://github.com/axboe/liburing/issues/886 I can confirm that indeed (apart form the mentioned bug) tap is going into poll mode, and doesn't launch io-wrk. However, I still miss a piece of the puzzle about io-wrk polling. Let's assume a different situation, let's say I have a dozen of sockets, and do a recvmsg on each of them, with IOSQE_ASYNC flag set. I would expect to see a dozen of io-wrk threads, however I'm not fully understanding what will they do. Will each do a full poll() over all the sockets? If a packet arrives to one socket, will all the io-wrk threads wakeup? What if there is a limit of 2 io-wrk threads and there are more sockets? I think I'm asking about the relationship between SQE's / sockets and io-wrk poller. Marek