On 3/10/22 03:00, Jens Axboe wrote:
On 3/9/22 7:11 PM, Artyom Pavlov wrote:
10.03.2022 04:36, Jens Axboe wrote:
On 3/9/22 4:49 PM, Artyom Pavlov wrote:
Greetings!
A common approach for multi-threaded servers is to have a number of
threads equal to a number of cores and launch a separate ring in each
one. AFAIK currently if we want to send an event to a different ring,
we have to write-lock this ring, create SQE, and update the index
ring. Alternatively, we could use some kind of user-space message
passing.
Such approaches are somewhat inefficient and I think it can be solved
elegantly by updating the io_uring_sqe type to allow accepting fd of a
ring to which CQE must be sent by kernel. It can be done by
introducing an IOSQE_ flag and using one of currently unused padding
u64s.
Such feature could be useful for load balancing and message passing
between threads which would ride on top of io-uring, i.e. you could
send NOP with user_data pointing to a message payload.
So what you want is a NOP with 'fd' set to the fd of another ring, and
that nop posts a CQE on that other ring? I don't think we'd need IOSQE
flags for that, we just need a NOP that supports that. I see a few ways
of going about that:
1) Add a new 'NOP' that takes an fd, and validates that that fd is an
io_uring instance. It can then grab the completion lock on that ring
and post an empty CQE.
2) We add a FEAT flag saying NOP supports taking an 'fd' argument, where
'fd' is another ring. Posting CQE same as above.
3) We add a specific opcode for this. Basically the same as #2, but
maybe with a more descriptive name than NOP.
Might make sense to pair that with a CQE flag or something like that, as
there's no specific user_data that could be used as it doesn't match an
existing SQE that has been issued. IORING_CQE_F_WAKEUP for example.
Would be applicable to all the above cases.
I kind of like #3 the best. Add a IORING_OP_RING_WAKEUP command, require
that sqe->fd point to a ring (could even be the ring itself, doesn't
matter). And add IORING_CQE_F_WAKEUP as a specific flag for that.
No, ideally I would like to be able to send any type of SQE to a
different ring. For example, if I see that the current ring is
overloaded, I can create exactly the same SQEs as during usual
operation, but with a changed recipient ring.
Your approach with a new "sendable" NOP will allow to emulate it in
user-space, but it will involve unnecessary ring round-trip and will
be a bit less pleasant in user code, e.g. we would need to encode a
separate state "the task is being sent to a different ring" instead of
simply telling io-uring "read data and report CQE on this ring"
without any intermediate states.
OK, so what you're asking is to be able to submit an sqe to ring1, but
have the completion show up in ring2? With the idea being that the rings
are setup so that you're basing this on which thread should ultimately
process the request when it completes, which is why you want it to
target another ring?
It'd certainly be doable, but it's a bit of a strange beast. My main
concern with that would be:
1) It's a fast path code addition to every request, we'd need to check
some new field (sqe->completion_ring_fd) and then also grab a
reference to that file for use at completion time.
2) Completions are protected by the completion lock, and it isn't
trivial to nest these. What happens if ring1 submits an sqe with
ring2 as the cqe target, and ring2 submits an sqe with ring1 as the
cqe target? We can't safely nest these, as we could easily introduce
deadlocks that way.
My knee jerk reaction is that it'd be both simpler and cheaper to
implement this in userspace... Unless there's an elegant solution to it,
which I don't immediately see.
Per request fd will be ugly and slow unfortunately. As people asked about
a similar thing before, the only thing I can suggest is to add a way
to pass another SQ. The execution will be slower, but at least can be
made zero overhead for the normal path.
--
Pavel Begunkov
