On 3/8/22 1:38 AM, Xiaoguang Wang wrote: > hi, > >>>> I'll take a look at liburing and see what we need to do there. I think >>>> the sanest thing to do here is say that using a registered ring fd means >>>> you cannot share the ring, ever. And then just have a >>>> ring->enter_ring_fd which is normally just set to ring_fd when the ring >>>> is setup, and if you register the ring fd, then we set it to whatever >>>> the registered value is. Everything calling io_uring_enter() then just >>>> needs to be modified to use ->enter_ring_fd instead of ->ring_fd. >>> ok, look forward to use this api. >> Can you take a look at the registered-ring branch for liburing: >> >> https://git.kernel.dk/cgit/liburing/log/?h=registered-ring >> >> which has the basic plumbing for it. Comments (or patches) welcome! > Sorry for late reply, spend time to read your patch today. Basically it looks ok, > there is one minor issue in "Add preliminary support for using a registered ring fd": > @@ -417,6 +425,10 @@ struct io_uring_sqe *io_uring_get_sqe(struct io_uring *ring) > > int __io_uring_sqring_wait(struct io_uring *ring) > { > - return ____sys_io_uring_enter(ring->ring_fd, 0, 0, > - IORING_ENTER_SQ_WAIT, NULL); > + int flags = IORING_ENTER_SQ_WAIT; > + > + if (ring->int_flags & INT_FLAG_REG_RING) > + flags |= IORING_ENTER_REGISTERED_RING; > + > + return ____sys_io_uring_enter(ring->ring_fd, 0, 0, flags, NULL); > } > > Here it should be enter_ring_fd. Ah good catch, I've fixed that up. >> Few things I don't really love: >> >> 1) You need to call io_uring_register_ring_fd() after setting up the >> ring. We could provide init helpers for that, which just do queue >> init and then register ring. Maybe that'd make it more likely to get >> picked up by applications. > Agree, that'd be better in some cases, but consider that currently the > capacity of ring fd cache is just 16, I'd suggest to let users make > their own decisions, in case some ring fds could not allocate one > empty slot, but some ring fds don't need them at all, for example, > ring fd which enable sqpoll may no need this feature. Agree, that's the route I ended up taking too. >> 2) For the setup where you do share the ring between a submitter and >> reaper, we need to ensure that the registered ring fd is the same >> between both of them. We need a helper for that. It's basically the >> same as io_uring_register_ring_fd(), but we need the specific offset. >> And if that fails with -EBUSY, we should just turn off >> INT_FLAG_RING_REG for the ring and you don't get the registered fd >> for either of them. At least it can be handled transparantly. > Storing enter_ring_fd in struct io_uring seems not good, struct > io_uring is a shared struct, as what you say, different threads that > share one ring fd may have differed offset in ring fd cache. I have > two suggestions: > 1) Threads keep their offset in ring fd cache alone, and pass it to > io_uring_submit, which may look ugly :) > 2) define enter_ring_fd in struct io_ring to be a thread_local type, > then your patches don't need to do any modifications. Maybe I'm missing something here, but I don't see how your number 2 is possible? I think for now I'll just keep it as-is. Yes, then you can't use a registered ring fd if you share the ring betwen threads, but so be it. -- Jens Axboe
