On 12/17/24 11:51 AM, Damien Le Moal wrote: > On 2024/12/17 10:46, Jens Axboe wrote: >>> Of note about io_uring: if writes are submitted from multiple jobs to >>> multiple queues, then you will see unaligned write errors, but the >>> same test with libaio will work just fine. The reason is that io_uring >>> fio engine IO submission only adds write requests to the io rings, >>> which will then be submitted by the kernel ring handling later. But at >>> that time, the ordering information is lost and if the rings are >>> processed in the wrong order, you'll get unaligned errors. >> >> Sorry, but this is woefully incorrect. >> >> Submissions are always in order, I suspect the main difference here is >> that some submissions would block, and that will certainly cause the >> effective issue point to be reordered, as the initial issue will get >> -EAGAIN. This isn't a problem on libaio as it simply blocks on >> submission instead. Because the actual issue is the same, and the kernel >> will absolutely see the submissions in order when io_uring_enter() is >> called, just like it would when io_submit() is called. > > I did not mean to say that the processing of requests in each > queue/ring is done out of order. They are not. What I meant to say is > that multiple queues/rings may be processed in parallel, so if > sequential writes are submitted to different queues, the BIOs for > these write IOs may endup being issued out of order to the zone. Is > that an incorrect assumption ? Reading the io_uring code, I think > there is one work item per ring and these are not synchronized. Sure, if you have multiple rings, there's no synchronization between them. Within each ring, reordering in terms of issue can only happen if the target response with -EAGAIN to a REQ_NOWAIT request, as they are always issued in order. If that doesn't happen, there should be no difference to what the issue looks like with multiple rings or contexts for io_uring or libaio - any kind of ordering could be observed. Unsure of which queues you are talking about here, are these the block level queues? And ditto on the io_uring question, which work items are we talking about? There can be any number of requests for any given ring, inflight. -- Jens Axboe
