On 12/17/24 12:20 PM, Damien Le Moal wrote: > On 2024/12/17 11:07, Jens Axboe wrote: >> On 12/17/24 11:51 AM, Damien Le Moal wrote: >>> On 2024/12/17 10:46, Jens Axboe wrote: >>>>> Of note about io_uring: if writes are submitted from multiple jobs to >>>>> multiple queues, then you will see unaligned write errors, but the >>>>> same test with libaio will work just fine. The reason is that io_uring >>>>> fio engine IO submission only adds write requests to the io rings, >>>>> which will then be submitted by the kernel ring handling later. But at >>>>> that time, the ordering information is lost and if the rings are >>>>> processed in the wrong order, you'll get unaligned errors. >>>> >>>> Sorry, but this is woefully incorrect. >>>> >>>> Submissions are always in order, I suspect the main difference here is >>>> that some submissions would block, and that will certainly cause the >>>> effective issue point to be reordered, as the initial issue will get >>>> -EAGAIN. This isn't a problem on libaio as it simply blocks on >>>> submission instead. Because the actual issue is the same, and the kernel >>>> will absolutely see the submissions in order when io_uring_enter() is >>>> called, just like it would when io_submit() is called. >>> >>> I did not mean to say that the processing of requests in each >>> queue/ring is done out of order. They are not. What I meant to say is >>> that multiple queues/rings may be processed in parallel, so if >>> sequential writes are submitted to different queues, the BIOs for >>> these write IOs may endup being issued out of order to the zone. Is >>> that an incorrect assumption ? Reading the io_uring code, I think >>> there is one work item per ring and these are not synchronized. >> >> Sure, if you have multiple rings, there's no synchronization between >> them. Within each ring, reordering in terms of issue can only happen if >> the target response with -EAGAIN to a REQ_NOWAIT request, as they are >> always issued in order. If that doesn't happen, there should be no >> difference to what the issue looks like with multiple rings or contexts >> for io_uring or libaio - any kind of ordering could be observed. > > Yes. The fixes that went into rc3 addressed the REQ_NOWAIT issue. So > we are good on this front. I do remember that fix, but curious if there's still something wrong here... >> Unsure of which queues you are talking about here, are these the block >> level queues? > > My bad. I was talking about the io_uring rings. Not the block layer > queues. Gotcha >> And ditto on the io_uring question, which work items are we talking >> about? There can be any number of requests for any given ring, >> inflight. > > I was talking about the work that gets IOs submitted by the user from > the rings and turn them into BIOs for submission. My understanding is > that these are not synchronized. For a simple fio "--zonemode=zbd > --rw=randwrite --numjobs=X" for X >> 1, the fio level synchronization will serialize the calls to >> io_submit() for > libaio, thus delivering the BIOs to a zone in order in the kernel. > With io_uring as the I/O engine, the same fio level synchronization > happens but is only around the IO getting in the ring. The IOs being > turned into BIOs and submitted will be done outside of the fio > serialization and can thus can endup being issued out of order if > multiple rings are used. At least, that is my understanding of > io_uring... Am I getting this wrong ? Yes, this is totally wrong. If we assume a single ring, and a single thread/task driving this ring, then any IO being put into the ring is issued IN ORDER on the kernel side by that very same task once said task does io_uring_enter() to submit whatever it put in the ring. Like I mentioned earlier, there's _zero_ difference in the ordering in which these are issued between libaio and io_uring, for a single ring (or context) and a single task doing the submit and/or wait for events call. Fio doesn't have multiple threads operating on a ring, there's always a single ring per IO thread. Yes if you use multiple jobs and expect those to be ordered, that would obviously be broken across the board, regardless of IO engine used. So I realize this isn't the case. But there still seems to be some misunderstandings here on how io_uring works, and maybe even then an issue that's not being fully understood or blamed on something else. -- Jens Axboe
