Hi, On 2023-10-24 12:12:23 +1100, Dave Chinner wrote: > [cc Jens, io-uring] Yes, good call, had been meaning to do that but forgot. > On Tue, Oct 17, 2023 at 07:50:09PM -0700, Andres Freund wrote: > > I had the idea to look at the stacks (via /proc/$pid/stack) for all postgres > > processes and the associated io-uring threads, and then to deduplicate them. > > > > 22x: > > ext4_file_write_iter (./include/linux/fs.h:1073 fs/ext4/file.c:57 fs/ext4/file.c:564 fs/ext4/file.c:715) > > io_write (./include/linux/fs.h:1956 io_uring/rw.c:926) > > io_issue_sqe (io_uring/io_uring.c:1878) > > io_wq_submit_work (io_uring/io_uring.c:1960) > > io_worker_handle_work (io_uring/io-wq.c:596) > > io_wq_worker (io_uring/io-wq.c:258 io_uring/io-wq.c:648) > > ret_from_fork (arch/x86/kernel/process.c:147) > > ret_from_fork_asm (arch/x86/entry/entry_64.S:312) > > io_uring does some interesting stuff with IO completion and iomap > now - IIRC IOCB_DIO_CALLER_COMP is new 6.6-rc1 functionality. This > flag is set by io_write() to tell the iomap IO completion handler > that the caller will the IO completion, avoiding a context switch > to run the completion in a kworker thread. > > It's not until the caller runs iocb->dio_complete() that > inode_dio_end() is called to drop the i_dio_count. This happens when > io_uring gets completion notification from iomap via > iocb->ki_complete(iocb, 0); > > It then requires the io_uring layer to process the completion > and call iocb->dio_complete() before the inode->i_dio_count is > dropped and inode_dio_wait() will complete. > > So what I suspect here is that we have a situation where the worker > that would run the completion is blocked on the inode rwsem because > this isn't a IOCB_NOWAIT IO and the fallocate call holds the rwsem > exclusive and is waiting on inode_dio_wait() to return. > > Cc Jens, because I'm not sure this is actually an ext4 problem - I > can't see why XFS won't have the same issue w.r.t. > truncate/fallocate and IOCB_DIO_CALLER_COMP delaying the > inode_dio_end() until whenever the io_uring code can get around to > processing the delayed completion.... The absence of a reproducer obviously is not proof of an absence of problems. With that said, I have run the workload on xfs for far longer on both bigger and smaller machines, without triggering the same symptoms. Greetings, Andres Freund
