On 5/28/20 12:35 PM, Jeff Moyer wrote: > Bijan Mottahedeh <bijan.mottahedeh@xxxxxxxxxx> writes: > >> Mark a REQ_NOWAIT request for a non-mq queue as unspported instead of >> retryable since otherwise the io_uring layer will keep resubmitting >> the request. > > Getting back to this... > > Jens, right now (using your io_uring-5.7 or linus' tree) fio's > t/io_uring will never get io completions when run against a file on a > file system that is backed by lvm. The system will have one workqueue > per sqe submitted, all spinning, eating up CPU time. > > # ./t/io_uring /mnt/test/poo > Added file /mnt/test/poo > sq_ring ptr = 0x0x7fbed40ae000 > sqes ptr = 0x0x7fbed40ac000 > cq_ring ptr = 0x0x7fbed40aa000 > polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256 > submitter=3851 > IOPS=128, IOS/call=6/0, inflight=128 (128) > IOPS=0, IOS/call=0/0, inflight=128 (128) > IOPS=0, IOS/call=0/0, inflight=128 (128) > IOPS=0, IOS/call=0/0, inflight=128 (128) > IOPS=0, IOS/call=0/0, inflight=128 (128) > IOPS=0, IOS/call=0/0, inflight=128 (128) > ... > > # ps auxw | grep io_wqe > root 3849 80.1 0.0 0 0 ? R 14:32 0:40 [io_wqe_worker-0] > root 3850 0.0 0.0 0 0 ? S 14:32 0:00 [io_wqe_worker-0] > root 3853 72.8 0.0 0 0 ? R 14:32 0:36 [io_wqe_worker-0] > root 3854 81.4 0.0 0 0 ? R 14:32 0:40 [io_wqe_worker-1] > root 3855 74.8 0.0 0 0 ? R 14:32 0:37 [io_wqe_worker-0] > root 3856 74.8 0.0 0 0 ? R 14:32 0:37 [io_wqe_worker-1] > ... > > # ps auxw | grep io_wqe | grep -v grep | wc -l > 129 > > With this patch applied, the test program will exit without doing I/O > (which I don't think is the right behavior either, right?): > > # t/io_uring /mnt/test/poo > Added file /mnt/test/poo > sq_ring ptr = 0x0x7fdb98f00000 > sqes ptr = 0x0x7fdb98efe000 > cq_ring ptr = 0x0x7fdb98efc000 > polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256 > submitter=33233 > io: unexpected ret=-95 > Your filesystem/driver/kernel doesn't support polled IO > IOPS=128, IOS/call=32/0, inflight=128 (127) > > /mnt/test is an xfs file system on top of a linear LVM volume on an nvme > device (with 8 poll queues configured). poll won't work over dm, so that looks correct. What happens if you edit it and disable poll? Would be curious to see both buffered = 0 and buffered = 1 runs with that. I'll try this here too. -- Jens Axboe
