On 5/28/20 1:01 PM, Jens Axboe wrote: > On 5/28/20 12:35 PM, Jeff Moyer wrote: >> Bijan Mottahedeh <bijan.mottahedeh@xxxxxxxxxx> writes: >> >>> Mark a REQ_NOWAIT request for a non-mq queue as unspported instead of >>> retryable since otherwise the io_uring layer will keep resubmitting >>> the request. >> >> Getting back to this... >> >> Jens, right now (using your io_uring-5.7 or linus' tree) fio's >> t/io_uring will never get io completions when run against a file on a >> file system that is backed by lvm. The system will have one workqueue >> per sqe submitted, all spinning, eating up CPU time. >> >> # ./t/io_uring /mnt/test/poo >> Added file /mnt/test/poo >> sq_ring ptr = 0x0x7fbed40ae000 >> sqes ptr = 0x0x7fbed40ac000 >> cq_ring ptr = 0x0x7fbed40aa000 >> polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256 >> submitter=3851 >> IOPS=128, IOS/call=6/0, inflight=128 (128) >> IOPS=0, IOS/call=0/0, inflight=128 (128) >> IOPS=0, IOS/call=0/0, inflight=128 (128) >> IOPS=0, IOS/call=0/0, inflight=128 (128) >> IOPS=0, IOS/call=0/0, inflight=128 (128) >> IOPS=0, IOS/call=0/0, inflight=128 (128) >> ... >> >> # ps auxw | grep io_wqe >> root 3849 80.1 0.0 0 0 ? R 14:32 0:40 [io_wqe_worker-0] >> root 3850 0.0 0.0 0 0 ? S 14:32 0:00 [io_wqe_worker-0] >> root 3853 72.8 0.0 0 0 ? R 14:32 0:36 [io_wqe_worker-0] >> root 3854 81.4 0.0 0 0 ? R 14:32 0:40 [io_wqe_worker-1] >> root 3855 74.8 0.0 0 0 ? R 14:32 0:37 [io_wqe_worker-0] >> root 3856 74.8 0.0 0 0 ? R 14:32 0:37 [io_wqe_worker-1] >> ... >> >> # ps auxw | grep io_wqe | grep -v grep | wc -l >> 129 >> >> With this patch applied, the test program will exit without doing I/O >> (which I don't think is the right behavior either, right?): >> >> # t/io_uring /mnt/test/poo >> Added file /mnt/test/poo >> sq_ring ptr = 0x0x7fdb98f00000 >> sqes ptr = 0x0x7fdb98efe000 >> cq_ring ptr = 0x0x7fdb98efc000 >> polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256 >> submitter=33233 >> io: unexpected ret=-95 >> Your filesystem/driver/kernel doesn't support polled IO >> IOPS=128, IOS/call=32/0, inflight=128 (127) >> >> /mnt/test is an xfs file system on top of a linear LVM volume on an nvme >> device (with 8 poll queues configured). > > poll won't work over dm, so that looks correct. What happens if you edit > it and disable poll? Would be curious to see both buffered = 0 and > buffered = 1 runs with that. > > I'll try this here too. I checked, and with the offending commit reverted, it behaves exactly like it should - io_uring doesn't hit endless retries, and we still return -EAGAIN to userspace for preadv2(..., RFW_NOWAIT) if not supported. I've queued up the revert. Jeff, the poll test above is supposed to fail as we can't poll on dm. So that part is expected. -- Jens Axboe
