On 10/12/2020 23:12, Pavel Begunkov wrote: > On 10/12/2020 20:51, Andres Freund wrote: >> Hi, >> >> When using hybrid polling (i.e echo 0 > >> /sys/block/nvme1n1/queue/io_poll_delay) I see stalls with fio when using >> an iodepth > 1. Sometimes fio hangs, other times the performance is >> really poor. I reproduced this with SSDs from different vendors. > > Can you get poll stats from debugfs while running with hybrid? > For both iodepth=1 and 32. Even better if for 32 you would show it in dynamic, i.e. cat it several times while running it. > > cat <debugfs>/block/nvme1n1/poll_stat > > e.g. if already mounted > cat /sys/kernel/debug/block/nvme1n1/poll_stat > >> >> >> $ echo -1 | sudo tee /sys/block/nvme1n1/queue/io_poll_delay >> $ fio --ioengine io_uring --rw write --filesize 1GB --overwrite=1 --name=test --direct=1 --bs=$((1024*4)) --time_based=1 --runtime=10 --hipri --iodepth 1 >> 93.4k iops >> >> $ fio --ioengine io_uring --rw write --filesize 1GB --overwrite=1 --name=test --direct=1 --bs=$((1024*4)) --time_based=1 --runtime=10 --hipri --iodepth 32 >> 426k iops >> >> $ echo 0 | sudo tee /sys/block/nvme1n1/queue/io_poll_delay >> $ fio --ioengine io_uring --rw write --filesize 1GB --overwrite=1 --name=test --direct=1 --bs=$((1024*4)) --time_based=1 --runtime=10 --hipri --iodepth 1 >> 94.3k iops >> >> $ fio --ioengine io_uring --rw write --filesize 1GB --overwrite=1 --name=test --direct=1 --bs=$((1024*4)) --time_based=1 --runtime=10 --hipri --iodepth 32 >> 167 iops >> fio took 33s >> >> >> However, if I ask fio / io_uring to perform all those IOs at once, the performance is pretty decent again (but obviously that's not that desirable) >> >> $ fio --ioengine io_uring --rw write --filesize 1GB --overwrite=1 --name=test --direct=1 --bs=$((1024*4)) --time_based=1 --runtime=10 --hipri --iodepth 32 --iodepth_batch_submit=32 --iodepth_batch_complete_min=32 >> 394k iops >> >> >> So it looks like there's something wrong around tracking what needs to >> be polled for in hybrid mode. -- Pavel Begunkov
