On 2020-12-10 23:15:15 +0000, Pavel Begunkov wrote: > On 10/12/2020 23:12, Pavel Begunkov wrote: > > On 10/12/2020 20:51, Andres Freund wrote: > >> Hi, > >> > >> When using hybrid polling (i.e echo 0 > > >> /sys/block/nvme1n1/queue/io_poll_delay) I see stalls with fio when using > >> an iodepth > 1. Sometimes fio hangs, other times the performance is > >> really poor. I reproduced this with SSDs from different vendors. > > > > Can you get poll stats from debugfs while running with hybrid? > > For both iodepth=1 and 32. > > Even better if for 32 you would show it in dynamic, i.e. cat it several > times while running it. Should read all email before responding... This is a loop of grepping for 4k writes (only type I am doing), with 1s interval. I started it before the fio run (after one with iodepth=1). Once the iodepth 32 run finished (--timeout 10, but took 42s0, I started a --iodepth 1 run. write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=3002, mean=7402, min=6683, max=22498 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=517838676, min=517774856, max=517901274 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=7365701186, min=7365642813, max=7365756630 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 Shortly after this I started the iodepth=1 run: write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=32, mean=30203322069, min=30203263000, max=30203381351 write (4096 Bytes): samples=1, mean=2216868822, min=2216868822, max=2216868822 write (4096 Bytes): samples=1, mean=2216868822, min=2216868822, max=2216868822 write (4096 Bytes): samples=1, mean=2216851683, min=2216851683, max=2216851683 write (4096 Bytes): samples=1, mean=1108526485, min=1108526485, max=1108526485 write (4096 Bytes): samples=1, mean=1108522634, min=1108522634, max=1108522634 write (4096 Bytes): samples=1, mean=277274275, min=277274275, max=277274275 write (4096 Bytes): samples=19, mean=5787160, min=5496432, max=10087444 write (4096 Bytes): samples=1185, mean=67915, min=66408, max=145100 write (4096 Bytes): samples=1185, mean=67915, min=66408, max=145100 write (4096 Bytes): samples=1185, mean=67915, min=66408, max=145100 write (4096 Bytes): samples=1703, mean=50492, min=39200, max=13155316 write (4096 Bytes): samples=9983, mean=7408, min=6648, max=29950 write (4096 Bytes): samples=9980, mean=7395, min=6574, max=23454 write (4096 Bytes): samples=10011, mean=7381, min=6620, max=25533 write (4096 Bytes): samples=9381, mean=7936, min=7270, max=47315 write (4096 Bytes): samples=9295, mean=7377, min=6665, max=23490 write (4096 Bytes): samples=9987, mean=7415, min=6629, max=23352 write (4096 Bytes): samples=9992, mean=7411, min=6651, max=23071 write (4096 Bytes): samples=9404, mean=7941, min=7234, max=24193 write (4096 Bytes): samples=9434, mean=7942, min=7240, max=62745 write (4096 Bytes): samples=5370, mean=7935, min=7268, max=24116 write (4096 Bytes): samples=5370, mean=7935, min=7268, max=24116 write (4096 Bytes): samples=5370, mean=7935, min=7268, max=24116 Greetings, Andres Freund
