On Tue, Oct 11, 2022 at 05:15:05PM +0200, Jaroslav Pulchart wrote:
> Hello,
>
> we disabled the wbt, issue is happening much sooner. The logs are attached
> 1/ "dmesg-20221011.log" form kernel messages
> 2/ "command.logs" from execution of
> (cd /sys/kernel/debug/block/vdc && find . -type f -exec grep -aH . {} \;)
>
>From command.logs, looks like it is one qemu or virtio-blk issue.
Maybe you can try to revert 0e9911fa768f ("virtio-blk: support mq_ops->queue_rqs()")
and see if it makes difference.
[1] all busy tags points to in-flight IOs
grep busy temp/command.logs | grep -v -E "dispatch_busy|busy=0"
./hctx27/tags:busy=3
./hctx27/busy:00000000e4e3af26 {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=167, .internal_tag=-1}
./hctx27/busy:00000000a9d6975f {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=168, .internal_tag=-1}
./hctx27/busy:00000000474663fc {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=169, .internal_tag=-1}
./hctx26/tags:busy=2
./hctx26/busy:00000000f32dd525 {.op=WRITE, .cmd_flags=SYNC, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=107, .internal_tag=-1}
./hctx26/busy:0000000055297a93 {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=108, .internal_tag=-1}
...
Thanks,
Ming
