On Apr 14, 2022 / 15:02, Luis Chamberlain wrote: > Hey folks, > > While enhancing kdevops [0] to embrace automation of testing with > blktests for ZNS I ended up spotting a possible false positive RCU stall > when running zbd/006 after zbd/005. The curious thing though is that > this possible RCU stall is only possible when using the qemu > ZNS drive, not when using nbd. In so far as kdevops is concerned > it creates ZNS drives for you when you enable the config option > CONFIG_QEMU_ENABLE_NVME_ZNS=y. So picking any of the ZNS drives > suffices. When configuring blktests you can just enable the zbd > guest, so only a pair of guests are reated the zbd guest and the > respective development guest, zbd-dev guest. When using > CONFIG_KDEVOPS_HOSTS_PREFIX="linux517" this means you end up with > just two guests: > > * linux517-blktests-zbd > * linux517-blktests-zbd-dev > > The RCU stall can be triggered easily as follows: > > make menuconfig # make sure to enable CONFIG_QEMU_ENABLE_NVME_ZNS=y and blktests > make > make bringup # bring up guests > make linux # build and boot into v5.17-rc7 > make blktests # build and install blktests > > Now let's ssh to the guest while leaving a console attached > with `sudo virsh vagrant_linux517-blktests-zbd` in a window: > > ssh linux517-blktests-zbd > sudo su - > cd /usr/local/blktests > export TEST_DEVS=/dev/nvme9n1 > i=0; while true; do ./check zbd/005 zbd/006; if [[ $? -ne 0 ]]; then echo "BAD at $i"; break; else echo GOOOD $i ; fi; let i=$i+1; done; > > The above should never fail, but you should eventually see an RCU > stall candidate on the console. The full details can be observed on the > gist [1] but for completeness I list some of it below. It may be a false > positive at this point, not sure. > > [493272.711271] run blktests zbd/005 at 2022-04-14 20:03:22 > [493305.769531] run blktests zbd/006 at 2022-04-14 20:03:55 > [493336.979482] nvme nvme9: I/O 192 QID 5 timeout, aborting > [493336.981666] nvme nvme9: Abort status: 0x0 > [493367.699440] nvme nvme9: I/O 192 QID 5 timeout, reset controller > [493388.819341] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Hello Luis, I run blktests zbd group on several QEMU ZNS emulation devices for every rcX kernel releases. But, I have not ever observed the symptom above. Now I'm repeating zbd/005 and zbd/006 using v5.18-rc3 and a QEMU ZNS device, and do not observe the symptom so far, after 400 times repeat. I would like to run the test using same ZNS set up as yours. Can you share how your ZNS device is set up? I would like to know device size and QEMU -device options, such as zoned.zone_size or zoned.max_active. -- Best Regards, Shin'ichiro Kawasaki
