On Nov 10, 2022 / 02:28, Dmitry Fomichev wrote:
...
> Yesterday, I encountered a test failure with the version 2 of this patch.
>
> When I create a /dev/nullb using this command -
>
> modprobe null_blk nr_devices=1 zoned=1 zone_size=16 memory_backed=1
>
> and run t/zbd/test-zbd-support /dev/nullb0 against this device, the test #34
> fails with verification errors:
>
> verify: bad magic header ffff, wanted acca at file /dev/nullb0 offset 33554432,
> length 28672 (requested block: offset=33554432, length=28672)
> verify: bad magic header ffff, wanted acca at file /dev/nullb0 offset 50331648,
> length 28672 (requested block: offset=50331648, length=28672)
> verify: bad magic header ffff, wanted acca at file /dev/nullb0 offset 16777216,
> length 28672 (requested block: offset=16777216, length=28672)
>
> I was hoping that v4 will fix this issue, but it is still happening in my
> testing. Setting zone_size to any other value or setting zone_capacity to a value
> other than zone_size makes the test run without an error. This is very peculiar.
> I wonder if you can reproduce this failure...
Thanks for finding this another bug:) Yes, the condition looks weird. I was able
to recreate the failure with zone size 2MB and 16MB. With zone size 1MB, 4MB,
8MB and 32MB, it was not observed.
I tracked down and found that get_next_rand_block() calls fio_file_reset() when
zone size is 2MB or 16MB. Zones are reset in fio_file_reset() path and the
verify data is screwed up. Hence the verify error.
The test case does random write to 4 zones with block size 28KB. To track random
write offsets, fio prepares a bitmap with number of bits = 4 * zone_size / 28KB.
Each bit in the bitmap indicates which block offset was used for random write.
When zone size is nor 2MB neither 16MB, all zones are filled and finished before
all bits of the bitmap marked. So get_next_rand_block() does not have chance to
call fio_file_reset(). When zone size is 2MB or 16MB, all bits are marked before
all zones get full, so get_next_rand_block() calls fio_file_reset(). This
failure path can be recreated with any zone size using block size almost same
but a bit smaller than zone size (e.g. zone size - 4k).
fio_file_reset() calls zbd_file_reset() which resets zones. zbd_file_reset()
checks the condition not to reset zones for verify, but that check did not
cover the failure path. So the fix should be in zbd_file_reset().
I create a quick fix patch below. td_runstate check is not good, since it is not
set to TD_VERIFYING when get_next_rand_block() calls fio_file_reset(). Instead,
I think td->io_hist_len should be referred. I confirmed this patch avoids the
failure. I will do some more work to brush up this fix. Will improve the test
case #34 also so that it can catch this failure.
diff --git a/zbd.c b/zbd.c
index 547c727f..3106831d 100644
--- a/zbd.c
+++ b/zbd.c
@@ -1270,7 +1270,7 @@ void zbd_file_reset(struct thread_data *td, struct fio_file *f)
* writing any data to avoid that a zone reset has to be issued while
* writing data, which causes data loss.
*/
- verify_ongoing = td->runstate == TD_VERIFYING || td->verify_batch;
+ verify_ongoing = td->io_hist_len || td->verify_batch;
if (td->io_hist_len && td->o.verify_backlog)
verify_ongoing = td->io_hist_len % td->o.verify_backlog;
if (td->o.verify != VERIFY_NONE && !verify_ongoing)
--
Shin'ichiro Kawasaki
