With norandommap for async I/O engines specifying I/O depth > 1, it is
possible that two or more writes with the same offset are queued at once.
When fio tries to verify the block, it may find a numberio mismatch
because the writes did not land on the media in the order that they were
queued. Avoid these spurious failures by disabling sequence number
checking. Users will still be able to enable sequence number checking
if they explicitly set the verify_header_sequence option.
fio -name=verify -ioengine=libaio -rw=randwrite -verify=sha512 -direct=1 \
-iodepth=32 -filesize=16M -bs=512 -norandommap=1 -debug=io,verify
Below is the truncated log for the above command demonstrating the issue.
This includes extra log entries when write sequence number is saved and
retrieved.
set: io_u->numberio=28489, off=0x5f2400
queue: io_u 0x5b8039e30d40: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
set: io_u->numberio=28574, off=0x5f2400
iolog: overlap 6235136/512, 6235136/512
queue: io_u 0x5b8039e75500: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
complete: io_u 0x5b8039e75500: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
complete: io_u 0x5b8039e30d40: off=0x5f2400,len=0x200,ddir=1,file=verify.0.0
retrieve: io_u->numberio=28574, off=0x5f2400
queue: io_u 0x5b8039e1db40: off=0x5f2400,len=0x200,ddir=0,file=verify.0.0
bad header numberio 28489, wanted 28574 at file verify.0.0 offset 6235136, length 512 (requested block: offset=6235136, length=512)
Signed-off-by: Ankit Kumar <ankit.kumar@xxxxxxxxxxx>
---
HOWTO.rst | 5 ++---
fio.1 | 5 ++---
init.c | 11 +++++++++++
3 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/HOWTO.rst b/HOWTO.rst
index fa6b44a5..ca0044ce 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -1568,9 +1568,8 @@ I/O type
disabled. If this option is used with :option:`verify` and multiple blocksizes
(via :option:`bsrange`), only intact blocks are verified, i.e.,
partially-overwritten blocks are ignored. With an async I/O engine and an I/O
- depth > 1, it is possible for the same block to be overwritten, which can
- cause verification errors. Either do not use norandommap in this case, or also
- use the lfsr random generator.
+ depth > 1, header write sequence number verification will be disabled. See
+ :option:`verify_write_sequence`.
.. option:: softrandommap=bool
diff --git a/fio.1 b/fio.1
index 07f09f07..fd580258 100644
--- a/fio.1
+++ b/fio.1
@@ -1371,9 +1371,8 @@ and that some blocks may be read/written more than once. If this option is
used with \fBverify\fR then \fBverify_header_seed\fR will be disabled. If this
option is used with \fBverify\fR and multiple blocksizes (via \fBbsrange\fR),
only intact blocks are verified, i.e., partially-overwritten blocks are
-ignored. With an async I/O engine and an I/O depth > 1, it is possible for
-the same block to be overwritten, which can cause verification errors. Either
-do not use norandommap in this case, or also use the lfsr random generator.
+ignored. With an async I/O engine and an I/O depth > 1, header write sequence
+number verification will be disabled. See \fBverify_write_sequence\fR.
.TP
.BI softrandommap \fR=\fPbool
See \fBnorandommap\fR. If fio runs with the random block map enabled and
diff --git a/init.c b/init.c
index 91f056e3..00a3a8c7 100644
--- a/init.c
+++ b/init.c
@@ -862,6 +862,17 @@ static int fixup_options(struct thread_data *td)
o->verify_header_seed = 0;
}
+ if (o->norandommap && !td_ioengine_flagged(td, FIO_SYNCIO) &&
+ o->iodepth > 1) {
+ /*
+ * Disable write sequence checks with norandommap and
+ * iodepth > 1.
+ * Unless we were explicitly asked to enable it.
+ */
+ if (!fio_option_is_set(o, verify_write_sequence))
+ o->verify_write_sequence = 0;
+ }
+
/*
* Disable rand_seed check when we have verify_backlog,
* zone reset frequency for zonemode=zbd, or norandommap.
--
2.25.1
