Currently we are using a list to log I/O history if:
* randommap is enabled and fio manages to allocate memory for it.
* there are no offset modfiers with any jobs.
For any different scenario we use an RB tree to handle offset overlaps,
which disables header seed checks for them.
This commit expands fio_offset_overlap_risk() such that it covers
file_randommap() cases.
For random workload with this change these are the possible scenarios
-----------------------------------------------------------------------
| | norandommap=0 | norandommap=1 |
|-----------------------------------------------------------------------|
| softrandommap=0 | list (No change) | RB tree |
| | (fio was able to allocate memory) | (No change) |
|-----------------|------------------------------------|----------------|
| | RB tree (Now always) | RB tree |
| softrandommap=1 |Even if fio was able to allocate mem| (No Change) |
-----------------------------------------------------------------------
With randommap enabled and softrandommap=1 we now always use an RB tree,
even when fio is able to allocate memory for random map. In this case
verify header seed check will be disabled. If users want to check header
seed they can either disable softrandommap or explicilty enable
verify_header_seed.
Effectively this changes randommap from being a per-file property to
per-job property.
This also fixes rand seed mismatch isues, that have been observed when
multiple files are used, such as for the below mentioned configuration.
[global]
do_verify=1
verify=md5
direct=1
[multi_file]
rw=readwrite
directory=.
nrfiles=2
size=32K
Here is the truncated log with debug=verify flag, and an extra log when
the seed gets generated as well as the mismatch.
verify 368109 file ./multi_file.0.1 seed 46386204153304124 offset=0, length=4096
verify 368109 file ./multi_file.0.0 seed 9852480210356360750 offset=0, length=4096
verify 368109 file ./multi_file.0.1 seed 4726550845720924880 offset=4096, length=4096
verify: bad header rand_seed 9852480210356360750, wanted 46386204153304124 at file ./multi_file.0.0 offset 0, length 4096 (requested block: offset=0, length=4096)
Earlier the I/O entries were getting logged in an RB tree, as we were
relying on file_randommap(), which was false for sequential workloads.
In RB tree, files are prioritized first and then the offset. Thus during
the verify phase the I/O entries are removed from tree in order of file
and then offset which is not how it was originally written. With the new
checks, for sequential workload we now store the entries in the list
instead of RB tree.
Even for sequential workload if the user fortuitously specified
norandommap or softrandommap, then I/Os will be stored in an RB tree.
However in this case header seed checks will be disabled.
fixes #740
fixes #746
fixes #844
fixes #1538
Signed-off-by: Vincent Fu <vincent.fu@xxxxxxxxxxx>
Signed-off-by: Ankit Kumar <ankit.kumar@xxxxxxxxxxx>
---
fio.h | 3 ++-
init.c | 7 +++----
iolog.c | 2 +-
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/fio.h b/fio.h
index c7c0d147..d6423258 100644
--- a/fio.h
+++ b/fio.h
@@ -802,7 +802,8 @@ extern void lat_target_reset(struct thread_data *);
static inline bool fio_offset_overlap_risk(struct thread_data *td)
{
- if (td->o.ddir_seq_add || (td->o.ddir_seq_nr > 1))
+ if (td->o.norandommap || td->o.softrandommap ||
+ td->o.ddir_seq_add || (td->o.ddir_seq_nr > 1))
return true;
return false;
diff --git a/init.c b/init.c
index feaa3f28..95f2179d 100644
--- a/init.c
+++ b/init.c
@@ -886,13 +886,12 @@ static int fixup_options(struct thread_data *td)
/*
* Disable rand_seed check when we have verify_backlog,
- * zone reset frequency for zonemode=zbd, norandommap, or
- * offset modifiers.
+ * zone reset frequency for zonemode=zbd, or if we are using
+ * an RB tree for IO history logs.
* Unless we were explicitly asked to enable it.
*/
if (!td_write(td) || (td->flags & TD_F_VER_BACKLOG) ||
- o->zrf.u.f || o->norandommap ||
- fio_offset_overlap_risk(td)) {
+ o->zrf.u.f || fio_offset_overlap_risk(td)) {
if (!fio_option_is_set(o, verify_header_seed))
o->verify_header_seed = 0;
}
diff --git a/iolog.c b/iolog.c
index 6b5dc9ab..dcf6083c 100644
--- a/iolog.c
+++ b/iolog.c
@@ -306,7 +306,7 @@ void log_io_piece(struct thread_data *td, struct io_u *io_u)
* the rb insert/lookup for handling. Sort writes if we have offset
* modifier which can also create duplicate blocks.
*/
- if (file_randommap(td, ipo->file) && !fio_offset_overlap_risk(td)) {
+ if (!fio_offset_overlap_risk(td)) {
INIT_FLIST_HEAD(&ipo->list);
flist_add_tail(&ipo->list, &td->io_hist_list);
ipo->flags |= IP_F_ONLIST;
--
2.25.1
