On 3/5/24 08:53, Thomas Weber wrote:
Hi all,
I noticed a change in the reported throughput when switching from fio version 3.17 to 3.36.
I narrowed it down to fio-3.26 which seem to introduce the throughput change.
The exemplary fio workload attached gives me different results since then.
Was that change intended?
Thanks Thomas
fio-3.25:
Run status group 0 (all jobs):
READ: bw=3658MiB/s (3835MB/s), 3658MiB/s-3658MiB/s (3835MB/s-3835MB/s), io=429GiB (460GB), run=120001-120001msec
WRITE: bw=2438MiB/s (2557MB/s), 2438MiB/s-2438MiB/s (2557MB/s-2557MB/s), io=286GiB (307GB), run=120001-120001msec
fio-3.26:
Run status group 0 (all jobs):
READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=55.5GiB (59.6GB), run=120001-120001msec
WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-332MB/s), io=37.1GiB (39.9GB), run=120001-120001msec
FIO_FREEPARM1=description=na FIO_FREEPARM2=description=na FIO_FREEPARM3=description=na FIO_FREEPARM4=runtime=120 FIO_FREEPARM5=time_based FIO_TARGETSPEC=directory=./fsstress_data FIO_NUMJOB=1 FIO_BS=64k FIO_SIZE=1g FIO_DIRECT=0 FIO_IOENGINE=sync FIO_IODEPTH=1 fio ./fs-stress2.fiojob --output-format=normal,terse --terse-version=3 --warnings-fatal --eta=never
fs-stress2.fiojob file:
[global]
ioengine=${FIO_IOENGINE}
iodepth=${FIO_IODEPTH}
size=${FIO_SIZE}
direct=${FIO_DIRECT}
numjobs=${FIO_NUMJOB}
${FIO_TARGETSPEC}
# free parameters that can be overwritten including the var= specifier
${FIO_FREEPARM1}
${FIO_FREEPARM2}
${FIO_FREEPARM3}
${FIO_FREEPARM4}
${FIO_FREEPARM5}
# environment setup
norandommap
allrandrepeat=1
startdelay=50ms-2s
ramp_time=5s
[smallstuff-seq-slow]
rw=rw
nrfiles=1000
openfiles=25
filesize=1k-1m
bssplit=4k/20:128k/80
rwmixread=60
There was a bug fix in fio-3.26 which seems relevant to your configuration:
https://github.com/axboe/fio/commit/4ef1562a013513fd0a0048cca4048f28d308a90f
commit 4ef1562a013513fd0a0048cca4048f28d308a90f
Author: Adam Kupczyk <akupczyk@xxxxxxxxxx>
Date: Mon Dec 28 14:55:17 2020 +0100
io_u: Fix bad interaction with --openfiles and non-sequential file
selection policy
Problem happens when --openfiles is set and file_service_type !=
FIO_FSERVICE_SEQ.
In function __get_next_file, we decrement file_service_left and if
0, we select
next file to operate on.
However, get_next_file_rand can return -EBUSY if too many files are
already opened,
and __get_next_file exits with error.
In next invocation of __get_next_file, we decrement
file_service_left again (from 0),
wrapping around to 2^32-1, effectively locking __get_next_file to
always select the same.
Algorithm to observe bad behavior:
fio --randseed=1 --ioengine=libaio --rw=randwrite --nrfiles=256
--bs=4k --size=256m \
--loops=50 --allow_file_create=1 --write_iolog=log.txt
--file_service_type=normal:20 \
--filename_format=object.\$filenum --name=x --openfiles=100
cat log.txt |grep write |cut -f 1 -d " " |sort |uniq -c | sort -n |
sed "s/[.]/ /" \
| while read a b c; do echo $c $b $a; done |sort -n
....
70 object 17
71 object 19
72 object 22
73 object 65296
74 object 65255
75 object 33
76 object 27
77 object 25
78 object 65243
79 object 36
80 object 49
81 object 47
....
Signed-off-by: Adam Kupczyk <akupczyk@xxxxxxxxxx>
