On Tue, 2024-03-05 at 12:11 -0500, Vincent Fu wrote:
> On 3/5/24 08:53, Thomas Weber wrote:
> > Hi all,
> >
> > I noticed a change in the reported throughput when switching from
> > fio version 3.17 to 3.36.
> >
> > I narrowed it down to fio-3.26 which seem to introduce the
> > throughput change.
> >
> > The exemplary fio workload attached gives me different results
> > since then.
> >
> > Was that change intended?
> >
> > Thanks Thomas
> >
> > fio-3.25:
> > Run status group 0 (all jobs):
> > READ: bw=3658MiB/s (3835MB/s), 3658MiB/s-3658MiB/s (3835MB/s-
> > 3835MB/s), io=429GiB (460GB), run=120001-120001msec
> > WRITE: bw=2438MiB/s (2557MB/s), 2438MiB/s-2438MiB/s (2557MB/s-
> > 2557MB/s), io=286GiB (307GB), run=120001-120001msec
> >
> > fio-3.26:
> > Run status group 0 (all jobs):
> > READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-
> > 497MB/s), io=55.5GiB (59.6GB), run=120001-120001msec
> > WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-
> > 332MB/s), io=37.1GiB (39.9GB), run=120001-120001msec
> >
> >
> > FIO_FREEPARM1=description=na FIO_FREEPARM2=description=na
> > FIO_FREEPARM3=description=na FIO_FREEPARM4=runtime=120
> > FIO_FREEPARM5=time_based FIO_TARGETSPEC=directory=./fsstress_data
> > FIO_NUMJOB=1 FIO_BS=64k FIO_SIZE=1g FIO_DIRECT=0 FIO_IOENGINE=sync
> > FIO_IODEPTH=1 fio ./fs-stress2.fiojob --output-format=normal,terse
> > --terse-version=3 --warnings-fatal --eta=never
> >
> > fs-stress2.fiojob file:
> > [global]
> > ioengine=${FIO_IOENGINE}
> > iodepth=${FIO_IODEPTH}
> > size=${FIO_SIZE}
> > direct=${FIO_DIRECT}
> > numjobs=${FIO_NUMJOB}
> > ${FIO_TARGETSPEC}
> >
> > # free parameters that can be overwritten including the var=
> > specifier
> > ${FIO_FREEPARM1}
> > ${FIO_FREEPARM2}
> > ${FIO_FREEPARM3}
> > ${FIO_FREEPARM4}
> > ${FIO_FREEPARM5}
> >
> > # environment setup
> > norandommap
> > allrandrepeat=1
> > startdelay=50ms-2s
> > ramp_time=5s
> >
> > [smallstuff-seq-slow]
> > rw=rw
> > nrfiles=1000
> > openfiles=25
> > filesize=1k-1m
> > bssplit=4k/20:128k/80
> > rwmixread=60
> >
>
> There was a bug fix in fio-3.26 which seems relevant to your
> configuration:
>
>
https://github.com/axboe/fio/commit/4ef1562a013513fd0a0048cca4048f28d308a90f
>
>
> commit 4ef1562a013513fd0a0048cca4048f28d308a90f
> Author: Adam Kupczyk <
> akupczyk@xxxxxxxxxx
> >
> Date: Mon Dec 28 14:55:17 2020 +0100
>
> io_u: Fix bad interaction with --openfiles and non-sequential
> file
> selection policy
>
> Problem happens when --openfiles is set and file_service_type
> !=
> FIO_FSERVICE_SEQ.
> In function __get_next_file, we decrement file_service_left and
> if
> 0, we select
> next file to operate on.
> However, get_next_file_rand can return -EBUSY if too many files
> are
> already opened,
> and __get_next_file exits with error.
>
> In next invocation of __get_next_file, we decrement
> file_service_left again (from 0),
> wrapping around to 2^32-1, effectively locking __get_next_file
> to
> always select the same.
>
> Algorithm to observe bad behavior:
> fio --randseed=1 --ioengine=libaio --rw=randwrite --nrfiles=256
> --bs=4k --size=256m \
> --loops=50 --allow_file_create=1 --write_iolog=log.txt
> --file_service_type=normal:20 \
> --filename_format=object.\$filenum --name=x --openfiles=100
>
> cat log.txt |grep write |cut -f 1 -d " " |sort |uniq -c | sort
> -n |
> sed "s/[.]/ /" \
> | while read a b c; do echo $c $b $a; done |sort -n
> ....
> 70 object 17
> 71 object 19
> 72 object 22
> 73 object 65296
> 74 object 65255
> 75 object 33
> 76 object 27
> 77 object 25
> 78 object 65243
> 79 object 36
> 80 object 49
> 81 object 47
> ....
>
> Signed-off-by: Adam Kupczyk <
> akupczyk@xxxxxxxxxx
> >
>
Thank you. I noticed that fix as well in the fio changelog.
>From the description it seems to affect only non-sequential IO.
However I saw huge throughput degradations for both rw=rw and
rw=randrw.
Do you think this is related to that bug fix too?
[smallstuff-seq-slow]
rw=rw
nrfiles=1000
openfiles=25
filesize=1k-1m
bssplit=4k/20:128k/80
rwmixread=60
[smallstuff-rand-slow]
rw=randrw
nrfiles=1000
openfiles=25
filesize=1k-1m
bssplit=4k/50:8k/25:64k/25
rwmixread=75
