On Fri, Jan 10, 2025 at 09:10:26AM +0000, Nirjhar Roy (IBM) wrote:
> This patch adds -q <n> option through which one can run a given test <n>
> times unconditionally. It also prints pass/fail metrics at the end.
>
> The advantage of this over -L <n> and -i/-I <n> is that:
> a. -L <n> will not re-run a flakey test if the test passes for the first time.
> b. -I/-i <n> sets up devices during each iteration and hence slower.
> Note -q <n> will override -L <n>.
This is great! It's something that I've wanted for a while, since at
the moment I implement {gce,kvm}-xfstests -C 10 is to run check ten
times, and doing something which does the looping inside check instead
of outside will be much more efficient.
One other thing that has been on my todo list to update, but which
perhaps you might be willing to do while you are doing work in this
area (nudge, nudge :-), is an optional mode which interates but which
stops once a test fails. This is essentially the reverse of -L, and
the reason why it's useful is when trying to bisect a flakey test,
which sometimes might only be failing 2-5% of the time, require
running a test 30-50 times. But the moment the test fails, we don't
need to run the test any more, so this would speed up bisection tests
which today I do via:
gce-xfstests ltm --repo linux.git --bisect-good v6.12 --bisect-bad \
v6.13-rc1 -C 50 -c ext4/inline_data generic/273
Because of this, I wonder if we should have one option to specify the
number of interations, and then a different option which specifies the
iteration mode, which might be "unconditional", "until first failure",
"only if the test initially fails", etc, instead of separate options
for -q, -L, etc.
Thanks,
- Ted
