"Theodore Ts'o" <tytso@xxxxxxx> writes:
> On Fri, Jan 10, 2025 at 09:10:26AM +0000, Nirjhar Roy (IBM) wrote:
>> This patch adds -q <n> option through which one can run a given test <n>
>> times unconditionally. It also prints pass/fail metrics at the end.
>>
>> The advantage of this over -L <n> and -i/-I <n> is that:
>> a. -L <n> will not re-run a flakey test if the test passes for the first time.
>> b. -I/-i <n> sets up devices during each iteration and hence slower.
>> Note -q <n> will override -L <n>.
>
> This is great! It's something that I've wanted for a while, since at
> the moment I implement {gce,kvm}-xfstests -C 10 is to run check ten
> times, and doing something which does the looping inside check instead
> of outside will be much more efficient.
Yup "-q 10" could be used which can give pass/fail metrics of how many
times the test passed v/s failed by doing unconditional looping within
xfstest's check script itself.
>
> One other thing that has been on my todo list to update, but which
> perhaps you might be willing to do while you are doing work in this
> area (nudge, nudge :-), is an optional mode which interates but which
> stops once a test fails. This is essentially the reverse of -L, and
> the reason why it's useful is when trying to bisect a flakey test,
> which sometimes might only be failing 2-5% of the time, require
> running a test 30-50 times. But the moment the test fails, we don't
> need to run the test any more, so this would speed up bisection tests
> which today I do via:
>
> gce-xfstests ltm --repo linux.git --bisect-good v6.12 --bisect-bad \
> v6.13-rc1 -C 50 -c ext4/inline_data generic/273
There is -I which stops iterating on encountering a failure.
Does that work for this usecase?
>
> Because of this, I wonder if we should have one option to specify the
> number of interations, and then a different option which specifies the
> iteration mode, which might be "unconditional", "until first failure",
> "only if the test initially fails", etc, instead of separate options
> for -q, -L, etc.
I like the idea of iteration mode here. I will let others determine on
how easy it is to kill any option in xfstests today and replace it with
other. However here is a summary of different iteration and looping
options.
We have 3 options in xfstests today:
1. -i <n> iterate the test list <n> times
2. -I <n> iterate the test list <n> times, but stops iterating further in case of any test failure
3. -L <n> loop tests <n> times following a failure, measuring aggregate pass/fail metrics
So we have -i/-I which are iterations and -q/-L which are loops.
Looping happens when we can just loop over a particular test <n> times
and give pass/fail metrics.
Whereas in case of iterations it goes over from the beginning which will
also source different rc/config files and prepares the device etc. (hence
it's a bit slow too)
Using -q will be faster over using -i or -I similar to how -L is faster. However -L
will only re-run when there is a failure the 1st time.
-q v/s -l: Can we kill current -l option if it is not in use by anyone?
I would prefer -l since it looks a short form for looping. I don't see
we using -l anymore. But I will let others comment.
-l line mode diff
-ritesh
