On Tue, Jun 21, 2022 at 06:01:51PM +0200, David Disseldorp wrote: > This RFC patchset adds support to loop on failed tests, as proposed by > Ted Ts'o in https://lwn.net/Articles/897061/: > add a mode that will immediately rerun a failed test 25 or 100 times > to establish a failure percentage. > > There are a couple of things which I'd like to resolve before dropping > the RFC flag, but would appreciate early feedback on the approach here. This is really exciting! I was hoping to try it out, but the first patch doesn't apply to origin/master on xfstests-dev. For example this patch hunk: @@ -729,9 +750,7 @@ function run_section() prev_seq="" for seq in $list ; do # Run report for previous test! - if [ "$tc_status" == "fail" ]; then - bad+=("$seqnum") - fi + _stash_test_status "$seqnum" "$tc_status" if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then _make_testcase_report "$prev_seq" "$tc_status" fi The relevant section of check looks like this: # Run report for previous test! if $err ; then bad="$bad $seqnum" n_bad=`expr $n_bad + 1` tc_status="fail" fi And "git blame" shows that this part of check hasn't changed since 2018, and I'm on the latest version upstream version of xfstests: commit 568ac9fffeb6afec03e5d6c9936617232fd7fc6d (HEAD, tag: v2022.06.05, origin/master, origin/HEAD, kernel/master) Author: Dave Chinner <dchinner@xxxxxxxxxx> Date: Fri Jun 3 11:54:13 2022 +1000 xfs/189: systemd monitoring of /etc/fstab sucks Was your patch based xfstests with some out-of-tree patches? > The caveats are: > - rerun tests will be tracked as a single failure in @try and @bad > + xunit reports do not include any rerun details > - .bad files generated on failure will be overwritten by test reruns > > For xunit reports, I think it'll make sense to stash the aggregates in a > separate <test>.agg-results file or something. Similarly for .bad file > overwrites, I could add a .<rerun #> suffix for capturing all failure > data. For xunit results fie, was assuming that simply we would just have multiple repeated testcase entries stored in the single results.xml file. For example: <testcase classname="xfstests.global" name="generic/476" time="354"> <failure message="Test failed, reason unknown" type="TestFail" /> <system-out> ... </testcase> <testcase classname="xfstests.global" name="generic/476" time="343"> </testcase> <testcase classname="xfstests.global" name="generic/476" time="353"> </testcase> ... I don't know that we need a separate file for the rerun tests, since it's not that hard to create a python script which parses the results and calculates the pass/fail percentages for any test which is run mutiple times. As far as haivng the .bad and .full files, I agree that some kind of .rerun-NN suffix would make a lot of sense. Cheers, - Ted
