On Fri, Jun 24, 2022 at 12:42:03AM -0400, Theodore Ts'o wrote: > On Tue, Jun 21, 2022 at 06:01:51PM +0200, David Disseldorp wrote: > > This RFC patchset adds support to loop on failed tests, as proposed by > > Ted Ts'o in https://lwn.net/Articles/897061/: > > add a mode that will immediately rerun a failed test 25 or 100 times > > to establish a failure percentage. > > > > There are a couple of things which I'd like to resolve before dropping > > the RFC flag, but would appreciate early feedback on the approach here. > > This is really exciting! I was hoping to try it out, but the first > patch doesn't apply to origin/master on xfstests-dev. Hi Ted, The origin/for-next has latest (testing) update. But for this issue you hit below, you might need this patchset: https://lore.kernel.org/fstests/20220620192934.21694-1-ddiss@xxxxxxx/ David sent that at first, to fix/change something before having this feature. I've acked that, and will merge it this week. For this patchset, I'd like to give it more time to get more reivew points. Thanks, Zorro > > For example this patch hunk: > > @@ -729,9 +750,7 @@ function run_section() > prev_seq="" > for seq in $list ; do > # Run report for previous test! > - if [ "$tc_status" == "fail" ]; then > - bad+=("$seqnum") > - fi > + _stash_test_status "$seqnum" "$tc_status" > if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then > _make_testcase_report "$prev_seq" "$tc_status" > fi > > The relevant section of check looks like this: > > # Run report for previous test! > if $err ; then > bad="$bad $seqnum" > n_bad=`expr $n_bad + 1` > tc_status="fail" > fi > > And "git blame" shows that this part of check hasn't changed since > 2018, and I'm on the latest version upstream version of xfstests: > > commit 568ac9fffeb6afec03e5d6c9936617232fd7fc6d (HEAD, tag: v2022.06.05, origin/master, origin/HEAD, kernel/master) > Author: Dave Chinner <dchinner@xxxxxxxxxx> > Date: Fri Jun 3 11:54:13 2022 +1000 > > xfs/189: systemd monitoring of /etc/fstab sucks > > > Was your patch based xfstests with some out-of-tree patches? > > > The caveats are: > > - rerun tests will be tracked as a single failure in @try and @bad > > + xunit reports do not include any rerun details > > - .bad files generated on failure will be overwritten by test reruns > > > > For xunit reports, I think it'll make sense to stash the aggregates in a > > separate <test>.agg-results file or something. Similarly for .bad file > > overwrites, I could add a .<rerun #> suffix for capturing all failure > > data. > > For xunit results fie, was assuming that simply we would just have > multiple repeated testcase entries stored in the single results.xml > file. For example: > > <testcase classname="xfstests.global" name="generic/476" time="354"> > <failure message="Test failed, reason unknown" type="TestFail" /> > <system-out> > ... > </testcase> > <testcase classname="xfstests.global" name="generic/476" time="343"> > </testcase> > <testcase classname="xfstests.global" name="generic/476" time="353"> > </testcase> > ... > > I don't know that we need a separate file for the rerun tests, since > it's not that hard to create a python script which parses the results > and calculates the pass/fail percentages for any test which is run > mutiple times. > > > As far as haivng the .bad and .full files, I agree that some kind of > .rerun-NN suffix would make a lot of sense. > > Cheers, > > - Ted >
