On Fri, Feb 16, 2024 at 10:03:31AM +0100, Andrey Albershteyn wrote:
> On 2024-02-15 17:08:02, Luis Chamberlain wrote:
> > Often times one is running a new test baseline we want to continue to
> > start testing where we left off if the last test was a crash. To do
> > this the first thing that occurred to me was to use the check.time
> > file as an expunge file but that doesn't work so well if you crashed
> > as the file turns out empty.
> >
> > So instead add super simple argument --start-after which let's you
> > skip all tests until the test infrastructure has "seen" the test
> > you want to skip. This does obviously work best if you are not using
> > a random order, but that is rather implied. If you do use a random
> > order --start-after still works, the final output will however just
> > be randomized of course, but it should let you skip a failed known
> > crash at least. The real value to --start-after though is for when
> > you use a non-randomized order.
> >
> > If the target test is not found in your test list we complain and
> > bail. This is not as obvious when you specify groups, so likewise
> > we do a special check when you use groups to ensure the test is at
> > least part of one group.
> >
> > Demo:
> >
> > root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/025
> > Start after test generic/025 not found in any group specified.
> > Be sure you specify a test present in one of your test run groups if using --start-after.
> >
> > Your set of groups have these tests:
> >
> > generic/476 generic/521 generic/522 generic/616 generic/617 generic/642 generic/650
> >
> > root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/522
> > SECTION -- xfs_reflink
> > RECREATING -- xfs on /dev/loop16
> > FSTYP -- xfs (non-debug)
> > PLATFORM -- Linux/x86_64 demo-xfs-reflink 6.5.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.13-1 (2023-11-29)
> > MKFS_OPTIONS -- -f -f -m reflink=1,rmapbt=1, -i sparse=1, /dev/loop5
> > MOUNT_OPTIONS -- /dev/loop5 /media/scratch
> >
> > generic/616
> > generic/617
> > generic/642
> > generic/650
> >
> > Signed-off-by: Luis Chamberlain <mcgrof@xxxxxxxxxx>
> > ---
> >
> > Changes since v1:
> >
> > This all addresses Anand Jain's feedback.
> >
> > - Skip tests completely which are not going to be run
> > - Sanity test to ensure the test is part of a group, if you listed
> > groups, and if not provide a useful output giving the list of all
> > tests in your group so you can know better which one is a valid test
> > to skip
> > - Sanity test to ensure the test you specified is valid
> > - Moves the trim during file processing now using a routine
> > trim_start_after()
> >
> > check | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 52 insertions(+)
> >
> > diff --git a/check b/check
> > index 71b9fbd07522..1c76f33192ba 100755
> > --- a/check
> > +++ b/check
> > @@ -18,6 +18,8 @@ showme=false
> > have_test_arg=false
> > randomize=false
> > exact_order=false
> > +start_after=false
> > +start_after_test=""
> > export here=`pwd`
> > xfile=""
> > subdir_xfile=""
> > @@ -80,6 +82,7 @@ check options
> > -b brief test summary
> > -R fmt[,fmt] generate report in formats specified. Supported formats: xunit, xunit-quiet
> > --large-fs optimise scratch device for large filesystems
> > + --start-after only start testing after the test specified
> > -s section run only specified section from config file
> > -S section exclude the specified section from the config file
> > -L <n> loop tests <n> times following a failure, measuring aggregate pass/fail metrics
> > @@ -120,6 +123,8 @@ examples:
> > check -x stress xfs/*
> > check -X .exclude -g auto
> > check -E ~/.xfstests.exclude
> > + check --start-after btrfs/010
> > + check -n -g soak --start-after generic/522
> > '
> > exit 1
> > }
> > @@ -204,6 +209,24 @@ trim_test_list()
> > rm -f $tmp.grep
> > }
> >
> > +# takes the list of tests to run in $tmp.list and skips all tests until
> > +# the specified test is found. This will ensure the tests start after the
> > +# test specified, it skips the test specified.
> > +trim_start_after()
> > +{
> > + local skip_test="$1"
> > + local starts_regexp=$(echo $skip_test | sed -e 's|\/|\\/|')
> > + local grep_start_after=" | awk 'f;/.*'$starts_regexp'/{f=1}'"
>
> Looks like grep_start_after is not used
> Otherwise, LGTM:
> Reviewed-by: Andrey Albershteyn <aalbersh@xxxxxxxxxx>
Thanks I've made these changes, will send a v3:
diff --git a/check b/check
index 1c76f33192ba..d0bf7ea43870 100755
--- a/check
+++ b/check
@@ -18,7 +18,6 @@ showme=false
have_test_arg=false
randomize=false
exact_order=false
-start_after=false
start_after_test=""
export here=`pwd`
xfile=""
@@ -216,7 +215,6 @@ trim_start_after()
{
local skip_test="$1"
local starts_regexp=$(echo $skip_test | sed -e 's|\/|\\/|')
- local grep_start_after=" | awk 'f;/.*'$starts_regexp'/{f=1}'"
if grep -q $skip_test $tmp.list ; then
rm -f $tmp.grep
@@ -266,7 +264,7 @@ _prepare_test_list()
exit 1
fi
- if [[ $start_after && $start_after_found -ne 1 ]]; then
+ if [[ "$start_after_test" != "" && $start_after_found -ne 1 ]]; then
echo $list | grep -q $start_after_test
if [[ $? -eq 0 ]]; then
start_after_found=1
@@ -278,7 +276,7 @@ _prepare_test_list()
done
group_all="$group_all $list"
done
- if [[ $start_after && $start_after_found -ne 1 ]]; then
+ if [[ "$start_after_test" != "" && $start_after_found -ne 1 ]]; then
group_all=$(echo $group_all | sed -e 's|tests/||g')
echo "Start after test $start_after_test not found in any group specified."
echo "Be sure you specify a test present in one of your test run groups if using --start-after."
@@ -301,7 +299,7 @@ _prepare_test_list()
trim_test_list $list
done
- if [[ $start_after ]]; then
+ if [[ "$start_after_test" != "" ]]; then
trim_start_after $start_after_test
fi
@@ -361,7 +359,10 @@ while [ $# -gt 0 ]; do
fi
;;
--start-after)
- start_after=true
+ if $randomize; then
+ echo "Cannot specify -r and --start-after."
+ exit 1
+ fi
start_after_test="$2"
shift
;;
