On Fri, Jul 13, 2018 at 04:40:39PM -0400, Jeff Mahoney wrote:
> On 7/13/18 12:44 PM, Luis R. Chamberlain wrote:
> > On Fri, Jul 13, 2018 at 11:39:55AM +0300, Amir Goldstein wrote:
> >> On Fri, Jul 13, 2018 at 5:43 AM, Luis R. Chamberlain <mcgrof@xxxxxxxxxx> wrote:
> >>> I had volunteered at the last LSF/MM to help with the stable work for
> >>> XFS. To help with this, as part of this year's SUSE Hackweek, I've
> >>> first generalized my own set of scripts to help track a baseline of
> >>> results from fstests [0], and extended it to be able to easily ramp up
> >>> with fstests on different distributions, and I've also created a
> >>> respective baseline of results against these distributions as a
> >>> further example of how these scripts and wrapper framework can be used
> >>
> >> Hi Luis!
> >>
> >> Thanks a lot for doing this work!
> >>
> >> Will take me some time to try it out, but see some questions below...
> >>
> >>> [1]. The distributions currently supported are:
> >>>
> >>> * Debian testing
> >>> * OpenSUSE Leap 15.0
> >>> * Fedora 28
> >>>
> >>> The stable work starts with creating a baseline for v4.17.3. The
> >>> results are visible as a result of expunge files which categorize the
> >>> failures for the different sections tested.
> >>
> >> So the only "bad" indication is a test failure?
> >
> > That is correct to a certain degree, ie, if xfsprogs / the kernel
> > config could run it we can assume it passed.
> >
> >> How about indication about a test that started to pass since baseline?
> >
> > Indeed, that is desirable.
> >
> > We have a few options. One is share the entire results directory for
> > a release / section, however this is rather big. For instance for a
> > full v4.17.3 run this is about 292 MiB alone. I don't think this
> > scales. IMHO lgogs should only be supplied onto bug reports, not this
> > framework.
> >
> > The other option is to use -R xunit to generate the report in the
> > specified unit. I have not yet run this, or tried it, however IIRC
> > it does record success runs? Does it also keep logs? Hopefully not. I'm
> > assuming it does not as of yet. I should note if one hits CTRL-C in the
> > middle one does not get the results. An alternative was being worked on
> > by Jeff which would sprinkle IIRC .ok files for tests which succeed,
> > then you could just scrape the results directory to determine which
> > tests did pass -- but you run into the same size problem as above.
>
> Eryu didn't like that idea, so I abandoned it. What I have now is a -R
> files mode that creates a bunch of files with the goal of just archiving
> the results for later comparison or import into a results db.
>
> For each test, there are:
> $seq.result.start.txt - start timestamp
> $seq.result.stop.txt - stop timestamp
> $seq.result.result.txt - simple result: pass/fail/expunged/notrun
> $seq.result.detail.txt - contains the contents of $seq.notrun/$seq.expunged
> $seq.result.{dmesg,kmemleak,full,check}.txt - contains the contents of
> the corresponding files
This is sexy, it also gives the person interpretting the results to
opt-in or not for the actuall full log of the output. You pick and
choose what info you want.
This is indeed nice.
> As an aside, IIRC, -R xunit doesn't catch all kinds of failures. Also,
> as you mentioned, if it's interrupted, all results are lost. This makes
> it difficult to identify test failures that crashed or hung the test system.
OK so indeed not my preference.
> I have some basic scripts that parse the output and generate an HTML
> report/table (and it does do what Amir asks WRT tests that started passing).
These scripts, are they for parsing your new -R files output?
I take it the patches are still being worked on?
Luis
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
