On Fri, Jul 13, 2018 at 09:44:20AM -0700, Luis R. Chamberlain wrote:
>
> We have a few options. One is share the entire results directory for
> a release / section, however this is rather big. For instance for a
> full v4.17.3 run this is about 292 MiB alone. I don't think this
> scales. IMHO lgogs should only be supplied onto bug reports, not this
> framework.
The results directory compress fairly well. A complete ext4 run (with
the configurations defined for gce-xfstests) is 101MB uncompressed,
and 2.3 MB as a tar.xz file. That's only 6 cents a month in Google
Cloud Storage, and for me it's worth it to keep them; it's
occasionally been interesting.
> The other option is to use -R xunit to generate the report in the
> specified unit. I have not yet run this, or tried it, however IIRC
> it does record success runs?
Yes, it does.
> Does it also keep logs? Hopefully not.
It does include some information, but not all of the information that
might be in the results directory.
% ls /tmp/results-ltm-20180709000722/ab/ext4/results-1k/results.xml
108 /tmp/results-ltm-20180709000722/ab/ext4/results-1k/results.xml
vs
% du -sh /tmp/results-ltm-20180709000722/ab/ext4/results-1k
3.0M /tmp/results-ltm-20180709000722/ab/ext4/results-1k
> Having the expunge files separately helps as we can annotate bug URLs to
> them optionally. Ie, we should be able to process both expunge lists
> and xunit file to construct a nice db schema to process results
> in a more easily viewable manner in the future.
I do this. You can see an example here with my annotations:
https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/root/fs/ext4/exclude
> So to establish a baseline, one first manually contstructs the expunge
> files needed to run a full test. In the future hopefully we can have
> a set of scripts to do all this for us.
Yep, I do this by using a command-line option when I run gce-xfstests
or kvm-xfstests to skip using the exclude files. It's definitely
useful.
> We'd have to then process that file to scrape out which tests were
> passed, if a user wanted that. Do we have scripts for processing
> xunit files?
I have some scripts which you may find useful. They can be found here:
https://github.com/tytso/xfstests-bld/tree/master/kvm-xfstests/test-appliance/files/usr/local/bin
https://github.com/tytso/xfstests-bld/tree/master/kvm-xfstests/test-appliance/files/usr/lib/python2.7
This is what I use to parse through the xunit xml files to generate
summaries like this:
ext4/4k: 430 tests, 1 failures, 43 skipped, 6565 seconds
Failures: generic/388
ext4/1k: 441 tests, 7 failures, 55 skipped, 7985 seconds
Failures: ext4/033 generic/018 generic/383 generic/388 generic/454
generic/475 generic/476
ext4/encrypt: 495 tests, 121 skipped, 4081 seconds
ext4/nojournal: 472 tests, 1 failures, 88 skipped, 4700 seconds
Failures: ext4/301
ext4/ext3conv: 429 tests, 1 failures, 43 skipped, 5965 seconds
Failures: generic/388
ext4/adv: 434 tests, 2 failures, 49 skipped, 5142 seconds
Failures: generic/399 generic/477
ext4/dioread_nolock: 429 tests, 1 failures, 43 skipped, 5870 seconds
Failures: generic/388
ext4/data_journal: 476 tests, 2 failures, 91 skipped, 6832 seconds
Failures: generic/388 generic/475
ext4/bigalloc: 414 tests, 11 failures, 50 skipped, 6963 seconds
Failures: ext4/033 generic/204 generic/219 generic/235 generic/273
generic/388 generic/456 generic/472 generic/494 generic/495
generic/496
ext4/bigalloc_1k: 428 tests, 11 failures, 64 skipped, 5458 seconds
Failures: ext4/033 generic/204 generic/235 generic/273 generic/383
generic/388 generic/454 generic/472 generic/494 generic/495
generic/496
Totals: 3801 tests, 647 skipped, 37 failures, 0 errors, 59147s
> > Tested that started to notrun since baseline?
>
> Its unclear if xunit captures this. Otherwise we have some work to do.
There are software packages that will process xunit XML files, store
them into a database and then generate reports against a defined
baseline. They'll also do fancy graphs, and some of them will show
flakey tests, etc. I haven't had time to investigate them, though,
but if you do find some cool tools to process the xunit files, I'd
definitely be interested.
Cheers,
- Ted
