On Wed, 20 Feb 2019 at 14:51, Jakub Jelinek <jakub@xxxxxxxxxx> wrote:
[..]
The aim is to avoid regressions, not zero FAIL rate, which e.g. for the
guality testcase is not really possible as the testing matrix is too large
for debug info coverage, -O levels x targets x ISA choices x GDB issues
and it is impossible to encode that into xfails.
Plus, the rpm builds test both normally and with additional
-fstack-protector-strong as that is what the whole distro does, while
normally the latter isn't included. So, some FAILs might be once with
normal options and once with -fstack-protector-strong, other tests only FAIL
with the latter e.g. when testing asan and having UB in the test to test it
how asan handles it and -fstack-protector-strong may stand in a way of that.
There are some flakey tests too (mostly in the go testsuite).
So, I have a directory with years of testsuite results from koji/brew
builds, essentially build logs from all the architectures massaged through
sed -n -e '/^==*TESTING/,/^==*TESTING END/p'
and compare that regularly for new builds.
OK. Jakub so what kind of method you are using to recognise that something is wrong with gcc/gdb/binutils if none of those packages test suites seems may exit with non-zero exit code and some failures are perfectly OK?
(Or maybe logic of the suite should be negated?)
Do you have some sort of script/tool extracting results from build logs? or it is some other method?
Syntax of the content /^==*TESTING/,/^==*TESTING END/p' looks a bit complicated ..
How to trace those results in more or less automated way?
kloczek
--
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
