On Tue, Apr 16, 2019 at 10:18:11PM -0700, nadav.amit@xxxxxxxxx wrote:
> From: Nadav Amit <nadav.amit@xxxxxxxxx>
>
> Currently, if a test is expected to fail, but surprisingly it passes,
> the test is considered as "failing".
It's not "failing", it's failing. If a test is expected to pass then
it shouldn't be getting reported with report_xfail().
If you know of an xfail test that is passing, and believe it should
be passing now, then please send a patch changing the report_xfail()
to report().
> This means that running on old
> KVM-unit-tests on new KVM versions can falsely report failures.
Why would one want to run old kvm-unit-tests on new kvm? We primarily
care about latest kvm-unit-tests running on latest kvm, but there are
cases where we want to run latest kvm-unit-tests on old kvm, like
when bisecting something with a new test. In some cases that can be
risky, though, if a test case can trigger host crashes. For those cases
we have the errata framework.
Thanks,
drew
>
> Fix it and simplify the logic.
>
> Signed-off-by: Nadav Amit <nadav.amit@xxxxxxxxx>
> ---
> lib/report.c | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/lib/report.c b/lib/report.c
> index ca9b4fd..8c8db14 100644
> --- a/lib/report.c
> +++ b/lib/report.c
> @@ -96,10 +96,12 @@ static void va_report(const char *msg_fmt,
> puts("\n");
> if (skip)
> skipped++;
> - else if (xfail && !pass)
> - xfailures++;
> - else if (xfail || !pass)
> - failures++;
> + else if (!pass) {
> + if (xfail)
> + xfailures++;
> + else
> + failures++;
> + }
>
> spin_unlock(&lock);
> }
> --
> 2.17.1
>
