On Sat, 15 Feb 2025 at 20:51, Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote:
On Sat, 2025-02-15 at 14:54 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> On Fri, Feb 14, 2025 at 02:40:29PM -0800, Adam Williamson wrote:
> > On Fri, 2025-02-14 at 16:31 -0500, Dusty Mabe wrote:
> > > IMO the bar would only need to be that high if the user had no way to ignore the test results.
> > > All gating does here (IIUC) is require them to do an extra step before it automatically flows
> > > into the next rawhide compose.
> >
> > again, technically, yes, but *please* let's not train people to have a
> > pavlovian reaction to waive failures, that is not the way.
>
> IMO, the bar for *gating* tests needs to be high. I think 95% true
> positives would be a reasonable threshold.
Do you mean 95% of failures must be 'real' (i.e. up to 5% can be
'false')? Is this after automatic retries and manual intervention by
the test system maintainers, or before?
Off the top of my head, 95% seems low. I'm pretty sure we do better
than that with openQA and people would complain if that was all we
managed. We usually maintain a 0% false failure rate after auto-retries
and <24h manual intervention -
https://openqa.fedoraproject.org/group_overview/2?limit_builds=100&limit_builds=400
has 0 false failures ATM.
Thanks, that's interesting. What do you call <24h manual intervention? One example that comes to mind would be to disable or snooze a test that started to trigger false failures in under 24h.
If that's the case, I think that sounds achievable.
If that's the case, I think that sounds achievable.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@xxxxxxxxxxxxx
https://www.happyassassin.net
--
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
