On Thu, 13 Feb 2025 at 10:13, Dan Horák <dan@xxxxxxxx> wrote:
On Thu, 13 Feb 2025 09:32:06 +0100
Clement Verna <cverna@xxxxxxxxxxxxxxxxx> wrote:
> cross posting from
> https://discussion.fedoraproject.org/t/gating-fedora-updates-on-fedora-coreos-ci/144566
>
> Hi all,
>
> Last year, the Fedora CoreOS working group implemented CI testing [1] for
> Bodhi updates on a set of critical packages [2]. Automatic updates are a
> key feature of Fedora CoreOS, and this testing helps us detect update
> related issues early, improving Fedora’s update stability and reducing
> troubleshooting time.
>
> While our long-term goal is to implement this CI testing in fedora-bootc
> with Bodhi gating integration, there's still significant work ahead before
> we can trigger fedora-bootc tests on Bodhi updates. It's worth noting that
> many of the tests currently running in Fedora CoreOS CI are essentially
> "image mode" tests rather than CoreOS-specific tests. Eventually, we expect
> to migrate these tests to fedora-bootc. However, until that infrastructure
> is ready, enabling gating on the FCOS suite provides immediate image mode
> coverage for critical packages.
>
> Given our experience running these tests, we would like to propose making
> the coreos.cosa.build-and-test a required gate for package updates in
> rawhide. We've already been successfully gating packages owned by the
> Fedora CoreOS working group [3], and we'd like to extend this requirement
> to the broader package set defined here [4].
>
> Following is the breakdown of passed vs failed builds by package on over
> 400 builds, this gives package maintainers an idea of how often an update
> might be gated. It is important to note that not all test failures here are
> related to the software in the proposed Bodhi update since there could be
> flakes; either due to the test infra environment or due to some transient
> test pipeline misconfiguration. In the case where failures are not related
> to updates , it would be easy to waive the test or coordinate with the
> Fedora CoreOS working group to disable the test.
gating based on flaky tests or flaky infra is a no-go, sorry ... You
should define an "acceptable false positive" rate first (1%?, 2%?), then
fix tests and infra and then think about gating. Even when half of the
presented failures are not caused by the package under test, it's too
much.
We are starting this conversation because we have good confidence that we flake and infrastructure failure are a minority of cases.
Theses test are running on the Fedora Infrastructure (OpenShift cluster) , so not a dedicated infra that would need something special to be fixed and our test framework has also features to make the tests resistant to flakes (we are re-runing failing tests) and we are also able to easily snooze tests for a period of time if needed. If with all of this a critical update is still blocked because of a false positive, it is fairly easy to waive the test in Bodhi (https://docs.fedoraproject.org/en-US/rawhide-gating/faq/#_how_do_i_unblock_an_update)
Our tests have caught regressions in the past which have landed in Fedora and affected Fedora users not just Fedora CoreOS users, so we believe that there is a lot of value in making these tests blocking.
Dan
--
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
