Sounds like a great addition Adam! Just to double check: if you have not enabled gating, then openQA will not be run at all? Cheers, Dan On June 9, 2022 7:48:29 PM UTC, Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote: >Hi folks! > >We've had openQA testing of updates for stable and branched releases, >and gating based on those tests, enabled for a while now. I believe >this is going quite well, and I think we addressed the issues reported >when we first enabled gating - Bodhi's gating status updates work more >smoothly now, and openQA respects Bodhi's "re-run tests" button so >failed tests can be re-triggered. > >A few weeks ago, I enabled testing of Rawhide updates in the openQA >lab/stg instance. This was to see how smoothly the tests run, how often >we run into unexpected failures or problems, and whether the hardware >resources we have are sufficient for the extra load. > >So far this has been going more smoothly than I anticipated, if >anything. The workers seem to keep up with the test load, even though >one out of three worker systems for the stg instance is currently out >of commission (we're using it to investigate a bug). We do get >occasional failures which seem to be related to Rawhide kernel slowness >(e.g. operations timing out that usually don't otherwise time out), but >on the whole, the level of false failures is (I would say) acceptably >low, enough that my current regime of checking the test results daily >and restarting failed ones that don't seem to indicate a real bug >should be sufficient. > >So, I'd like to propose that we enable Rawhide update testing on the >production openQA instance also. This would cause results to appear on >the Automated Tests tab in Bodhi, but they would be only informational >(and unless the update was gated by a CI test, or somehow otherwise >configured not to be pushed automatically, updates would continue to be >pushed 'stable' almost immediately on creation, regardless of the >openQA results). > >More significantly, I'd also propose that we turn on gating on openQA >results for Rawhide updates. This would mean Rawhide updates would be >held from going 'stable' (and included in the next compose) until the >gating openQA tests had run and passed. We may want to do this a bit >after turning on the tests; perhaps Fedora 37 branch point would be a >natural time to do it. > >Currently this would usually mean a wait from update submission to >'stable push' (which really means that the build goes into the >buildroot, and will go into the next Rawhide compose when it happens) >of somewhere between 45 minutes and a couple of hours. It would also >mean that if Rawhide updates for inter-dependent packages are not >correctly grouped, the dependent update(s) will fail testing and be >gated until the update they depend on has passed testing and been >pushed. The tests for the dependent update(s) would then need to be re- >run, either by someone hitting the button in Bodhi or an openQA admin >noticing and restarting them, before the dependent update(s) could be >pushed. > >In the worst case, if updated packages A and B both need the other to >work correctly but the updates are submitted separately, both updates >may fail tests and be blocked. This could only be resolved by waiving >the failures, or replacing the separate updates with an update >containing both packages. > >All of those considerations are already true for stable and branched >releases, but people are probably more used to grouping updates for >stable and branched than doing it for Rawhide, and the typical flow of >going from a build to an update provides more opportunity to create >grouped updates for branched/stable. For Rawhide the easiest way to do >it if you need to do it is to do the builds in a side tag and use >Bodhi's ability to create updates from a side tag. > >As with branched/stable, only critical path updates would have the >tests run and be gated on the results. Non-critpath updates would be >unaffected. (There's a small allowlist of non-critpath packages for >which the tests are also run, but they are not currently gated on the >results). > >I think doing this could really help us keep Rawhide solid and avoid >introducing major compose-breaking bugs, at minimal cost. But it's a >significant change and I wanted to see what folks think. In particular, >if you find the existing gating of updates for stable/branched releases >to cause problems in any way, I'd love to hear about it. > >Thanks folks! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
