On Wed, 2020-12-02 at 09:23 -0500, Ben Cotton wrote: > > == How To Test == > See QA test cases : https://fedoraproject.org/wiki/Category:CoreOS_Test_Cases > > We also have regular tests days, for example > https://fedoramagazine.org/fedora-coreos-test-day/ So...yeah, that's not really enough to call something a Fedora Edition :) I think this has a lot of the issues we had with IoT, but turned up to 11. We have an entire process for releasing a thing called "Fedora" which is based around the idea that all the bits that make up a "Fedora release" get built, tested, and signed off together. IoT stretched this process a bit, because it's not actually built as part of the same composes as all the other "Fedora" bits. So we had to implement an entire parallel validation process just for IoT composes. But at least it's built *the same way* as Fedora, so we could more or less just split existing things into two paths and use them, which is what we did. We now have both mainline and IoT composes and validation events. Even there, the process wasn't perfect for Fedora 33; if you look at the log of the Fedora 33 Go/No-Go meeting[1], which is supposed to be where we go over the status of the bits-to-be-released in great detail and decide whether to sign them off, there is precisely one line about IoT: 17:58:36 <pwhalen> IoT is also covered - https://fedoraproject.org/wiki/Test_Results:Fedora-IoT_33_RC_20201020.0_General#ISO_Install ...no-one else said a word about it. So yeah, we clearly don't have this whole "releasing from multiple streams" entirely down yet. CoreOS is going to be the same only worse, because it's not even built the same way as the rest of Fedora. It's not built by Pungi, we don't get the same messages published when CoreOS builds happen (we don't get messages published at all, IIRC), the metadata for CoreOS builds is not in productmd[2] form like the metadata for Pungi builds, the whole process is entirely different. So to boil this down into a representative question: when we are doing the Fedora 34 Go/No-Go meeting in ~four months' time, how do we decide whether to release "Fedora CoreOS 34"? What does the question "is Fedora CoreOS 34 ready to go" even mean, in the context of how CoreOS is built and released? What set of bits will we be deciding to ship or not ship, and how will that have been decided and communicated? Where will we look to find the test results and criteria on which we would base that decision? Are any of these silly questions, which would indicate that our release process is based on assumptions which need to be fundamentally re- examined as part of this Change? All of this is stuff we could kind of handwave so long as CoreOS wasn't an official Edition; we could *more or less* leave the CoreOS team to decide what a CoreOS release looked like and when it would happen and when it was good enough to ship, and so on. But if we're going to call it an Edition, which is one of the primary things that defines what Fedora *is*, I don't think we can do that any more. We need more groups to think about and decide on the answers to questions like the above. [1]: https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2020-10-22/f33-final-go_no_go-meeting.2020-10-22-17.00.log.html [2]: https://github.com/release-engineering/productmd -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx
