Thanks Adam for bringing this up, I don't want to give big explanations on the topics, but I just want to mention a few things here: On Tue, Mar 17, 2020 at 2:01 PM Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote: > > So, today is the Fedora 32 Beta release date. If you examine the > release announcement: > > https://fedoramagazine.org/announcing-the-release-of-fedora-32-beta/ > > and the current state of the download site: > > https://getfedora.org/ > > there are three fairly significant things entirely missing. Those three > things are the things that getfedora.org calls our 'Emerging Fedora > Editions' - the things that are allegedly the 'future of Fedora': IoT, > CoreOS and Silverblue. > > This is approximately how I currently understand things regarding those > editions: > > 1) Silverblue - this is kind of the simplest. Silverblue is built as > part of the regular Fedora composes. Silverblue image builds failed in > the actual Beta compose, which was unfortunate but not release-blocking > as SB is not yet tagged release-blocking, so the Beta we signed off has > no SB images. At the go/no-go and readiness meetings, IIRC, we agreed a > rough plan that we would nominate images from a nightly compose built > after the packages in Beta had been pushed stable to release alongside > the Beta. > https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200314.n.0/compose/Silverblue/ > should work fine for that purpose, but it seems not everything > necessary to actually get that plan implemented has happened, at least > not yet - I see no mention of this in the release announcement or on > getfedora.org . I uploaded the Silverblue nigthly to https://dl.fedoraproject.org/pub/alt/unofficial/releases/32/test/ where we store our "unofficial" releases. But that doesn't mean its pointed on getfedora.org, someone has to manually update it in websites and we have currently 0 resources working on websites. relrod is just helping us out on that. > > 2) IoT - as I've been trying to annoy people about for a few weeks now, > IoT is not built as part of our main 'Fedora' composes: > https://pagure.io/fedora-iot/issue/24 > this is a significant difference from just about everything else in > Fedora. We have several of these kinda 'variant' composes, but the > others are only used to do nightly builds of specific images for stable > releases *after* the release happens. For Rawhide and Branched > releases, those images are built in the main 'Fedora' compose. IoT is > the first and only thing (aside from CoreOS...which we'll get to in a > minute) which is being built outside of the 'Fedora' compose for > Branched and Rawhide. This means we don't have IoT images in the Beta > compose or indeed in any of the main Fedora 32 Branched composes and it > falls effectively entirely outside the existing processes for testing > and releasing stuff. That wouldn't be a problem if IoT were some > unimportant thing we didn't care about, but that doesn't seem to be the > case. It also wouldn't be a problem if this had all been figured out > ahead of time and a plan had been put in place for how exactly we were > going to validate and ship IoT releases, but that doesn't seem to have > been the case either. I keep filing tickets and bugging people and > there have been some back channel discussions and things, and a few of > us as individuals are making up a validation process as we go along as > best as we can, but there doesn't seem to be anything on paper aside > from > https://docs.fedoraproject.org/en-US/iot/edition-promotion/ , which > doesn't really address QA and release process stuff much and seems to > be rather old. As far as F32 Beta goes, again we roughly decided at > Go/No-Go and readiness to identify a specific compose that 'matched' > Beta and ship the bits from that - > https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-32-20200314.0/compose/IoT/ > would seem to fit the bill, and pwhalen has done validation testing on > it at > https://fedoraproject.org/wiki/User:Pwhalen/QA/IoT/Fedora-IoT-32-20200314.0 , > but again this plan doesn't seem to have got as far as getfedora.org or > the Beta release announcement. Peter and I talked about it a while ago about merging IoT into regular Fedora composes. But we never got to that point. I am not sure whats their plan moving forward. My idea is to merge them together and keep their composes as it is for their testing. > > 3) CoreOS - CoreOS is just a *whole* other thing. It is not built like > the rest of Fedora at all. It's not built as Pungi composes, whatever > compose process it does have doesn't run alongside our other compose > processes or output to the same locations, it's like a whole other > product. It also doesn't seem to be terribly well documented (I can't > find any detail on the wiki, docs sites, or any of the git repos I > happen to know of) or even introspectable - for instance, ISO locations > look like > https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20200223.3.0/x86_64/fedora-coreos-31.20200223.3.0-live.x86_64.iso > , but if you try and browse around that tree by visiting URLs like > https://builds.coreos.fedoraproject.org/prod/streams/ , you just > get 'Access Denied'. So if the process appears to have broken down and > some poor monkey like me is left trying to help people figure out where > they can get a 'Fedora CoreOS 32' image to try - it isn't very easy. > Right now I simply have no clue if there is something you could > reasonably call 'Fedora CoreOS 32 Beta' or a decent analogue of it. Its a rolling release, I am not sure if we can tag something as F32 Beta, but I will leave that to FCOS team. > > So, in practical terms for Fedora 32 Beta: it would be good if we could > complete the plan of identifying SB and IoT nightly images to place > alongside the Beta release, and update getfedora.org (and maybe the > Beta release announcement) to refer to these. For CoreOS, it would be > good if we could...I don't know...do something? To call something > 'Fedora CoreOS 32 Beta' and let people install it? Or at least > explicitly say that there isn't one for $REASONS? > > In wider terms: I've been bugging specific people/teams about this in > less public channels for a while now, but mattdm said I can be more > annoying about it, so now I'm gonna bug people about it in these more > public channels! For our so-called Emerging Editions (especially IoT > and CoreOS, Silverblue pretty much gets a pass here) - the 'future of > Fedora' - we need better process than this. We need more bureaucracy > than this! > > I know bureaucracy isn't exciting and there are always 'real' problems > to fix instead. I know our existing compose/test/release process is > old-fashioned and unwieldy and monolithic and takes forever. But it has > the advantage that it is a complete, end-to-end, more-or-less-fully- > documented process that all the teams involved understand. The > developers of the existing Editions (and other spins/labs), releng, QA, > websites and communications folks all understand the existing sausage > factory and all the bits are joined up well enough that when we push > the big red button, everything works. The bits get built, the release > validation process kicks in, the bits get tested. The Beta release > announcement talks about Workstation and Server and the spins and labs. > getfedora.org provides download links for them. > > I totally get that the existing process can be constricting and it's > reasonable to want to do things a little differently (IoT) or a lot > differently (CoreOS). But if we're gonna do that *and make these things > important parts of Fedora and not just sandbox projects*, then we need > to start working on making the new way of doing things as well-oiled > and integrated with other teams as the old way of doing things. The new > processes doen't just have to work for the team that are making the > things, they have to work for other teams all the way down the line so > that this kind of situation doesn't happen; there needs to be clarity > on how we are actually going to build and validate and ship and > announce these things. It doesn't have to work the same way we've been > doing it the last few years at all. We can experiment with different > schedules for different editions, we can experiment with different > release approaches that make sense for different products, we can do > all kinds of stuff. But in order to do it properly, we need better > process. > > Some specific ideas: FESCo and/or the Council should be taking a > stronger lead here. I'd view the Fedora.next process as a good model > for this. In Fedora.next there was a really strong model set up for how > we were going to create these 'Editions': every Edition had to have a > PRD and a tech spec, every edition had to be built and tested and > shipped together in the same release process, we had to have release > criteria and validation testing for every edition, and all the editions > had to be presented together with good strong unified branding on > download pages and so on. The whole process of bootstrapping the > editions was extensively tracked with issues and meetings and regular > status updates and all of that kind of stuff. FESCo kept strong lines > of communication between the teams working on the editions and teams > like releng and QA and websites and docs so that everyone understood > what was happening and what the schedules were and what needed to be > done by when. > > This is what I feel like we're missing with these 'emerging Editions'. > I like that as a catch-all for these three things that are obviously > important to Fedora's future, but right now all it seems to be is a > phrase on the download page. There doesn't seem to be a process for how > an 'emerging Edition' becomes a fully-fledged Fedora edition and takes > its place alongside (or replaces one of) our existing ones. I can't, > for instance, find a FESCo or Council ticket somewhere for 'Fedora IoT > becoming an official Edition' where the concrete things necessary to > make that actually happen are defined and tracked. No-one has really > come to a QA meeting and said "hey, so we have this idea to make IoT > and CoreOS fully-fledged, release-blocking Fedora editions in future, > how do you think that would look from a QA perspective? What needs to > happen? Want to come to a FESCo meeting and talk about it?" and then > followed up on it, IIRC. All of that happened for the initial > Fedora.next process. I've had various conversations like that > informally with individual people, but there is no paper trail, little > of it seems to be written down anywhere, and it seems to be very easy > for different people to have different understandings of what's going > on and what the current status of things is and what the future > expectations are. I just went through the last two months of meeting > logs for FESCo - > https://meetbot-raw.fedoraproject.org/teams/fesco/ - and the Council - > https://meetbot-raw.fedoraproject.org/teams/council/ - and unless I'm > missing something, the string 'coreos' doesn't appear one time and the > string 'iot' appears only in the word "Patriots" in a discussion of the > NFL playoffs. This doesn't seem like how things ought to be. > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
