On Wed, May 27, 2015 at 11:37 AM, Adam Miller <maxamillion@xxxxxxxxxxxxxxxxx> wrote: > On Thu, May 21, 2015 at 4:46 PM, Matthew Miller > <mattdm@xxxxxxxxxxxxxxxxx> wrote: >> On Thu, May 21, 2015 at 04:14:51PM -0500, Dennis Gilmore wrote: >>> > 1. F22 trees built nightly (already happening) >>> there is new atomic trees built as part of the package updates >>> process managed by bodhi. >> >> Cool. >> >>> > 2. Images built from those nightly (I think right now only rawhide?) >>> once f22 is final only rawhide. >> >> What would it take to make those happen post-release? > > If we could get a clearly defined list of what these items are I would > be happy to work towards getting them done. I just don't know how that > all needs to fit into current processes right now. This is definitely > something I'd like to at least have outlined before the Rel-Eng FAD[0] > so maybe some progress could be made before then and/or during. > > I do have an open pull request to enable two week ISO composes with > the rel-eng scripts but it's still being reviewed and iterated on[1]. > > I also submitted a patch to the pungi 4 dev branch to enable atomic > ISOs[2] be built as a koji task once we migrate to pungi4 and can take > advantage of that capability. > >> >>> > 3. Images go through automated tests, automatically (tunir; not fully >>> > automatic yet, right?) >>> there is zero automated testing available, but we really need it, afaik it is >>> all manual >> >> See <https://fedoraproject.org/wiki/Changes/tunir>. Plus glorious >> Taskotron future, I'm sure. :) > > I'm not sure where in the phases this should fit, it seems like a > giant forklift of work to me given that the base Fedora distro that > Atomic is going to be pulling it's bits from doesn't even have this > functionality. Am I mistaken or possibly missing something? (Please > correct me if I am) > >> >>> > 4. Every two weeks, latest image to pass those tests gets linked >>> > on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE) >>> > - probably need a manual override >>> > - page will present these images with appropriate setting of >>> > expectations ("fedora qa not to blame for anything here"). >>> > - need something to happen if there aren't any that pass for >>> > two weeks, which should not happen I hope, but, y'know) >>> > * automatic message about image being out of date >>> > * next successful image automatically posted? Or do we >>> > skip two weeks? >>> > - also needs: commitment to follow and fix if things break >>> We need people to test, and a process to move things through stages. >>> build all the images, etc. a location to put things, i,e lots of >>> things >> > tl;dr - Two week Atomic builds is currently blocked but we're working on it. I wanted to follow up on this particular point. Recently at the Fedora Activity Day for Release Engineering[0] we started to dig into the Pungi4 code that was recently released into the wild. The unfortunate data point is that nothing works right now, we are chewing through it the best we can but this is all new code to almost everyone on the team (myself included). The test cases currently pass entirely on the set of dummy rpms that are fed to the tests but as soon as we open the flood gates of real Fedora content this fails in new and exciting ways. These runs take over an hour per test so I'm going to focus on how to trim that down so we're iterating on a few hundred packages in a compose instead of 10s of thousands. It's high on my personal priority list to have this resolved so that we can enable Fedora Atomic as well as be more flexible with other things. On a similar note, the koji run-root plugin is making it's way to production soon (already in stage and testing is looking good) so I have high hopes there. These two things combined will enable the Atomic builds as well as eventually enabling the pxe-to-live stuff. I'm not sure how much people actually care about this since I never got a response to my previous email. However, I did at least want to follow up with a status update because this is something I have a personal interest in as I'm a big fan of Atomic and really want to make sure it succeeds. -AdamM [0] - https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015 > This largely piggy backs off my previous point. Automatic "graduation" > will require a decent amount of work and probably commitment from the > QA group. I won't speak to their capacity to cater to this but I think > what is ultimately being asked of them should be brought to their > attention as early as possible to get their input. > >> Let's work at building the full list of lots of things. If we want to >> have people testing, I'd say that'd go something like: >> >> - every two weeks, latest image to pass is flagged as needing human >> testing, a la a package update in bodhi >> - some sort of karma system — maybe it _is_ bodhi, but don't want to >> block on that > > If it were to be Bodhi, would the Atomic tree (or image) then be just > another build artifact pushed through and sync'd like an rpm? So for > at least the near term, it would require human intervention of > submitting an update? > >> - enough +1s, it goes to the atomic.fpo page >> - maybe expectation that new image is flagged on Monday, passed by... >> Wednesday? Something like that. >> >> Alternately, we could skip that, or slot it into a later phase, and >> just say "these images are auto-generated". >> >> What process is missing for building the images? >> >> For location, I guess >> <https://dl.fedoraproject.org/pub/alt/fedora-atomic/images/>? Maybe >> with separate subdirs for "all", "passed autotest", "passed human >> tests" — and/or a subdir for the two-week "releases" separate from the >> nightlies. >> >>> >>> > Phase II >>> > >>> > 5. Ability to include packages from a side tag, repo, something. >>> > All full, legit Fedora packages destined for main repo but maybe at >>> > a different speed. (Implementation needs work) >>> > - possibly initially just selected packages from updates-testing? >>> really we just need to push things stable. if we do updates-testing composes. >>> we should be able to get the testing done and pushed through. there there is >>> no need to go off and do special things >> >> Atomic devs have expressed a strong worry about the lack of something >> like this, unless I'm misunderstanding. Possibly an alternate image >> including updates-testing would suffice (or, maybe, is actually >> separately necessary, since atomic doesn't let individual testers just >> pull in individual RPMs to test). > > If I remember correctly, there are also concerns about things ending > up in Atomic trees that actually never see the light of day in Fedora > because they might never make it to stable. What's the policy on that > sort of thing? (Or is there one? I suspect this could be uncharted > territory because Atomic is effectively a new Operating System product > with it's own lifecycle aiming to operate within the Fedora umbrella) >> >>> > Phase III >>> > 6. Something to only expose Atomic users to _update sets_ that pass the >>> > tests (because, hey, we have tests!) >>> it would be most ideal to have things signed off as good. we kinda >>> have it for updates with bodhi, we likely need something new or to >>> extend something existing. >> >> It's kind of different from the Bodhi, in that these will be >> integration tests rather than per-package tests. >> >> I'm also worried about sending updates through a human-intervention >> gate _twice_ will be a big bottleneck. > > I suspect that if/when the testing automation is in place the double > gate wouldn't be needed, especially if this is just considered a > development release. But I think that's where part of the problem lies > is that the Atomic dev teams wants to use a stable OS as something > that is re-rev'd with components updated and replaced but still call > it "Fedora $VERSION" when in fact, it is not the same as what others > will get from Fedora $VERSION unless they install from either the > Atomic image or this dev tree. (depending on what the decisions are > there) > >> >>> > 7. Presumably, there will be a switch to F23 as base at F23 release; >>> > will there be overlap (beta images) to make that smooth? >>> we probably should do all supported releases and rawhide >> >> I know Atomic devs have previously suggested wanting to encourage >> people to stay on $current, at least while the thing is under such >> rapid development. Possibly the right thing here is to generate >> nightlies for all but point the phase I two-week-selection process at >> only $current. >> > > I basically covered this already, but short version is that $current > is no longer $current as soon as you ship things that aren't actually > a part of $current as the distro is currently defined. If this is > something that lands within the scope of the Rings concept such that > what the Atomic devs want to rip/replace is outside of the central > Ring and the outer rings have flexibility then I suspect that would be > fine, but that's for the Council to decide. However given that > Anaconda and systemd might need a rebase that might not be an option. > > The Fedora Rel-Eng process needs a solid process definition documented > for the scope of a full release somewhere so that we can then look at > how Atomic can work within that scope. I fear there are going to be a > lot of places they will collide since they are effectively two > different distros, composed of almost the same content, but with > completely different release cycles and cadences. I know Rawhide has > been listed as something that poses challenges but it honestly feels > more well suited for something that wants to move this fast, I just > think it's a hard sell to say "that thing that is cut every 6 months, > I want to re-rev it every 2 weeks but it still be the 'same'" > > I might be off base and if I am please feel free to explain to me why I am. > >> >>> > 8. What else? >>> > >>> > Phase IV >>> > >>> > 9. Expand tests beyond tunir — test installs on hardware, etc. >>> test everything :) including other parts of fedora. automated tests for cloud >>> images would be good also. I fear we currently do not do anywhere near the >>> testing that we should. >> >> Absolutely. > > +1 > >> >>> >>> > 10. Maybe create a new, special Fedora repository for packages >>> > with version skew mainline Fedora — e.g. newer systemd >>> > or hold older Docker version for some reasons. (Up for >>> > discussion — that's why this is in phase 3!) >>> We could do so, I would want to see a really good reason to do so. I >>> would hope that what is done for one part of fedora does not break or >>> have negative impacts on other parts of fedora, so just pushing the >>> updates in should be good. making rawhide more of a place to be, for >>> me should be our goal. get the new awesome things in there and used >>> by people, leave the stable releases stable >> >> The concern I'm hearing is that rawhide is hard to develop on, because >> parts other than what you're working on are may change from underneath >> you too quicky. So, there's a desire to develop on top of the stable >> release — but since Atomic isn't really an "on top of" thing entirely, >> it might need new systemd or something that the mainline isn't ready >> for. This area _definitely_ needs clear details, I agree. > > Basically this again. If anything I think if Fedora Atomic wants to be > it's own product then it should exist slightly differently such that > it has it's own set of koji tags, inherits from $current, but > subplants the components it needs. This would however be a massive > shift because as it turns out that is something that's never been done > before and I think this would require a whole new Bodhi stream since > it's effectively a new product//release. If it's not Rawhide but it's > not Fedora $current, then It's basically it's own Distro that's just > based on Fedora and if it were to exist within the Fedora umbrella > then I think there need to be policy changes and that's going to be a > giant pile of other work to sort out how that's all going to fit. > > Again, I might be off in the weeds but that's just how it plays out in my head. > > Looking forward to comments. > > -AdamM > > [0] - https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015 > [1] - https://lists.fedoraproject.org/pipermail/rel-eng/2015-April/019822.html > >> >> >> Thanks Dennis! >> >> -- >> Matthew Miller >> <mattdm@xxxxxxxxxxxxxxxxx> >> Fedora Project Leader >> _______________________________________________ >> cloud mailing list >> cloud@xxxxxxxxxxxxxxxxxxxxxxx >> https://admin.fedoraproject.org/mailman/listinfo/cloud >> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct _______________________________________________ cloud mailing list cloud@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
