On Tue, Mar 24, 2020 at 5:33 AM Aleksandra Fedorova <alpha@xxxxxxxxxxxx> wrote: > > https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose > I've finally had an opportunity to digest this change proposal, and I've got some overall questions and feedback. > == Summary == > > The goal of the ELN project is to continuously build Fedora Rawhide > packages and composes in the way which resembles the CentOS and RHEL > build process and to provide a feedback loop for Fedora maintainers on > the status of those builds. > > == Owner == > > * Name: [[User:Bookwar| Aleksandra Fedorova]] > * Email: alpha@xxxxxxxxxxxx > * Name: [[User:Sgallagh | Stephen Gallagher]] > * Email: sgallagh@xxxxxxxxxx > > == Detailed Description == > > This Change supersedes the previously-approved [[Changes/Additional > buildroot to test x86-64 micro-architecture update|Change]] to enable > an additional buildroot. During development, its scope has expanded to > include the entire process of how the Fedora sources are built and > composed into shippable artifacts. > > ELN is an evolution of the request for an alternate buildroot for > newer x86_64 processors. The reasoning behind that new buildroot was > that we expected that the next major release of RHEL would likely drop > support for older hardware and therefore could take advantage of > enhancements and processor extensions available for newer hardware. As > plans for this proceeded, they expanded into a desire to do more than > just test out the processor architecture. Instead, we want to have a > complete alternative compose of Fedora Rawhide that resembles the way > that Red Hat and CentOS builds their packages. The idea being that > Fedora developers and third-party vendors who rely on Red Hat > Enterprise Linux have a place where they can directly contribute to > what will eventually become the next RHEL. > I'd like to point out here that if your proposal on x86_64 changes for ELN are anything like the ones proposed before that were rejected in Fedora, I literally would have nearly zero servers that can run ELN software. Server lifetimes are long, and depending on what you're optimizing for, you're not as likely to have all the newest whiz-bang "high end" instructions that Intel puts in their CPUs. And don't forget that workstations are even worse off in corporate environments. Remember that Intel segments instruction availability across CPU lines and price points. You *really* don't want to alienate your *paying customers* by having them find out all their hardware is no good or that you're effectively raising the TCO by a significant degree by forcing "high end" instructions. > ELN (Enterprise Linux Next) is going to be that place. What we want to > do is establish a set of RPM conditionals that can be used for the set > of packages that may need to be built differently for a RHEL-like > target as compared to a Fedora one. (For example, we may want to skip > regenerating documentation during package-build and instead use > pre-built docs from the tarball or SRPM.) > > This includes: > > * buildroot configuration, rpm macro and compile flags, > * comps files and the compose content, > * compose itself and the pipeline which builds it. > > Under this umbrella we are going to have: > * RPM variables for conditionals to allow tweaking rpm spec files > * A new Koji tag which allows control of the buildroot, > * A fork of the pungi configuration which describes the compose configuration, > * A pipeline which builds a periodic compose based on that configuration, > * Storage for such composes and additional pipelines which will verify > the quality of the compose and test additional scenarios on top of it. > > The RPM variables: > > * `%{dist}` will return `.eln` (no numerical suffix) > * `%{fedora}` will return `.fcXX` (where XX is the Fedora version > represented by Rawhide). > * `%{rhel}` will be set and will return a number one higher than the > most recent major release of Red Hat Enterprise Linux (at present, > that will be `9`). > * `%{eln}` will be set and will expand to `%{rhel}` > > The reasoning behind this approach is so that we break as little as > possible when we implement this new build and compose. Existing > packages that are using conditionals such as `if 0%{?rhel} > 7` will > transition over to building "like RHEL" automatically. Any packages > that do not have a shared Fedora/RHEL specfile will continue to build > as normal. Some unknown number of packages that rely on `if ! > 0%{?fedora}` will require manual intervention. > > The `%{eln}` variable will be set primarily to allow for the > possibility of needing conditionals that apply to ELN only and should > not carry over to RHEL, but we expect its use to be exceptionally > rare. > This sounds interesting.... > == Benefit to Fedora == > > Who benefits from the implementation of this feature: > * Fedora Infrastructure > ** As we are going to try and test new, more easily maintainable > infrastructure for composes. This is weird and fuzzy, what do you mean by that? Do you have a concrete proposal of infrastructure changes you want to test? > * Fedora Minimization > ** As we are going to modify the compose content according to > minimization goals and provide feedback to Fedora maintainers on the > results Uhh, what? What does this mean? > * CentOS Stream, EPEL and RHEL > ** We are going to build Fedora packages into a compose similar to the > multi-repo structure of CentOS and RHEL. OK, this actually makes sense... > * Fedora Community > ** The feedback pipelines built for the project will allow downstream > developers to open up their work and bring it closer to Fedora. In > particular, it will enable projects like OpenShift to do their work in > Fedora. ... and this doesn't. Nothing has ever stopped any project from using Fedora other than their own lack of desire to do so. And as a member of the OKD Working Group, I've brought this point up when the CentOS CoreOS was starting to be discussed. In fact, in all the discussions I've had across oVirt, RDO, and OpenShift, every single group has said the problem with Fedora *for them* is that it isn't slow/stale enough for them. Some of them have also brought up the point that they are not given the necessary bandwidth to be able to integrate with the Fedora community and do proper maintenance of their stacks. Consequently, we wound up in situations like with RDO, where they were massively behind on bringup for the next RHEL. They were mostly caught flat-footed on moving to Python 3 and are struggling because they never bothered to incrementally evolve like those that were integrated into Fedora (GlusterFS and Ceph) did. This Change proposal solves none of those issues. > ** Making Fedora friendlier to its downstream will justify bringing > more resources and more contributions to Fedora Rawhide. So that > downstream not only tries to catch up with Fedora going forward, but > actively participates in Fedora development and testing. I don't see anything in here that concretely makes Rawhide more attractive. Could you please explain how this would improve it over the current situation? > ** The tooling developed for this project will allow further > development of changes in Fedora Infrastructure without blocking the > regular Fedora packaging and release process. > Uhh, what? Again, nothing in here seems to indicate this is true. > == Scope == > * Proposal owners: > ** Setup build environment for a new disttag (`eln`). I think an unversioned disttag is asking for trouble. Koji's NVR uniqueness constraint will bite us on this over time. Can we make this something like .eln.elXX? There's going to have to be an redhat-eln-release package anyway that defines these things, and it's better to be versioned upfront to avoid future potential insanity. > ** Setup automation to trigger new ELN build every time there is a new > update submitted to Fedora Rawhide. > ** Post build result to Fedora Messaging, so that it appears in Bodhi > and can be used as a > [https://docs.fedoraproject.org/en-US/rawhide-gating/ gating test]. Wouldn't the build AMQP message be a consequence of a build happening from the trigger anyway? And if the gating flows you've been working on for Rawhide are working, this should just be a natural consequence of things. So this could be simplified to "run Fedora Rawhide submission and gating process". > ** Setup automation to run periodic partial composes (via ODCS) > without installation media to generate repositories with these > packages. What the heck is ODCS? And why no install media if you want this to be useful to people? > ** Update packaging documentation to mention new disttag and how it can be used. > > * Other developers: > *: Anyone who wants to produce different content for the ELN compose > will need to implement the new macros in their specfile. The > overwhelming majority of packages will require no modification. > > * Release engineering: [https://pagure.io/releng/issue/9154 #9154] > > * Policies and guidelines: > [https://pagure.io/packaging-committee/issue/955 #955] > > * Trademark approval: not needed. There is no new shippable artifact. > Nope. Something needs to ship. Otherwise, how would all these downstreams you list as beneficiaries be able to do anything with it? > == Upgrade/compatibility impact == > > N/A > > Though it is a System Wide change it doesn’t impact Fedora upgrades, > as it is a development environment tracking Fedora Rawhide. > Actually, I would suggest we should consider noting that EL8 -> ELN upgrades should be considered here, since this an attempt to integrate rhel-rawhide into Fedora. > == How To Test == > > There is going to be a development snapshot of the state of the ELN > repositories. > > It is under discussion whether this snapshot will have its own > installation media. For now the preferred way to test ELN composes > would be to use standard Fedora Rawhide images and then include ELN as > an additional repository. > This makes no sense. A cursory evaluation of what this Change entails means that under no point would this "ELN" tree be able to be validated as a freestanding tree if this remained the case. If this is intended to be properly useful, it should contain the full scope of deliverables, including installation media. It should become eventually possible to use OpenQA and everything else we have against ELN and validate it accordingly. > == User Experience == > > There is no user-facing part in this change. No ELN artifacts are > going to be shipped to the end-user. > This is not okay. Without a shipped component, nobody can do anything with it. > == Dependencies == > > There is no dependency on other changes, though some of the > infrastructure services like ODCS composes and Fedora CI will be > involved. > That sounds like dependencies? Please elaborate on the specifics here. Dependencies are not just about Change proposals, but components, infrastructure, and people too. > == Contingency Plan == > > * Contingency mechanism: If this effort doesn’t land in Fedora 33 we > review it and decide whether it needs to be moved further to Fedora 34 > or cancelled. > * Contingency deadline: N/A as it is a continuous process in Fedora > Rawhide and it is not pinned to a specific release. > * Blocks release? No > * Blocks product? No > > Additional risk: Infrastructure resources taken by ELN builds can > create load on Fedora build system > This is absolutely guaranteed. Especially since you plan to schedule rebuilds automatically. > Mitigation: In case of such load we cancel or reschedule ELN tasks > (package and compose builds and tests) on request from the Fedora > Infrastructure team. At least initially, we will also set ELN tasks to > low-priority, similar to how COPR does it. > This mitigation strategy will not work for the same reason that it didn't work for MBS. It's not the priority of the tasks, it's the quantity. In another thread, I did the calculation of how much builder resources we have, and as it currently stands, we're being pushed to the limits with 3~4 distro releases and module streams for each of those, plus three addon distros (EPEL). And I'm not including the Koschei stuff and the non-RPM builds we do (Flatpaks, containers, disk images, ISOs...)! Please come up with a plan on how to scale up Fedora build infrastructure to handle the additional demand. I'm not saying implement said plan immediately, but we need a strategy to add more resources to build infrastructure sooner rather than later. This combined with the Fedora CI work and various auto-rebuild strategies already in place will likely push us past reasonable performance thresholds for the infrastructure, and I'd rather us be ready for that. > == Documentation == > > https://fedoraproject.org/wiki/ELN > > See also previous discussions: > * "Alternative buildroot as a development tool" @ devel > https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/3Z5SMF6CPL3MK2CNYPUW4OZYMA6TZJBN/#MKXPMQOZNLECPPWRM6UBUMR2WHVG5DL6 > > == Release Notes == > > N/A > > Though it is a System-Wide Change it has no user-facing component. We > may announce it through other channels. > What's the point if there's no user-facing component? This seems fundamentally broken. How do people use it? Consume and build on it? I know that's not the aim, but from my perspective as a Fedora contributor, package maintainer, and ISV developer, I *should* be attracted to this Change. But I'm not, because it feels like it's written for Red Hat people to move their sandbox outside of Red Hat into Fedora without anyone else to be able to play in the sandbox. The number of times it was indicated that there are no installation media or end-user deliverables just reinforces this perception. Please realize that I *want* to be excited by this. So give me something to be excited about! :) -- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
