----- Original Message ----- > From: "Stephen Gallagher" <sgallagh@xxxxxxxxxx> > To: devel-announce@xxxxxxxxxxxxxxxxxxxxxxx, "Development discussions related to Fedora" > <devel@xxxxxxxxxxxxxxxxxxxxxxx> > Sent: Tuesday, March 31, 2020 5:31:38 PM > Subject: Fedora 33 System-Wide Change Proposal: ELN Buildroot and Compose V3 > > I sent out the V2 version of the Change on Friday and then promptly > managed to injure myself and be away from email until today. I've read > through the email threads again this morning and I decided that, > rather than try to address them one by one, I'd try again with a V3 > that hopefully answers some of the repeated questions and concerns on > that list. > > Please see the newly-updated > https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose > for more details[1]. > > > [1] > https://fedoraproject.org/w/index.php?title=Changes%2FELN_Buildroot_and_Compose&type=revision&diff=569904&oldid=569809 > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx > Ok I took the time to read all the previous threads and the proposal as well as the clarifications. In general I am in favor of the idea, however I concur with my fellow packagers, that the current proposal is having some serious issues, and I'm very reluctant in agreeing with the given rationale of various details. I also feel that I am iterating a lot of the feedback that was given already however I believe it was neither heard nor taken into account. So to start: > ELN (Enterprise Linux Next) is going to be that place. What we want to do is establish a set of RPM conditionals that can be used for the set of packages that may need to be built differently for a RHEL-like target as compared to a Fedora one. (For example, Fedora often ships with experimental or less-commonly-used features enabled for packages that Red Hat would not want to support for Enterprise Linux customers). Some packagers are fine with adding conditionals in their SPECs and that is ok. However as one of the maintainers of the main python interpreter as well as various alternative ones, and hundreds of libraries, not only throughout Fedora but across the RHEL ecosystem, I can say that this change vastly complicates my work. As a team we maintain approximately 35+ different branches of various python interpreters and if it was making sense to have single SPECs by using conditionals, we would pretty much do it without hassle. Different projects/products have differents needs and different goals (and different branches but more on that later). I could just add RHEL and SCL conditionals however not only it would clutter the SPEC file to insane amounts of unnecessary craft, the only people who would know how to work with it, would be me and my team. I want to encourage contributions from the community at my packages, not make them more complex and deter others from contributing. So in that respect, in Fedora I wouldn't personally add any RHEL (or ELN) conditionals, as our RHEL work can vary significantly. And it would drive potential collaborators away if every change visible in Fedora, would also have an effect on a black box that would be ELN. Why would they even bother if they don't have access to that? And what's their motivation to even work with those conditionals? Am I supposed, for example, to reject PR's that could break ELN due to its different compiler flags, but have no effect on Fedora, from a user and a contributor working to better a package on our OS? Also contributors *are* users of Fedora and driving them away, may very well drive a portion of our userbase away as well. """ As a sister effort to this, the OSCI team will also be implementing automated tests that will run against ELN composes and provide non-blocking feedback to the package maintainers about potential issues in the Enterprise Linux configuration. """ I don't like this part either. Feedback is still feedback blocking or not. Seeing a big red sign that a change broke ELN is just nagging people to fix issues that might not even affect them in any significant way. I would like the ELN testing feedback to be opt-in for packagers who do not maintain something in ELN or RHEL and only the actual RHEL maintainer to get the notification. If not, that is unnecessary noise. """ The reasoning behind this approach is so that we break as little as possible when we implement this new build and compose. Existing packages that are using conditionals such as if 0%{?rhel} > 7 will transition over to building "like RHEL" automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue to build as normal. There will be some unknown number of packages whose existing conditionals will need to be updated to account for ELN. At present, we are estimating that there will be around 200 packages that need such attention. The ELN SIG will be providing guidance and pull-requests to assist with this. """ The number of packages here is very optimistic and frankly quite unreal. Having worked on mass scale package changes, from Python rebases, to decoupling python2 from RHEL8 I can assure you this number would much much higher, in my opinion at least. Guidance and pull requests, while helpful, shouldn't be required for packagers who don't want them, whatever their reasons might be. """ The %{eln} variable will be set primarily to allow for the possibility of needing conditionals that apply to ELN only and should not carry over to RHEL, but we expect its use to be exceptionally rare. """ I don't understand this part. So the %{eln} macro will be in Fedora and ELN, but not actually RHEL? " Such fixes will be done via a pull request that states a time limit. We want the regular maintainers to see / comment / commit, but we also don't want things to stall for months and get forgotten about. For less trivial fixes, we may provide a pull-request or use other methods to communicate with the maintainers to determine a path forwards that is acceptable to them. """ """ As long as your package builds in ELN then just maintain your package like normal. If there is a build failure, the ELN SIG may provide a PR as described above or will discuss alternative approaches on an individual basis with you. The ELN SIG will not dictate how you must proceed, but will try to find a solution that you are comfortable with. """ It is reasonable to provide a pull request to fix potential issues with ELN. What is not reasonable, is to impose a time limit and also expect the maintainers to follow up with that. And what exactly would be the way forward for packagers that do not want those conditionals in their package, while it also fails to build in ELN, potentially blocking other packages? Would you merge it as a proven packager, despite possible drawbacks? I can easily see a game of revertions if that was to happen. So please explain what other possible approaches there would be in that case, and I hope it is not to actually convince the packager to accept the conditionals. """ This adds no value to the current approach where Red Hat maintainers would manually merge their changes into the internal build infrastructure. There's no way to automate the sync from the master branch to the eln branch that wouldn't break and require maintainer involvement. Attempting to branch only individual packages would introduce significant complexity in the build process as well, leading to far more opportunity for bugs. Lastly, even the most diligent of maintainers can forget to sync every change to a new branch, thus leaving us in a situation where the eln branch has fallen behind and is no longer providing an accurate view of whether the package is still building or functioning in that environment. """ Automatically synced branches or symbolic references to branches is a great solution to that, and I really like the idea that Fabio proposed to provide an optional eln branch for packagers who either do not want to merge the changes, or if the change is too complex to make sense for Fedora. """ Classically, Fedora sees a flurry of activity from Red Hat developers in the run-up to a new Red Hat Enterprise Linux release. These Red Hat teams frantically attempt to push a few years' downstream effort up into Fedora before it is branched for the next RHEL development. It is not uncommon for these teams to disappear downstream again once this has happened. This harms both Fedora and Red Hat with the lack of upstream involvement. With ELN, we hope to make Fedora Rawhide a more appealing (and canonical) place for those developers to work. This should increase the amount of ongoing developer involvement in Fedora, and also make Fedora a more valuable resource for Red Hat, which can help to ensure funding and support. """ That is true to an extend. I don't see that as a failure with the procedures though, more like a failure to communicate to those teams/developers, to stress just how important Fedora is for our downstream work. And honestly I haven't interacted with many developers who don't think Fedora is important enough, so that point from my experience is a bit moot, in respect to pushing such an intrusive change. I'm obviously biased here, but looking, as an example at the state of the Python, Ruby, Perl ecosystem across Fedora and RHEL, you can see the Fedora work is not only important, but the main drive for a healthy RHEL ecosystem. And I'm happy to say my management understands that. """ ELN artifacts will be made available for testing and development purposes, but we will not be shipping any content produced from ELN directly to the general public (such as on the standard mirror network or via getfedora.org). This does not necessarily mean that they will be shipped directly from Koji, merely that they won't come from the standard locations. """ It would be better to provide artifacts, there is no reason to keep this in a black box, maybe I'd like to spin VM's, maybe containers. Just updating a regular rawhide installation to eln in order to test a simple fix is not something I would do very enthusiastically. Some more thoughts: Since the current Fedora infrastructure is already stretched thin, especially with modularity, koschei rebuilds, the CI and so on, has a possible impact on the load been considered/measured? I know for starters that the tasks will have a lower priority, however has any assessment been made to get a rough idea on what and if any additional resources will be required? I'd also like to point out that all the past system-wide changes, except from maybe rebases of important packages, have always had detailed descriptions on how they were going to be implemented and a well laid-out contingency plan. They weren't mere ideas, and while I understand that not all questions can be answered at this point, starting with a very high level goal and iterating on it, is not something to be done as a fedora system-wide change, not at least since some details need to be cemented on the aforementioned points. A system-wide change and the fedora devel thread of it, is a place to gather feedback from the community, incorporate that and move forward from there, not somewhere where blind trust should be placed on the change owners and accuse people and even elected representatives of ignoring feedback. It really does seem like projection when placing the whole conversation into context. I understand that especially during those trying times, everyone might be a bit on edge while trying to stay productive and within the deadlines, however it really does seem that the change owners and the community are talking entirely past each other at this point. Some time ago, during the discussion of the default modular streams, there was a big pushback on some aspects of it from the community and the feedback was rejected and/or ignored in a similar manner that happens throughout those ELN threads. In the end, the eclipse module did exactly that and it affected a big portion of our userbase: https://bugzilla.redhat.com/show_bug.cgi?id=1780827 So overall, I'm -1 on the proposal and as things currently stand, I won't be adding any of those conditionals at the Python SPEC files or any other smaller library that I maintain. I'll happily reconsider if the feedback is heard and the pain points have been addressed or are at least planned to be addressed. P.S. It took me some days to actually draft this mail, as in general those times, emotions can run wild and I wanted to be objective, if that's even possible. And I understand that some things might have changed as the thread progressed, so apologies if any of my points are not valid anymore. -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
