On Fri, Mar 3, 2023 at 5:42 AM Daniel P. Berrangé <berrange@xxxxxxxxxx> wrote: > > A few weeks back there was a little snafu (since resolved) with the > capstone package in EPEL. > > This package is being introduced to RHEL 9.2, and thus has been added > to CentOS 9 Stream repos. A bug was filed against capstone in EPEL > to notify that capstone should be retired when RHEL 9.2 is released: > > https://bugzilla.redhat.com/show_bug.cgi?id=2124181 > > Unfortunately the package was accidentally retired immediately rather > than waiting until 9.2 release. This is an easy mistake for a maintainer > to make when they see a bug whose title is "Remove capstone from epel9", > so I don't want anyone to blame the maintainer for this accident. > > Yes the comments make it clear that removal should wait until 9.2 > release date, but I can totally understand how actions will be guided > by the initial bug subject if you're just skimming. It was pointed out > that this risk has been identified before: > > https://lists.fedoraproject.org/archives/list/epel-devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/LUWUIN5BRPJZGVB5PIH625PFV6IBD6YO/ > > and now we have a real world example of it going wrong exactly as > was predicted. > > > My bigger concern though is what happens when RHEL-9.2 is actually > released for real. The bug remains open and targets (immediate) > retirement of capstone from EPEL on / near 9.2 release date. > > > This doesn't feel like it takes into account the real world usage of > enterprise distributions. The kind of people/organizations who deploy > RHEL are rarely going to rush into upgrading their deployment processes > to use 9.2 immediately. They may have all sorts of review and testing > processes to go through, integration of 3rd party apps and services, > validating against hardware, before they consider starting deployment > of RHEL 9.2. > > All this time they will continue deploying machines with 9.1, despite > 9.2 being available, so won't get access to new 9.2 packages. The only responsible way to pin to a minor version is with RHEL EUS. Anything else is just running an outdated OS with unpatched security vulnerabilities. EUS is meant as a stop-gap to keep getting security patches until you can upgrade to the next minor version. My understanding is that it is not recommended for new deployments. For everyone else that is pinning their OS version without updates, I think it's acceptable to have a bit of friction associated with this anti-pattern. Users that choose to stay behind have multiple options to obtain the package. They can manually download the RPM from CentOS Stream Koji, Fedora Koji, or the EPEL snapshot archive. They can also clone the distgit repo, roll back the retirement commit, and build the RPM themselves. Having easier access to the package is a good incentive to get onto the current minor version of the OS. > > If we immediately retire capstone when 9.2 is released, aren't we > going to break any ability to keep deploying capstone from EPEL on > existing/new 9.1 installs ? EPEL has never accommodated older minor versions, EUS or otherwise. There are many EPEL packages that already do not install on older minor versions. When a RHEL minor version library change results in an EPEL package being uninstallable, the maintainer typically rebuilds the package quickly to fix installs for users on the current minor version. This in turn quickly breaks the installation for users pinned on an older minor version. The current model does not allow us to have it both ways. > > There is also the case of the RHEL rebuilds whose users consume EPEL > packages. Depending how quick they are, the rebuild distros might not > have their 9.2 rebuild ready for some days/weeks/months after RHEL-9.2 > is first available. My projects' upstream CI is all based on AlmaLinux > and I don't want to see it broken again by premature capstone retirement > from EPEL. Historically, when CentOS was a rebuild, many EPEL maintainers would wait for corresponding CentOS rebuild release before changing their EPEL packages to work on RHEL. This was true both for soname rebuilds and retirements. CentOS would usually take about a month to catch up to RHEL minor versions. The new rebuilds are doing much better in this area. Alma is routinely getting their minor versions out 1-2 days after RHEL. The other rebuilds aren't far behind. If we were to delay package retirements, I don't think it's necessary to delay for more than a few days. > > Unless I'm mistaken in some way, it feels like we should *NOT* sync > the removal of packages from EPEL with their release in RHEL 9.2. It > needs to be synced with user deployment of 9.2, which might be somewhat > later. This "user deployment" is not a concrete thing we can target. For every user that delays updating, many more are updating at their first opportunity in order to stay current. > > > > The package added to RHEL should (must) have a newer NEVR than the > existing package in EPEL, so even if we leave it in EPEL, it will > be ignored if the newer version is present in RHEL repos. The current "retire immediately" stance originated at a time when RHEL did less coordination with EPEL, and there was no CentOS Stream to publicly see what was planned for the next minor version. It was common for RHEL to choose an older NVR than what was available in EPEL, or for the EPEL maintainer to continue to do version/release bumps after a version had been selected to bake into the next RHEL minor version. Either way resulted in EPEL having a higher NVR than what ended up in RHEL. This necessitated quick retirement so that users that were on the current RHEL minor versions got the RHEL package, not the EPEL package. EPEL's core principle is to only be extra packages and not disturb or replace RHEL packages. We also are not able to mandate what RHEL ships. If they choose to ship foo 2.5, but EPEL has been shipping foo 3.1, foo still must be retired from EPEL. > > The main important thing I see is that the EPEL branch in dist-git > must be made read-only and/or prevented from submitting new builds, > once the package has been built in CentOS Stream. ie to ensure the > EPEL NEVR doesn't creep back above the pending RHEL package's NEVR. As far as I know we don't currently have a mechanism to do this in the current structure. The only mechanism to prevent changes to the distgit branch and prevent new builds is retirement, which also removes the package from the repo. > > > If that's done it should be fine to leave it in EPEL for a prolonged > period after 9.2 is released, to allow sufficient time for users to > actually switch from deploying 9.1 over to 9.2. > > I guess someone might say that EPEL only targets the very latest > y-stream of RHEL, so continuing to use 9.1 after 9.2 is released > is unsupported ? Even if that is technically the case, I think > in practice users won't take that into account, and thus expect > it to keep working and generally that should be fine. Only if a > EPEL package is rebuilt such that it depends on a newly added > API/ABI is that a problem. Other than nitpicking over the over-loaded term "support", I would say that's accurate. We don't have to theorize about this, as this already happens in practice. Oftentimes EPEL packages install correctly on older minor versions. When they don't, we see users file bugs or ask in communication channels about the error they see. They're informed that EPEL only targets the current minor versions,and then directed towards the koji history, the archive snapshots, or the spec file to build it themselves. It's a thing that happens periodically, and I wouldn't characterize it as overwhelming. Typically the response is "oh yeah I know I need to get updated to the current minor version". > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > _______________________________________________ > epel-devel mailing list -- epel-devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to epel-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/epel-devel@xxxxxxxxxxxxxxxxxxxxxxx > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue -- Carl George _______________________________________________ epel-devel mailing list -- epel-devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to epel-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/epel-devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
