On Fri, Mar 3, 2023 at 3:42 AM Daniel P. Berrangé <berrange@xxxxxxxxxx> wrote:
A few weeks back there was a little snafu (since resolved) with the
capstone package in EPEL.
This package is being introduced to RHEL 9.2, and thus has been added
to CentOS 9 Stream repos. A bug was filed against capstone in EPEL
to notify that capstone should be retired when RHEL 9.2 is released:
https://bugzilla.redhat.com/show_bug.cgi?id=2124181
Unfortunately the package was accidentally retired immediately rather
than waiting until 9.2 release. This is an easy mistake for a maintainer
to make when they see a bug whose title is "Remove capstone from epel9",
so I don't want anyone to blame the maintainer for this accident.
Yes the comments make it clear that removal should wait until 9.2
release date, but I can totally understand how actions will be guided
by the initial bug subject if you're just skimming. It was pointed out
that this risk has been identified before:
https://lists.fedoraproject.org/archives/list/epel-devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/LUWUIN5BRPJZGVB5PIH625PFV6IBD6YO/
and now we have a real world example of it going wrong exactly as
was predicted.
My bigger concern though is what happens when RHEL-9.2 is actually
released for real. The bug remains open and targets (immediate)
retirement of capstone from EPEL on / near 9.2 release date.
This doesn't feel like it takes into account the real world usage of
enterprise distributions. The kind of people/organizations who deploy
RHEL are rarely going to rush into upgrading their deployment processes
to use 9.2 immediately. They may have all sorts of review and testing
processes to go through, integration of 3rd party apps and services,
validating against hardware, before they consider starting deployment
of RHEL 9.2.
All this time they will continue deploying machines with 9.1, despite
9.2 being available, so won't get access to new 9.2 packages.
If we immediately retire capstone when 9.2 is released, aren't we
going to break any ability to keep deploying capstone from EPEL on
existing/new 9.1 installs ?
There is also the case of the RHEL rebuilds whose users consume EPEL
packages. Depending how quick they are, the rebuild distros might not
have their 9.2 rebuild ready for some days/weeks/months after RHEL-9.2
is first available. My projects' upstream CI is all based on AlmaLinux
and I don't want to see it broken again by premature capstone retirement
from EPEL.
Unless I'm mistaken in some way, it feels like we should *NOT* sync
the removal of packages from EPEL with their release in RHEL 9.2. It
needs to be synced with user deployment of 9.2, which might be somewhat
later.
The package added to RHEL should (must) have a newer NEVR than the
existing package in EPEL, so even if we leave it in EPEL, it will
be ignored if the newer version is present in RHEL repos.
The main important thing I see is that the EPEL branch in dist-git
must be made read-only and/or prevented from submitting new builds,
once the package has been built in CentOS Stream. ie to ensure the
EPEL NEVR doesn't creep back above the pending RHEL package's NEVR.
If that's done it should be fine to leave it in EPEL for a prolonged
period after 9.2 is released, to allow sufficient time for users to
actually switch from deploying 9.1 over to 9.2.
I guess someone might say that EPEL only targets the very latest
y-stream of RHEL, so continuing to use 9.1 after 9.2 is released
is unsupported ? Even if that is technically the case, I think
in practice users won't take that into account, and thus expect
it to keep working and generally that should be fine. Only if a
EPEL package is rebuilt such that it depends on a newly added
API/ABI is that a problem.
With regards,
Daniel
Hi Daniel,
Thank you for your insight as an end user. Sometimes repo and package maintainers are always trying to get to the latest and greatest and rush things.
We are currently in the middle of changing the workflow of retiring these packages. We are changing it so that the package maintainer doesn't do the removing. It will be automated, or semi-automated, so there is a consistent time when all of them are removed.
The first step is re-working the bug that get's opened.
It will still be against the package that is to be retired, and linked to the tracker.
The subject and initial comments will tell the maintainer that they do not need to do anything, that it will be taken care of for them.
An automation of some type (currently it is manual) will add another comment assuring them that it will be done automatically and there is nothing for them to do.
The automation will then assign the bug to the appropriate person/group.
But then comes the part that you talk about, and it's a good thing to have a discussion about.
When do you actually remove the packages from the EPEL repository?
It has been agreed that it will be after both Alma and Rocky have their latest release out.
But how long after?
Immediately after? a month? 6 months?
I personally am leaning towards a month after.
Here is my reasoning.
At the time a new RHEL release is released, we take a snapshot of the EPEL repo and put it in the archives.
So all the packages that were built, and run on RHEL 9.1 are available in that archive.
There are always some packages in EPEL that no longer install on the newer libraries of a new release. Depending on the library, that might be a couple, it might be alot.
It takes roughly a month for those EPEL packages to get built, go through testing, and released.
Anyway, if some users are still doing new installs of RHEL 9.1 (or compatible) after that month, then they should probably add the epel 9.1 archive to their yum repositories.
There is the argument that why remove them at all if they are a lower NVR than RHEL's?
But then you will have to figure out which packages are higher than the RHEL versions.
It's much easier and cleaner to remove them.
Troy
_______________________________________________ epel-devel mailing list -- epel-devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to epel-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/epel-devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
