On Mon, 1 Mar 2021 at 20:18, Davide Cavalca via devel <devel@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
On Mon, 2021-03-01 at 15:49 -0500, Stephen John Smoogen wrote:
> That would require a lot of changes in both EPEL and in Fedora. In
> Fedora there is a general expectation that if a 'branch' is active
> then it is maintained by someone.. usually the primary maintainer.
> Many Fedora maintainers are only interested in maintaining packages
> in the latest release. THis is why the ELN packages are being
> 'watched/maintained' by the ELN team and sig. Maintainership usually
> means dealing with bug reports, build failures, etc which can take up
> a good amount of time.
That's a fair point. To be clear, I would not expect this to happen by
itself -- it this idea turns out to be feasible, I would be happy to
pitch in and/or try and find resources to help with this from my end.
> This is part of the reason why EPEL packages are not auto-forked for
> each EL release. Most maintainers are only interested in supporting
> maybe EL-6 or EL7 and we need to find someone new to do the EL-8
> branch.
Interesting, I had not realized this was the case. My (likely naive)
assumption was that if there's an epelX branch for a given package,
there would likey be an epelX+1 too, and when that wasn't the case it
was mostly because the maintainer hasn't gotten around it yet or nobody
had asked. My assumption was that by providing a readily updated
"epeln" branch we would save the maintainer some of this work.
Are you saying that for some packages it is expected that the
maintainer would only care about say epel7 and not epel8? In that case,
do we / would we track the maintainer for epel8 specifically somewhere,
if one were to volunteer?
So mainly a package maintainer only worries about what is deployed at their workplace. And I would guess from the size of unanswered bugs and other things, some of these maintainers did a one-time build to get what they wanted and went onto other things. Others would update things until it could no longer build with older RHEL-6 or 7 code and stop maintaining it. This is where i get most of my answers.. someone asks 'why is it not in EL-?' and I go ask the maintainers. Usually the Fedora maintainer is like 'I don't use EL and don't want to unless paid to' and then I find on the list of co-maintainers who might be an EPEL maintainer. This usually gets the answer of 'look we are a EL-6 or EL-7 shop and I can only really do it for that. Can you find someone else?' Then someone eventually volunteers, we find out they aren't a packager and have to go look again. Eventually we get someone who is a packager who gets added to the packaging list for that package and takes over that branch.
Since EPEL was not really a major thing the Fedora maintainers or the base community have wanted to invest in, there has been no drive to do more book-keeping than what is needed to keep the system going. We could track EPEL/non-EPEL maintainers in the package database system and now we can track some level of control in pagure I believe but it has been pretty much 'is it there? good.'
> We would need to have a dedicated team of people to do this with ELN
> related items. We would also need to have additional build and
> storage resources to deal with these artifacts, not alone the growth
> of 'just need this' packages.
Is there an easy way to ballpark estimate the resource demand that an
effort like this would require?
up to 8 full time people for basic packaging, builds, dealing with bugs, etc. [This is going on a decision that packages being 'maintained' are only at 'it built, ship it' quality. If we need higher levels then you will need more staff.]
4 people to do documentation, upkeep and other management duties of keeping things on track
4 people to work through QA items and actually figure out tests relevant to EPEL.
6-8 people to design and work with the koji upstream to fix the fedora build system to work with two different modularities. Currently MBS can only build a limited type of modules which is what is breaking PHP and I think perl apps.
The work plan would be that most of the first 6-8 months would be that last group working on getting a build system which works for EPEL. Because EPEL uses a koji 'hack' to get to the RHEL binaries it does not 'know' really what those packages are in the same way it does for packages it built itself. The same goes with MBS. This means that you have to come up with a way to fool them into using the proper modules when they are needed etc. HOWEVER you also have to not break the Fedora build system at the same time so that fedora packages can continue to build. At the middle of the initial period it would be either done or found to be outside of easily possible and some other system would be needed. At that point it is either finish it up or build a new system.
The package branching/building/qa can go on for a subset of packages but some will be impossible because of the modularity problem.
After either of the systems are in place, a planning period goes into place on how to build out modules and for what. There are some things which make sense (PHP, python, ruby) and then there are specific complicated apps which make sense. However there need to be consistent rules, documentation on how to do it, and training to make them. Also a consistent testing process would need to be done.
By the time this is done then it will be time to work on the next set of EL branching
[I had an older proposal which went over the above but undercut the number of people needed.. when we tried this with EPEL-8 it was overwhelming at 1/2 the staff numbers so I am hoping I am getting this right at this level.]
> When I was trying to make EPEL-8 1:1 with EPEL-7 I found I was having
> to add more and more packages just to get the new build 'requires:'
> done. I stopped around a thousand 'new' packages to the tree when I
> was reminded that we don't do automatic branching for EPEL. I really
> don't know how many packages would be needed to make it work, but do
> know it is a full time job to get set up and keep going. While ELN is
> probably 4000 src.rpms we would be looking at needing to
> build/rebuild double that for EPEL.
Ah, that's fair. I hadn't thought about potentially neededing to branch
extra packages to meet new build dependencies, but you're absolutely
right, that would be a major issue here.
I should probably expand on why I'm thinking about this in the first
place. I want to use ELN as a proxy for the next CentOS Stream release
to streamline its qualification on our infrastructure. The idea being
that if we can continously test ELN on a small subset of production,
that will detect potential issues early on, allow us to get ahead of
policy changes that might require internal work, and hopefully surface
bugs that we can report and fix upstream long before branch time.
However, we use EPEL a lot in our infrastructure, to the point that I
suspect it would be difficult to do a meaningful prod-like deployment
with ELN alone. I could probably hack something around this, but I
figured it'd be worth to see if we can build a generic solution instead
that could benefit Fedora and CentOS upstream, rather than something
Facebook specific. To be clear, I'm aware that this would require work
and resources within Fedora, and I'd be happy to help with both as
needed if this is deemed feasible.
I do see the need for this, but I really think this might be the straw which broke the camel's back. ELN is already getting a lot of pushback because the tools send regular failure notices to maintainers who have nothing to do with ELN. Adding in 10,000 more packages would basically push a lot of maintainers past the breaking point. There is also the fact that ELN-EPEL would be building these against s390x and p64le on builders which are basically overloaded already and you don't care if you got a package from.
It would also not drive a 'fix' to the problems of where this package goes from being in ELN and is now in RHEL so the Fedora build system no longer 'knows about it'. So you would find that when EL10 came out, that the EPEL system would be 'broken' in many ways not covered by the ELN testing.
Cheers
Davide
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
--
Stephen J Smoogen.
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
