On Wed, Apr 1, 2020 at 7:28 PM Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote:
On Wed, 2020-04-01 at 11:49 +0200, Clement Verna wrote:
> On Wed, 1 Apr 2020 at 09:47, Panu Matilainen <pmatilai@xxxxxxxxxx> wrote:
>
> > On 3/31/20 8:44 PM, Adam Williamson wrote:
> >
> > > I understand there are practical resource considerations and so on
> > > here, but I still think this merits more high level and serious
> > > consideration. At the very least, if we have somehow reached a point
> > > where Red Hat is no longer willing to provide sufficient resources to
> > > run Fedora on the lines the Fedora community wants it to be run, we
> > > need to recognize that this is a significant problem that needs to be
> > > properly aired and discussed and resolved. In this context I'll note
> > > that the apparent significant headcount reduction of RH people working
> > > on Fedora infrastructure over the last few years is in itself a
> > > worrying trend, particularly if you consider it while reading Clement's
> > > email.
> >
> > This.
> >
>
> I don't think this is correct, at least not in CPE, the team has grown over
> the past year and every person leaving the team has been replaced (even by
> 2 persons in some cases).
Okay, I didn't mean to get into detail, but I don't see how this is
possible. Unless you're considering 'CPE' as a whole (which includes
the infra admin team, release engineering, and docs) versus 'people
doing development'.
Around two-three years ago, IIRC, we had all these people working on
Fedora app/infra development:
puiterwijk (who counts as about ten people, admittedly, replacing
Patrick with anything besides an entire team was always going to be a
challenge)
Randy (bowlofeggs)
pingou
jcline
Ralph
jflory
abompard
Right now, I see these people who appear to be on the CPE team and
clearly working on Fedora infra/app development:
abompard?
lrossetti/odra (although I had not heard of or seen him till I started
doing research for this post, he seems to be working on fasjson which I
don't really run into)
pingou
scoady
cverna
I put a question mark by abompard as he clearly works for RH and works
a lot on infra, but (looking at internal systems) does not seem to be
quite in the same management chain as the others, not sure why that is.
Jim and I are co-managers of the CPE team and the reporting is shared between us for logistical reasons (i.e. number of direct reports)
I'm also not entirely sure about Adam Saleh, but he does not have any
infra activity I can find since the end of January and lists himself as
working for Exponea on Github and his personal blog.
Adam is working in CPE. CPE isn't entirely about Infra, he is working on CI improvements at the moment alongside some others
If I'm missing anyone, please do correct me.
Other developers in our pool right now are:
- Ryan Lerch
- Michal Konecny
- Mohan Boddu
- Tomas Hrcka
- Nils Philippsen
- Vipul Siddharth
- James Richardson
We additionally have 2 new Ops folks joining us over the next 2 weeks.
Across them, the majority are working on the Fedora aspects (Infra, Dev, Releng) of the CPE remit.
Mattia Verga seems to do a
lot of work on Bodhi, but AFAICS he is not part of RH CPE. We could
count Ryan Lerch (UX) and/or Kevin Fenzi (who's officially admin, but
as we all know, does everything) in both list or neither.
Maybe the people I remember were not all there at exactly the same
time, I'm not 100% sure as I'm not on that team. But it does not feel
to me like we (RH) have the same level of personpower working on Fedora
app/infra development now as we did in 2018, put it that way.
The number of active developers on Fedora initiatives has gone up drastically since I joined the team in 2019. You are possibly not seeing that as the team have moved from a model of siloed work on multiple apps, swimming against the tid working 16 hour days, to working on team oriented initiatives to add real value to the ecosystem. So the noise of working on multiple small things at once is not as loud as it was in 2018 which is giving that illusion.
The list at
https://fedoraproject.org/wiki/Community_Platform_Engineering seems to
be rather out of date; to my knowledge, Randy, Sayan, and Sinny are no
longer on the team, correct? Others in the list seem to work more or
less entirely on CentOS.
Yup it's missing some folks, will get it updated.
> The problem in my opinion is that a lot of the
> people that have setup and written the original services and applications
> are gone, taking with them most of the knowledge about How, What and Why
> something was done this way. That leaves people in the team now with a big
> amount of legacy applications to take care of and not much clue of what is
> going on.
I can believe this, though I'd also note that few of the apps are
original to the "last gen" team either. Bodhi was written by Luke, back
in the day, for instance, but Ralph did a great job of taking it over
(and fixing it up). fedmsg was written by Ralph, then handed off
successfully (AFAICS, anyway) to jcline. So we seem to have done these
kinds of hand off before with success. What changed there?
An entire code base handed over to one person is not viable for a whole host of reasons. For Rawhide Gating we had 6 Engineers on Bodhi for almost 7 months of last year and even that took one person to mind the lights on work while the actual development progressed.
> There is also an historical taste to write in house applications for things
> that don't really seems critical to the Fedora Project, for example do we
> really need a custom calendar application ? or election application ? It
> seems that every time we have a problem the solution is let's write
> something to solve that problem, instead of trying to find a compromise and
> reuse existing solutions.
So on the whole I agree with this. I talked to Jim about this whole
situation at OSS 2018 and he was already thinking along the lines of
cutting some of the stuff we really don't need to be writing. I
entirely agree that it's a reasonable approach to ditch stuff like this
that is *relatively* external to the process of actually building a
distribution. I'd be kinda sad if it was replaced with proprietary SAAS
stuff, but I don't think it'd be as big of a problem as if we did that
with our dist-git.
I don't think we need to go into it in detail, but I *do* also
understand why we did it in the first place, and I think there's still
merit in that. For a start, it genuinely was the case back when a lot
of these apps were written that you couldn't so easily go out and "buy
one off the shelf", so to speak. But there's also the whole idea of
dogfooding, and the ecosystem. Again there's a historical angle to
this: it wasn't just for us but for the whole industry that it was more
common to build stuff in-house than outsource absolutely everything
besides your "core competency", as is the management fad these days. So
for us to build our own infrastructure was kind of a proof-of-concept
that Fedora *was* a suitable environment for *anyone* to do that: at
least at the time we felt that there was value in demonstrating that
Fedora provided an environment in which you *could* build and deploy
and maintain and use, you know, calendaring systems! and authentication
systems! and election apps! and so on and so forth.
I genuinely tip my hat to the developers and sys admins that created such solutions and made them open. Unfortunately as time went on and contributors moved on to pastures new, the CPE team have inherited those apps and must keep them alive, fix issues in them and in some cases enhance them. That is what has bloomed our remit, not the creation of apps to solve a problem that it was needed for, the long term maintenance and ownership model of said projects. Many were created in the here and now and relied heavily on that creator being engaged and responsive.
I agree that this has fallen out of fashion to an extent, and the
"let's just outsource everything that isn't actively building bits into
OS images" mindset is in tune with Current Industry Thinking and there
is a sustainable logic behind it. Personally I still think it comes
with significant issues, though, because there were a lot of what
management is pleased to call "intangibles" that went along with doing
all that work. We had an awful lot of "in-house" knowledge about a
whole range of stuff because we *did* build and maintain and deploy and
support all of these bits. Why do you think Patrick and Kevin can fix
just about anything? In large part because of all that experience.
Still, we're getting a bit into the weeds at that point. I'm not
arguing that we *must* continue to develop elections apps and calendars
and all of that stuff. To be clear, I'm not even arguing that we should
continue developing Pagure as part of CPE, or even necessarily host
Gitlab ourselves.
> Now when the CPE team goes and ask for more people because we struggle with
> current situation, I can only guess that these non critical applications
> are mentioned. If I was putting my own money to sponsor a team to help
> building a Linux distribution I would be asking why do we have to develop a
> calendar application or why do we need a custom git forge. I personally
> find really great that the different use cases and requirements for the use
> of Pagure were gathered and I am convinced that people working on this did
> their very best to find a use case to justify the investment needed in
> Pagure but it seems that we don't have such a thing.
I think other people are following up on this, but from following the
discussion, it appears to me that there seems to have been a large slip
somewhere along the line from Ben submitting a list of ~50 Fedora
submissions, to Leigh suggesting that (after those were, mysteriously,
"summarized" somehow)
Can you help me understand what the mystery is about? We took in 300+ requirements that we distilled down into the generic list that we came up with, many of which are buckets / Epics. Every single requirement was analysed. I have said this multiple times but please, if this is still a mystery to you, let me know how I can help clarify it.
Fedora does not have much in the way of specific
requirements. I also think the confusion between "git forge" and "dist-
git" is a major issue here. On the whole I suspect people using
Pagure.io for project hosting don't have a lot of very specific
requirements, because - let's be honest - Gitlab and Github can
certainly do nearly everything Pagure does as a generic project hosting
system. I could probably move my projects to a Gitlab instance in an
afternoon (depending on how much of a pain it is to set up F/OSS CI on
Gitlab, I have not looked into that yet, and yes, for anyone who hasn't
been looking lately, Pagure does actually have CI integration now).
However, I think we (Fedora) certainly *do* have a whole pile of
requirements for a dist-git system, and Leigh's responses so far (of
the "well we haven't figured all that out yet!" variety) do not give me
confidence that this was fully appreciated in the decision-making
process.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Leigh Griffin
Engineering Manager
Communications House
Cork Road, Waterford City
lgriffin@xxxxxxxxxx
M: +353877545162 IM: lgriffin
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
