Re: Workflow and other problems with the Fedora container infrastructure

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote:
> Hello.
> 
> My name is Lumír and I maintain a few layered container images in
> Fedora/RHEL/upstream related to Python and S2I (source to image) platform -
> namely python3, s2i-core, and s2i-base.
> 
> TL;DR: We (maintainers of layered container images for languages, runtimes
> and databases) are considering migrating our images from Fedora
> infrastructure and Fedora containers registry somewhere else. Below are the
> reasons for that. If you think it’s a bad idea, let us know why. If you are
> a user of those images, let us know as well so we can stay in touch with
> you.
> 
> Forgive me for my wording but I’m really sad about the current situation.

yeah, it is sad. ;( 

> I have to honestly admit that I’m disappointed about the lack of attention
> containers in Fedora are getting. Maintaining them is painful,
> infrastructure is unstable and unreliable and there was literally no
> movement forward in recent years. Let me summarize it.
> 
> Container SIG is inactive. See important issues opened for years with no
> movement further: https://pagure.io/ContainerSIG/container-sig/issues
> 
> Even a simple task like merging rawhide branch to f35 for example is far
> from smooth because they are never the same, because manual changes are
> required for each branch:
> https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no
> reply)
> 
> S2I container images depend on each other (python3 -> s2i-base -> s2i-core)
> but there is no automation for chain rebuilds so if I want to update
> s2i-core image (to fix a CVE or something like that) and all dependant
> images, it takes weeks because I have to rebuild them one by one and let
> them one by one pass through Bodhi updates until I can rebuild next one in
> the queue. And it’s really frustrating given the insufficient stability of
> the infra. There seems to be a way how to make this faster, but nobody
> knows, how it works: https://pagure.io/ContainerSIG/container-sig/issue/52
> and https://github.com/containerbuildsystem/atomic-reactor/issues/1730
> 
> The same applies also if the fix is done on the RPM level because there are
> no freshmaker nor automated rebuilds.
> 
> The branching structure does not really fit container needs. I maintain the
> s2i Python container and I don’t really care whether it’s based on Fedora 34
> or 35 (both have Python 3.10 as the main one) so I’d ideally have one branch
> for Python 3.10, one for 3.9 etc. instead of two separated branches (f34,
> f35) where the Python is the same version.
> https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old)

Yeah, I think all the above is kind of a consequence of there not being
a active containers SIG. Infrastructure and releng doesn't really have
cycles or interest in driving containers forward, so it needs some other
folks attention. ;( 

> An example of unstable infrastructure: I’m not able to build an image for
> three weeks now and a response I got is to report the problem upstream:
> https://pagure.io/fedora-infrastructure/issue/10437
> I know that nothing is 100% reliable but if you take a look at these issues,
> there are a lot of them: https://pagure.io/fedora-infrastructure/issues?search_pattern=container&status=Closed

Do note that that issue was filed on December 21st. 
Myself and much of the infrastructure team were off on holidays. 
I only got back yesterday and there's a ton of things to do, but I did
try and look into this some. 

The reason there's so many issues I think is because the entire pipeline
is just way overkill and complex for just building containers. Basically
we have to have koji, koji builders, and 2 openshift clusters (one for
x86 and one for aarch64) to build containers, they have all kinds of
fragile handoffs between each other as well. 

> And it seems that the packages in the infrastructure are outdated a lot: 
> https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuecomment-1009279161

To clarify, thats installed in the build container... so, _fedora_
packages are outdated. ;( I have seen atomic-reactor being old, but I've
been afraid to update it since I don't know the pipeline. 
Co-maintainers of those packages in Fedora would sure help out. 

> If we migrate our container images to some other registry (e.g. a common
> fedora space on quay.io), we’ll be able to rebuild them after every merged
> PR or every week, do chain rebuilds and push them to the registry directly
> from Github CI. This will make our lives significantly easier, and in turn
> our images better.

We are looking into dropping out registry and moving to quay.io
ourselves. ;) 

> Are there any benefits to using Fedora infrastructure and Fedora containers
> registry which I don’t see that we should consider?

I'm not sure... hopefully folks who know the pipeline will chime in
here. 

kevin

Attachment: signature.asc
Description: PGP signature

_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux