On 14. 06. 21 14:40, Alexander Bokovoy wrote:
On ma, 14 kesä 2021, Miro Hrončok wrote:
On 14. 06. 21 13:40, Alexander Bokovoy wrote:
On ma, 14 kesä 2021, Victor Stinner wrote:
Congrats, that's amazing! :-) Let's fix remaining broken packages!
Right now the biggest broken package to us (FreeIPA) is mod_wsgi which
relied on Python C API which was removed in Python 3.10. The same API
was used by uWSGI as well, so it would really be great if Python core
developers and Python maintainers team could have helped mod_wsgi/uWSGI
upstreams to migrate from the old API.
Sadly, this was only found after the fact when a side-tag was already
merged, so FreeIPA right now is broken in Fedora Rawhide.
The problem in mod_wsgi was actually found and reported in November 2020 [1],
which is exactly why we test with each pre-release of Python: To report
problems *early* and give maintainers time to fix them, so we can land the
new version with limited negative impact.
Unfortunately, the mod_wsgi maintainer did not respond to the problem :(
While we try really hard, we cannot report each failure to upstreams directly.
The fact that this blocks FreeIPA was indeed only discovered by chance while
the side tag rebuild was already in progress (and about to be merged). I
wonder haw can we improve the process to ensure problems like this are known
to FreeIPA maintainers since the beginning and prioritized accordingly.
(Ideally, the process would not only be improved for FreeIPA but the entire
distro.)
Well, this is a dependency problem in the first place. When an ABI
change happens, like a Python ABI change with 3.10 mass-rebuild, it
should be assumed and expected that until all previously successfully
built packages were to be rebuilt, there will be broken dependencies.
Yes, however getting "all previously successfully built packages rebuilt" is an
impossible goal. Instead we strive to get "most of them rebuilt" and "the
important ones rebuilt". Getting most of them rebuilt is not an easy task but
at least is is quite easy to quantify: 3333 are rebuilt, 293 to go, that is
~91%. OTOH knowing what is "important" is hard. There is the "critical path"
thing (but mod_wsgi is not in it and the list seems heavily outdated).
Perhaps, we could extend our existing checks for a broken compose to be
done on a side-tag on demand? This way mass-rebuilders could ask for
such a run one they consider to be ready to merge and see how that
side-tag merge would affect the distribution. I don't think we have a
better way to detect it before the merge.
We try hard not to break the compose. It is not fully automated and having a
way to run a test compose would be awesome.
However, this particular problem does not break the compose, the compose is
done now even when mod_wsgi is not installable.
We have also validated the following groups are installable before merging the
side tag:
@anaconda-tools
@base-x
@cloud-server-environment
@container-management
@core
@domain-client
@firefox
@fonts
@guest-agents
@headless-management
@input-methods
@kde-apps
@kde-desktop-environment
@kde-media
@libreoffice
@multimedia
@networkmanager-submodules
@printing
@server-product
@standard
@workstation-product-environment
Because we parsed them from the hopefully compsoe-blocking media kickstarts.
What I do not see as acceptable is what Python core team did in November
2020 when this issue was made clear in the original PR to remove the
'old' parser that broke mod_wsgi/uWSGI/unbound and some more packages.
Instead of following up in providing a supported API function, the
comment[1] from Victor was practically ignored.
I am not happy either that Python upstream keeps breaking things and often the
migration path is "go figure". Whenever and wherever I can, I argue against that :(
I assume that opening bugzillas for all the dependent packages for each
failure would be considered as spam by many.
It depends on what that bug opening would mean. If you are doing it
early together with an upstream that breaks its ABI/API, helping them to
see the issues early is one of your goals -- as you stated above.
Realizing through reverse dependencies a damage such breakage could
cause is something you (as Python maintainers in Fedora) could
definitely do and raise the issue earlier also with the package
maintainers/upstreams. In this case FreeIPA is affected but we missed it
completely in November 2020 as nobody told us mod_wsgi would not work
anymore.
This is a chicken-egg problem: Until we measure the impact, we don't know it.
And reporting many dependent-packages bugzillas early seems like a waste of
effort if the bug is getting fixed soon. What we need to detect (and we don't
know how to automate that yet) is:
- a bugzilla for a problem is stalled AND
- the package is dependent on by "important stuff" or by a large stack
Even when not automated, we manually rise priorities for bugzillas that block
the builds of large stacks.
What would be particularity helpful would be if *all* the package maintainers
triaged their bugzillas and responded to them in timely manner, notify
upstreams and ask for help if they need it. However, that is not realistic.
In the meantime, I've marked mod_wsgi as "blocker" for the Python 3.11
rebuild next year, but that is not a systematic solution.
I saw that there is some discussion and an effort to help mod_wsgi
upstream already, thank you!
There is a proposed fix [2] but we don't know enough about Apache to say if
it is ideal. In case this blocks you very much, I suggest you work with the
Fedora's mod_wsgi maintainer and backport this patch to rawhide, keeping an
eye on it in case it breaks your use cases, and report problems to us, so we
can adapt it. At this point of the Fedora 35 development schedule, I think
this patch should be good enough for it.
I already nominated mod_wsgi bug as Fedora 35 beta release blocker
because it violates Fedora Server release criteria[2]. I am not a
maintainer for mod_wsgi myself and have no rights to that package.
I know and I voted +1 for a blocker. I am not a maintainer for mod_wsgi either.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
