'It's a trap!' </joke> :D On 8/27/20 5:13 PM, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/Python_Upstream_Architecture_Names > > == Summary == > Use CPython upstream architecture naming in Fedora's Python ecosystem > (mostly in filenames) instead of the previously patched Fedora names. > For example, have > <code>/usr/lib64/python3.9/lib-dynload/array.cpython-39-powerpc64le-linux-gnu.so</code> > instead of <code>/usr/lib64/python3.9/lib-dynload/array.cpython-39-ppc64le-linux-gnu.so</code>. > This makes packaging of Python itself a tad trickier, but it moves > Fedora's Python closer to upstream and solves interoperability > problems with ppc64le manylinux wheels. > The change has impact only on '''ppc64le''' and '''armv7hl''' > (considering the architectures built by koji.fedoraproject.org). > Packages assuming the filenames always contain > <code>%{_arch}-linux%{_gnu}</code> will need to be adapted. > > == Owner == > * Name: [[User:Churchyard|Miro Hrončok]] > * Name: [[User:Lbalhar|Lumír Balhar]] > * Email: python-maint@xxxxxxxxxx > > > == Detailed Description == > > === The Saga === > > ==== A Long Time Ago in the Fedora Galaxy ==== > > Many releases ago, when Fedora wasn't being built for power and arm > yet, the Python maintainers mapped the Python "platform triplet" or > "multiarch name" to <code>%{_arch}-linux%{_gnu}</code>. This worked. > For example, on x86_64, this is <code>x86_64-linux-gnu</code> on > Fedora and this is consistent with the "platform triplet" used in > filenames in upstream. > > ==== The Phantom Technical Debt ==== > > Later around the year 2015, as more architectures were added, Python > build scripts were patched to use "the Fedora's architecture names": > > * <code>powerpc</code> was changed to <code>ppc</code> > https://src.fedoraproject.org/rpms/python3/c/0efd3d31 > * the arm triplet was patched > https://src.fedoraproject.org/rpms/python3/c/2f6352e7 > * even the mips one https://src.fedoraproject.org/rpms/python3/c/4bc70e0c > > At the time, that was a reasonable decision: the idea of cross-Linux > builds was sci-fi, and Fedora was not trying to stay close to upstream > as it is now (we had around 60 patches; today we're down to around 6). > > ==== Rise of the Manylinux Wheels ==== > > In the meantime, cross-Linux builds become a thing. The > [https://www.python.org/dev/peps/pep-0513/ manylinux1 standard] was > created in 2016, allowing to build Python wheels with compiled > extension modules on one Linux platform and ship them to many. The > first manylinux version only supported x86_64 and i686 and hence it > was not impacted by Fedora's patching decisions. > > The manylinux standard arguably made the upstream Python packaging > ecosystem a much nicer place. Installing packages with compiled > extension modules was no longer such a pain. One could just run > <code>pip install numpy</code> and not worry about a disturbing lack > of a Fortran compiler. For that reason, manylinux wheels become widely > adopted by the most popular projects. > > ==== A New Architecture ==== > > With the third manylinux version -- > [https://www.python.org/dev/peps/pep-0599/ manylinux2014] (created in > 2019, named after the oldest Linux it supports -- CentOS 7), support > for more architectures was introduced: x86_64, i686, aarch64, armv7l, > ppc64, ppc64le, s390x. The adaption of new architectures is somehow > slow, because the [https://github.com/pypa/manylinux#manylinux2014 > official manylinux2014 containers] only currently (August 2020) exist > for x86_64, i686, aarch64, ppc64le and s390x. > > ==== Revenge of the Patches ==== > > We have discovered [https://github.com/pypa/manylinux/issues/687 a > problem with the ppc64le manylinux2014 wheels]: The CentOS 7 > manylinux2014 container images ship upstream Python without > RHEL/CentOS/EPEL patches. When an extension module is built there, it > is named with an upstream named suffix: > <code>.cpython-XY(m)-powerpc64le-linux-gnu.so</code>. The wheel is > installable on Fedora (with Fedora's patches), but the module won't > (even be considered for) import, because Fedora's Pythons expect the > extension to be <code>.cpython-XY(m)-ppc64le-linux-gnu.so</code>. > > In theory, we have the same problem on armv7hl, but there are no > manylinux2014 containers available for that platform, so there are no > such wheels out there yet (to our knowledge). > > The same problem also exists the other way around, albeit it's > arguably less severe. It is possible to build manylinux wheels on > (some version of) Fedora or EL (using the Python from that > distribution). However extension modules from such ppc64le wheels > won't import on other Linux distributions. > > ==== The Workaround Awakens ==== > > To allow importing extension modules from ppc64le manylinux wheels, we > have [https://src.fedoraproject.org/rpms/python3.9/pull-request/24 > patched Pythons (3.5+) in Fedora] to consider both "Fedora's" and > upstream extension suffixes when importing extension modules. This > workarounds works well for users installing manylinux wheels on > Fedora, but does not solve the problem when building the wheels on > Fedora. > > === The Change === > > With this change proposal, we plan to > [https://src.fedoraproject.org/rpms/python3.9/pull-request/28 switch > to use the upstream architecture names] and keep the workaround to > preserve backwards compatibility. When we do that the following will > happen: > > # The Python standard library extension module suffixes will change to > <code>.cpython-39-powerpc64le-linux-gnu.so</code> and > <code>.cpython-39-arm-linux-gnueabihf.so</code>. Python will still > import extension modules with the legacy suffixes > <code>.cpython-39-ppc64le-linux-gnu.so</code> and > <code>.cpython-39-arm-linux-gnu.so</code>. Other architectures not > built by koji.fedoraproject.org will also be renamed, see > [https://src.fedoraproject.org/rpms/python3.9/pull-request/28#comment-0 > the pull request for a complete regex]. This will happen for Python > 3.7, 3.8, 3.5, 3.6 and 3.9. > # The newly built Python packages with extension modules will also > change the suffixes. Packages that assume the platform triplet is > always <code>%{_arch}-linux%{_gnu}</code> (e.g. in the > <code>%files</code> section) will need to be adapted (see the > [[#New_Macros|New Macros]] section). A mix of legacy and upstream > suffixes will co-exist and work together. > # When safe, we will drop the workaround to support the legacy names. > For example, when we initially package Python 3.10, it will be > packaged without the workaround. On the other hand, older Python > versions might never be able to drop it, because users will carry > their own built extension modules from previous releases. > > === New Macros === > > For packagers' convenience we will add 2 new Python macros: > > '''%python3_ext_suffix''' > > Defined as: > > %python3_ext_suffix %(%{__python3} -Ic "import sysconfig; > print(sysconfig.get_config_var('EXT_SUFFIX'))") > > Values will be: > > * <code>.cpython-39-x86_64-linux-gnu.so</code> > * <code>.cpython-39-powerpc64le-linux-gnu.so</code> on Fedora 34+ / > <code>.cpython-39-ppc64le-linux-gnu.so</code> on older Fedoras > * <code>.cpython-39-arm-linux-gnueabihf.so</code> on Fedora 34+ / > <code>.cpython-39-arm-linux-gnu.so</code> on older Fedoras > * etc. > > Beware that due to chnages in Python, this macro only works with > Python 3. For Python 3.4, the value is <code>.cpython-34m.so</code> on > all architectures (similarly on older Python 3 versions, but we don't > have such in Fedora or EPEL). > > '''%python3_platform_triplet''' > > Defined as: > > %python3_platform_triplet %(%{__python3} -Ic "import sysconfig; > print(sysconfig.get_config_var('MULTIARCH'))") > > Values will be: > > * <code>x86_64-linux-gnu</code> > * <code>powerpc64le-linux-gnu</code> / <code>ppc64le-linux-gnu</code> > * <code>arm-linux-gnueabihf</code> / <code>arm-linux-gnu</code> > * etc. > > Beware that due to changes in Python, this macro only works with > Python versions 3.6 and newer. > > Both macros will be backported to stable Fedoras and EPEL 7+ and will > have the corresponding <code>%python_</code> variant. > > == Feedback == > Eighth_Doctor: it's the cleverest written change proposal I've seen > in over a decade :) > > == Benefit to Fedora == > > Users of ppc64le and armv7hl Fedora (and future RHEL) will have a > closer-to-upstream Python experience and will no longer suffer from > compatibility issues when they install or build manylinux wheels. The > upstream-downstream balance will be restored. > > == Scope == > * Proposal owners: > ** Review and merge https://src.fedoraproject.org/rpms/python3.9/pull-request/28 > ** Add the new macros to all Fedoras and EPEL 7+ and mention them in > the guidelines > ** Monitor builds of Python packages with extension modules, fix > issues if they arise. > ** Backport to Python 3.5-3.8 > ** Drop the workaround from Python 3.10 once packaged > > * Other developers: Mostly nothing, adapt the <code>%files</code> > section if needed > * Release engineering: a check of an impact with Release Engineering > is not needed > * Policies and guidelines: the new macros will be documented in the > Python packaging guidelines > * Trademark approval: not needed for this Change > * Alignment with Objectives: no > > > == Upgrade/compatibility impact == > No significant user visible upgrade/compatibility problem is anticipated. > Filenames will be different, but the old filenames are still supported. > Scripts that hardcode filename assumptions might break. > > == How To Test == > On ppc64le, try to install a manylinux wheel and import from it. It > should work on any Python ≥ 3.5. E.g.: > > pip install simple-manylinux-demo > python -c 'from dummyextension import extension' > > On ppc64le, try to build a manylinux wheel and import from it on > another Linux. It should work on any Python ≥ 3.5. E.g.: > > pip wheel . # on some project with extension module > auditwheel repair ...whl > wormhole send ...whl # or any other way > > On another ppc64le Linux (such as Debian or openSUSE): > > wormhole receive ... > pip install ...whl > python -c 'from ... import ...' > > You can also build a regular (non-manylinux) wheel on Fedora 33/32 and > install and import it on Fedora 34. It should work. > The other way around will most likely also work, unless Fedora 34 has > an incompatible glibc update. > > == User Experience == > Users of ppc64le and armv7hl Fedora (and future RHEL) will have a > closer-to-upstream Python experience and will no longer suffer from > compatibility issues when they install or build manylinux wheels. > > == Dependencies == > No known dependencies. May the force be with us. > > == Contingency Plan == > * Contingency mechanism: Revert the change and rebuild all affected packages. > * Contingency deadline: Soft before the mass rebuild, so we could > leverage it for the revert-rebuilds. Hard before the beta freeze. > * Blocks release? No > * Blocks product? No > > == Documentation == > This page is the documentation. > > > -- Zdenek Dohnal Software Engineer Red Hat Czech - Brno TPB-C
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
