https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setuptools == Summary == We'll do two things to reduce the number of Python packages unnecessarily Requiring python3-setuptools: # python3-devel will no longer Require python3-setuptools # packages with console_scripts/gui_scripts entry points will no longer automatically Require python3-setuptools == Owner == * Name: [[User:Thrnciar|Tomáš Hrnčiar]] * Name: [[User:Churchyard|Miro Hrončok]] * Email: python-maint@xxxxxxxxxx == Detailed Description == Historically, for over a decade, [https://pypi.org/project/setuptools/ setuptools] was omnipresent in Python packaging. For that reason, the python3-devel package required python3-setuptools. This is however changing, with [https://www.python.org/dev/peps/pep-0517/ PEP517]- and [https://www.python.org/dev/peps/pep-0518/ PEP518]-based upstream packaging, setuptools changed to a very popular yet only optional way of building packages, many upstreams switched to poetry or flit instead. Since we received multiple user reports about an unnecessary dependency, we wanted to [https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/message/ARFZB56ULRQNT3FEAC4YXXZDIUHG4M5P/ get rid of the dependency], but it was not possible until now. We'll do the two following changes (neither one depends on the other): === python3-devel will stop Requiring python3-setuptools === When we wanted to remove the dependency before, we have encountered a problem, which is best summarized in the Python spec file: # This is not "API" (packages that need setuptools should still BuildRequire it) # However some packages apparently can build both with and without setuptools # producing egg-info as file or directory (depending on setuptools presence). # Directory-to-file updates are problematic in RPM, so we ensure setuptools is # installed when -devel is required. # See https://bugzilla.redhat.com/show_bug.cgi?id=1623914 # See https://fedoraproject.org/wiki/Packaging:Directory_Replacement Requires: (python3-setuptools if rpm-build) To avoid this problem, python3-devel currently requires `(python3-setuptools if rpmbuild)` for backwards compatibility. Other than that, python3-setuptools was always brought into the buildroot transitively with python3-rpm-generators. Since the [https://lists.fedoraproject.org/archives/list/python-devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/57QTZGF7FJPE2QQSGPJ3RGI4G5SLN6BM/ generators no longer use setuptools but packaging] we can finally remove the dependency. Packages that don't BuildRequire python3-setuptools but use setuptools during the build will fall into one of the following three categories: * They will fail to build from source with errors like: `ModuleNotFoundError: No module named 'setuptools'`. * They will successfully build from source because they will fallback to the standard library distutils module - their `.egg-info` directory will become a text file instead, causing errors when RPM tries to update the package (and they will possibly miss some runtime dependencies because the metadata generated by distutils is not complete). * They will continue to build unchanged because one or more packages they BuildRequire pulls in python3-setuptools transitively - at any point in the future, this can change and either (1) or (2) will happen. We tested this change in the past when we [https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/message/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/ asked] other maintainers to explicitly BuildRequire python3-setuptools. Some of them responded, but many did not. Our analysis was based on grepping sources for any mention of setuptools, such as `import setuptools` or `from setuptools import `. Nowadays, the same grep-based query showed us roughly the same number (~350) of packages relying on python3-setuptools without BuildRequiring it. To see the direct impact, we have also briefly removed the Requires from python3-devel package used in testing [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/packages/ python3.10] in COPR. It revealed only 140 packages that fail to build from the source because of the change, however many packages bring in python3-setuptools transitively. Some of the packages might build successfully even without python3-setuptools, because they fallback to distutils (from the Python standard library) if dependency on setuptools is not satisfied. To avoid such problems we will: * Mass update packages we analyzed as impacted to add the missing BuildRequires on python3-setuptools. * Do the change while doing the [[Changes/Python3.10|update from Python 3.9 to Python 3.10]] so the path of the `.egg-info` changes from `/usr/lib(64)/python3.9/site-packages/...egg-info` to `/usr/lib(64)/python3.10/site-packages/...egg-info`, hence no longer causing troubles on upgrades if some packages actually do change it from directory to file (and changing it the other way around in the future is safe). ==== How do we mass update the spec files ==== We plan to write a script to detect what notation for BuildRequires definition each spec file uses. Example of the most common occurrences: BuildRequires: python%{python3_pkgversion}-setuptools BuildRequires: %{py3_dist setuptools} BuildRequires: python3-setuptools BuildRequires: python3dist(setuptools) We will try to keep the added BuildRequires as much consistent as possible with the existing Python-related BuildRequires (including e.g. whitespace). We will not bump the release or add a changelog entry (the Python 3.10 rebuild will take care of it). To show how the proposed modifications will look like we plan to create a git repository with all the spec files and commit the change there. The diff will be shared on the devel and python-devel mailing lists, so maintainers can easily see the upcoming change and can provide feedback to us. ==== When do we update the spec files ==== We plan to mass update the spec files in the middle of April 2021, if you wish to opt-out from the mass update, please let us know prior to 15th of April 2021. ==== What spec files we will update ==== Below is the list of packages we plan to add BuildRequires for python3-setuptools to. If you wish to opt-out of this update, please edit this page and move your package to the list of packages to be ignored (listed below this one). '''Packages to be updated:''' * 0ad ignatenkobrain pcpa pwalter * OpenMolcas jussilehtola * PyGreSQL hhorak jmlich odubaj panovotn pkubat praiskup * R2spec pingou * ViTables tnorth zbyszek * ansible-review dcallagh ttrinks * barman slaanesh tsao * blender design-sw hobbes1069 ignatenkobrain kwizart luya roma s4504kr slaanesh * btrfs-progs ignatenkobrain josef ngompa sandeen * bumpversion duriantang jdornak * calypso rathann * cinch greghellings * commissaire-client mbarnes smilner * cppcheck c72578 jussilehtola sgrubb * cranc lenkaseg * crudini apevec jruzicka pbrady * cxxtest mgieseki * datanommer ralph * dlib bizdelnick luya * ec2-hibinit-agent davdunc * electrum tredaell * fbthrift dcavalca filbranden salimma * fedora-messaging abompard * firefox alexl caolanm erack gecko-maint jgrulich kalev kengert mbarnes rhughes rstrode sharkcz stransky tpopela ueno xhorak * folly dcavalca filbranden salimma * fontforge frixxon kevin pnemade * gajim michich suraia * gau2grid jussilehtola * gaupol music * gfal2-python adev andreamanzi * ginga lupinix * git-filter-repo asn * gns3-gui kwizart * gns3-net-converter kwizart * gns3-server kwizart nucleo * gpsd fab mlichvar ttorling * h5py stevetraylen terjeros * khard mathstuf sdyroff * kismon fab * koji ausil kevin mikem puiterwijk * lammps ellio167 junghans * legofy lkf williamjmorenor * libcaca hubbitus slaanesh thias * libiptcdata dcm hobbes1069 jchaloup * libnuml sagitter * libreoffice caolanm dtardon erack sbergmann * lldb airlied daveisfera jankratochvil sergesanspaille siddharths tstellar * llvm dmalcolm ignatenkobrain jakub jistone kyle scottt sergesanspaille siddharths tstellar * llvm10 sergesanspaille tstellar * llvm11 sergesanspaille tstellar * llvm7.0 jistone petersen sergesanspaille tstellar * llvm9.0 jistone sergesanspaille tstellar * mercurial kiilerix nbecker pstodulk * mod_wsgi jdornak jkaluza jorton lmacken mrunge * moose zbyszek * mypaint avsej * mysql-connector-python hhorak hubbitus hvad mschorm * nototools mfabian pwu * officeparser rebus * offlineimap cicku dodji sergesanspaille teuf * openbabel alexpl jussilehtola rathann * percol hubbitus * poetry churchyard thrnciar * poezio fantom louizatakk * porcupine kushal * protontricks atim * protonvpn-cli calexandru2018 jflory7 * prunerepo frostyx praiskup * pyaudio chkr * pyhunspell mfabian * pyodbc fjanus hhorak * pyscard orion sjenning * pyserial stingray * python-CommonMark jujens * python-GridDataFormats rathann * python-OBD rathann * python-Pyped uggla * python-Rtree volter * python-acoustid terjeros * python-agate jujens * python-aiodns fantom * python-ansicolors fab * python-apprise lead2gold * python-apsw cicku dfateyev maci * python-arviz sergiopr * python-astroplan sergiopr * python-astropy-healpix lupinix * python-astroquery lupinix * python-astroscrappy lupinix * python-asttokens zbyszek * python-audioread terjeros * python-autobahn fab jujens * python-autopep8 mrunge ndipanov * python-b2sdk jonny * python-bloom cottsay rmattes * python-blosc tnorth zbyszek * python-box dmsimard fab * python-btchip jonny xenithorb * python-cached_property adamwill immanetize * python-carbon jsteffan piotrp * python-ccdproc lupinix * python-certbot-apache nb * python-chai kevin pingou ralph * python-cmigemo hubbitus * python-colorspacious rathann * python-construct moezroy terjeros * python-contextlib2 abompard pingou ralph tjikkun * python-cookiecutter chedi wakko666 * python-crochet abompard * python-css-parser zbyszek * python-dbfread jujens * python-debrepo ktdreyer * python-decopatch zbyszek * python-descartes qulogic * python-dialog mjakubicek noodles raphgro sundaram zbyszek * python-dijitso limb * python-django-contact-form mrunge * python-django-health-check dmsimard * python-django-registration orphan * python-django-reversion mrunge * python-django-tagging jdornak mrunge piotrp * python-django-tastypie bkabrda cquad mrunge stevetraylen * python-docx kushal124 * python-dtfabric fab * python-dukpy zbyszek * python-editorconfig barracks510 * python-ephem fab * python-et_xmlfile jujens * python-etcd mbarnes smilner * python-fasteners mrunge * python-ffc zbyszek * python-fields cottsay * python-fisx zbyszek * python-fitsio lupinix * python-flake8-docstrings cottsay * python-flask-gravatar devrim orphan * python-flask-htmlmin devrim iztokf * python-flask-paranoid devrim orphan * python-flask-rstpages rmarko * python-flask-security devrim orphan * python-flask-sphinx-themes devrim orphan * python-flask-wtf-decorators frostyx * python-formats uggla * python-fypp rathann * python-gevent-eventemitter atim * python-google-i18n-address pwouters * python-googletrans lyessaadi * python-graphql-relay fab * python-heapdict qulogic * python-html5-parser kevin * python-htmlmin jujens * python-humblewx rickardlindberg * python-hupper kevin * python-i3ipc alebastr * python-inotify jfilak stevetraylen terjeros * python-jep raphgro * python-jinja2-cli jujens * python-jinja2-time chedi wakko666 * python-jnius raphgro * python-joblib besser82 ignatenkobrain sergiopr * python-jsonmodels oanson * python-jsonrpclib ihrachyshka jonny * python-kerberos rcritten simo * python-kitchen kevin pingou ralph * python-landslide echevemaster salimma * python-lark-parser totol * python-leather jujens * python-libsass nonamedotc * python-libusb1 jonny * python-lmdb jruzicka pspacek tkrizek * python-logfury jonny * python-makefun zbyszek * python-managesieve stevetraylen * python-matrix-nio ankursinha * python-meld3 kevin stevetraylen tsao * python-minibelt uggla * python-mmtf rathann * python-mnemonic jonny * python-music21 zbyszek * python-mwclient adamwill rdieter tuxbrewr * python-myhdl filiperosset * python-nbclient nonamedotc * python-nbxmpp michich suraia * python-networkmanager jdulaney * python-notario ktdreyer * python-oauth2 ignatenkobrain pjp spot sundaram * python-openoffice sharkcz * python-ouimeaux kni * python-pandas-datareader sergiopr * python-paste-script andreamanzi * python-patsy sergiopr * python-pbkdf2 jonny * python-pecan-notario ktdreyer * python-pelican firemanxbr mrunge * python-pexpect amcnabb fabiand ignatenkobrain radez swt2c tomspur * python-plaster-pastedeploy abompard * python-plumbum greghellings lorenzodalrio * python-poetry-core churchyard thrnciar * python-polib cicku dchen diegobz dshea ivazquez moezroy suanand * python-precis_i18n michich * python-proteus sharkcz * python-publicsuffix2 rathann * python-pulsectl pfrields * python-pvc raphgro * python-pycares fantom * python-pycha potty sharkcz * python-pygeoip kevin ralph * python-pylons-sphinx-themes abompard * python-pymc3 sergiopr * python-pynn ankursinha * python-pyotp icon * python-pypng kevin ralph * python-pyqtgraph swt2c * python-pyramid_sawing abompard * python-pysb zbyszek * python-pysignals kni * python-pyswip pampelmuse * python-pyte terjeros * python-pytest-astropy-header sergiopr * python-pytest-cases zbyszek * python-pytest-harvest zbyszek * python-pytest-mock fab jujens * python-pytest-repeat cottsay * python-pytest-steps zbyszek * python-pytest-watch jujens * python-pyvo lupinix * python-readthedocs-sphinx-ext jjames * python-recommonmark jujens * python-relatorio sharkcz * python-restructuredtext-lint jujens * python-requests-cache codeblock hobbes1069 * python-retrying apevec * python-rmtest lberk mgoodwin nathans * python-rosdep cottsay rmattes thofmann * python-sanction kevin ralph * python-scikit-learn besser82 ignatenkobrain lupinix sergiopr * python-scrapy echevemaster * python-setuptools-lint jdulaney * python-shamir-mnemonic jonny * python-simplemediawiki lmacken potty ralph * python-slixmpp fantom louizatakk * python-smbc twaugh zdohnal * python-snappy jujens * python-social-auth-core cqi * python-soupsieve zbyszek * python-spdx jbertozzi * python-spdx-lookup jbertozzi * python-sphinxcontrib-bibtex jjames * python-statsd pabelanger tdecacqu * python-steam atim * python-tables tnorth zbyszek * python-tempdir rathann * python-timeout-decorator jcapitao * python-tinydb suanand * python-tortilla uggla * python-tqdm ignatenkobrain sgallagh * python-tree-format chedi wakko666 * python-trezor jonny * python-twilio mich181189 * python-txaio fab jujens * python-unidecode pjp sundaram * python-unidiff dcallagh * python-upt-cpan jbertozzi * python-upt-fedora jbertozzi * python-upt-pypi jbertozzi * python-upt-rubygems jbertozzi * python-urwidtrees ttomecek * python-vdf atim * python-wand barracks510 * python-watchdog jsteffan jujens pingou * python-webencodings abompard * python-webpy mrunge * python-wsaccel jujens * python-xlwt leamas moezroy rathann * python-xvfbwrapper mrunge totol * python-yarl fab ignatenkobrain * python-zstandard rathann * python3-postgresql hhorak * python3-pytest-asyncio jujens * python3-saml dcallagh tchaikov * qemu berrange bonzini crobinso dwmw2 ehabkost jforbes lkundrak quintela rjones * rdkit giallu * root ellert * rpmspectool nphilipp * salt blarson dmurphy18 krionbsd * sepolicy_analysis vmojzis * solaar brouhaha rathann richardfearn tibbs * spec2scl jstanek * starcal hedayat * stomppy stevetraylen * sugar-speak callkalpa chimosky pbrobinson tuxbrewr * swid-tools adelton * swift-lang tachoknight * terminator dmaphy mattrose ohaessler * texlive-base spot * toot alciregi * translate-toolkit cicku dwayne petersen suanand * trellis lkundrak somlo * tryton sharkcz * trytond sharkcz * trytond-account sharkcz * trytond-account-be sharkcz * trytond-account-de-skr03 sharkcz * trytond-account-invoice sharkcz * trytond-account-invoice-history sharkcz * trytond-account-invoice-line-standalone sharkcz * trytond-account-product sharkcz * trytond-account-statement sharkcz * trytond-account-stock-anglo-saxon sharkcz * trytond-account-stock-continental sharkcz * trytond-analytic-account sharkcz * trytond-analytic-invoice sharkcz * trytond-analytic-purchase sharkcz * trytond-analytic-sale sharkcz * trytond-company sharkcz * trytond-company-work-time sharkcz * trytond-country sharkcz * trytond-currency sharkcz * trytond-dashboard sharkcz * trytond-google-maps sharkcz * trytond-ldap-authentication sharkcz * trytond-party sharkcz * trytond-party-siret sharkcz * trytond-product sharkcz * trytond-product-cost-fifo sharkcz * trytond-product-cost-history sharkcz * trytond-product-price-list sharkcz * trytond-project sharkcz * trytond-project-plan sharkcz * trytond-project-revenue sharkcz * trytond-purchase sharkcz * trytond-purchase-invoice-line-standalone sharkcz * trytond-sale sharkcz * trytond-sale-opportunity sharkcz * trytond-sale-price-list sharkcz * trytond-stock sharkcz * trytond-stock-forecast sharkcz * trytond-stock-inventory-location sharkcz * trytond-stock-location-sequence sharkcz * trytond-stock-product-location sharkcz * trytond-stock-supply sharkcz * trytond-stock-supply-day sharkcz * trytond-timesheet sharkcz * ubertooth avsej * uhd jskarvad * upt jbertozzi * uwsgi kad * watchman dcavalca filbranden salimma * wine-mono mooninite * winpdb spot * xrootd ellert simonm * xtensor-python sergesanspaille * yawn jsafrane miminar vcrhonek '''Packages to be ignored:''' * fontforge - false positive * qemu - false positive * xrootd - false positive === Packages with console_scripts/gui_scripts entry points will no longer automatically Require python3.Xdist(setuptools) === Packages that use the `console_scripts` or `gui_scripts` entry points used to generate scripts for them that imported `pkg_resources` (from setuptools) on runtime and thus all such packages needed to Require setuptools on runtime. This is an example of such generated script from Fedora 32: #!/usr/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg' __requires__ = 'fedpkg==1.40' import re import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('fedpkg==1.40', 'console_scripts', 'fedpkg')() ) The Python dist RPM dependency generator (`/usr/lib/rpm/fileattrs/pythondist.attr`) hence generated a Requires on `python3.Xdist(setuptools)` for all packages with scripts like the one above. Starting from Python 3.8 and setuptools 47, this is no longer required, because the scripts will use `importlib.metadata` from the standard library instead of `pkg_resources` from setuptools. The scripts now looks like this: #!/usr/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg' import re import sys # for compatibility with easy_install; see #2198 __requires__ = 'fedpkg==1.40' try: from importlib.metadata import distribution except ImportError: try: from importlib_metadata import distribution except ImportError: from pkg_resources import load_entry_point ... Since the scripts are generated during build, the packages no longer need to Require setuptools on runtime. The Python dist RPM dependency generator will no longer generate the Requires on `python3.Xdist(setuptools)`. The change is prepared in a pull request (https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/38). We plan to merge it as soon as it is ready. It only changes the behavior for Python 3.10 or newer, hence it will take affect in the Python 3.10 copr only, until Python is updated to 3.10 in Fedora proper. Packages that actually import from the `setuptools` or `pkg_resources` module on runtime should explicitly require setuptools, but some of the upstreams may not do that. We don't expect many packages negatively affected by this, but since this is not strictly backwards compatible, we are only planning to do it in Fedora 35+. == Feedback == There was some discussion on various mailing list threads, but no other alternatives were proposed. * https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/ * https://lists.fedoraproject.org/archives/list/python-devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/ * https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/BL7XMXVEHSDZDMH22YET3I4EK66PK4NI/ * https://lists.fedoraproject.org/archives/list/python-devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/BL7XMXVEHSDZDMH22YET3I4EK66PK4NI/ == Benefit to Fedora == >From [https://docs.fedoraproject.org/en-US/packaging-guidelines/#buildrequires Fedora Packaging Guidelines]: ''It is important that your package list all necessary build dependencies using the BuildRequires: tag.'' As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when [https://docs.fedoraproject.org/en-US/packaging-guidelines/Directory_Replacement/ upgrading directories to files]. == Scope == Some packages use distutils as a fallback in case setuptools is unavailable. The problem is that setuptools generates egg.info directory and distutils creates egg.info file. This would lead to failures caused by the known issue of RPM that cannot change directory to file. To avoid this we plan to do a mass spec update before we upgrade Python to version 3.10. This way we ensure that path will be correct after the mass rebuild. * Proposal owners: # Test affected packages in Copr. # Do the mass spec update to introduce missing BuildRequire to them. * Other developers: If your package requires setuptools on runtime, make sure it specifies it explicitly. There should be no additional work for developers other than that. There is a possibility that some packages will slip through our grep based query and fail to build, but we'll detect this in our testing Copr and fix them. * Release engineering: not needed for this Change * Policies and guidelines: already exist * Trademark approval: not needed for this Change * Alignment with Objectives: minimization == Upgrade/compatibility impact == All affected packages needs rebuild after the mass spec update but since we plan to do it with the upgrade to Python 3.10 we will rebuild it altogether. Users should not be affected by upgrades in any way. == How To Test == This does not work yet, but will soon: Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/ Python 3.10 COPR] whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch. == User Experience == Regular distro users shouldn't notice any change in system behaviour, except for packages that use setuptools or pkg_resources on runtime and fail to specify that dependency. Such packages will fail to run with ModuleNotFoundErrors and should be fixed by their maintainers. == Dependencies == 300+ packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update. == Contingency Plan == * Contingency mechanism: Commit removing python3-devel Requires on `(python3-setuptools if rpmbuild)` will be reverted. The change in Python dist RPM depndency generator will be reverted (This mechanisms can be applied independently.). * Contingency deadline: mass rebuild * Blocks release? No == Documentation == This page is a documentation, this is not a user facing change. -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
