https://fedoraproject.org/wiki/Changes/PythonSafePath == Summary == The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P` flag] will be added to the Python shebang macros (`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that adhere to those macros will change their Python shebangs from `#! /usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result, will no longer have the directory of the script (such as `/usr/bin`) in `sys.path`. An opt-out mechanism exists. == Owner == * Name: [[User:Churchyard|Miro Hrončok]], [[User:Vstinner|Victor Stinner]] * Email: python-maint@xxxxxxxxxx == Detailed Description == All Python 3 shebang RPM macros will be changed to contain one more flag: `-P`. Previously, they contained `-s`, now they will contain `-sP`. >From the [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P documentation for the `-P` option]: :Don’t prepend a potentially unsafe path to `sys.path`: : :* `python -m module` command line: Don’t prepend the current working directory. :* `python script.py` command line: Don’t prepend the script’s directory. If it’s a symbolic link, resolve symbolic links. :* `python -c code` and `python` (REPL) command lines: Don’t prepend an empty string, which means the current working directory. In shebangs, only the middle option (''don’t prepend the script’s directory'') is relevant. Consider the following executbale script installed as `/usr/bin/let-there-be-fun`: #! /usr/bin/python3 -s import abc ... When the script is directly executed (e.g. by running `let-there-be-fun` from the console), the script's directory (`/usr/bin`) is prepended to `sys.path`. Python tries to locate an importable `abc` module in `/usr/bin` first. This can cause real issues: [https://bugzilla.redhat.com/2057340 python3-notebook: ImportError: bad magic number in six] and [https://github.com/benjaminp/six/issues/359 bad magic number in six]. When the shebang includes `-P`: #! /usr/bin/python3 -sP import abc ... ...the script's directory (`/usr/bin`) is '''not''' prepended to `sys.path`. The change owners consider this approach safer for the majority of Fedora's RPM packages. By default, '''all standardly RPM-packaged Python packages with scripts in `/usr/bin` will gain the `-P` flag in their shebang''', assuming the software is packaged in a way that respects the Python shebang RPM macros (see below for opt-out and explicit opt-in mechanisms). Due to the variety of ways such scripts can be created/packaged, there will likely be packages that will not be affected by the change automatically. (In other words, the change is applied on RPM macro level, no added mechanics to force the flag, such as BRP scripts, are planned as part of this change.) === List of RPM macros that will gain `-P` === * `%{py3_shbang_opts}` * `%{py3_shbang_opts_nodash}` * `%{py3_shebang_flags}` * `%{py_shbang_opts}` * `%{py_shbang_opts_nodash}` * `%{py_shebang_flags}` === Opting out === If the new behavior is not desirable to your package amend the macros (e.g. with `sed`) to remove the `P` flag. If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ current Python packaging guidelines], e.g. `%pyproject_wheel` and `%pyproject_install`, use: # Don't add -P to Python shebang # This package only works when /usr/bin is in sys.path (use your own rationale here) %global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//) If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ 201x-era Python packaging guidelines], e.g. `%py3_build` and `%py3_install`, use: # Don't add -P to Python shebang # This package only works when /usr/bin is in sys.path (use your own rationale here) %global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//) (The only difference is the name of the macro.) === Opting in === If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ current Python packaging guidelines], e.g. `%pyproject_wheel` and `%pyproject_install`, the standard set of Python shebang flags is applied to all files with Python shebangs installed in `/usr/bin/`. If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ 201x-era Python packaging guidelines], e.g. `%py3_build` and `%py3_install`, the standard set of Python shebang flags might be applied to some files and not applied to others depending on the exact structure of the packaged software. If you wish to explicitly apply the standard set of Python shebang flags on a certain file that is not handled automatically, use [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#py3_shebang_fix the `%py3_shebang_fix` macro]. === What if the packager changes `%__python3` to an older version of Python === The `-P` flag was introduced in Python 3.11. When `%__python3` is redefined to an older Python version, e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would break the scripts. Hence, the flag will be included conditionally, presumably somehow like this: <nowiki>%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P' if hasattr(sys.flags, 'safe_path') else '')")</nowiki> === What if the admin/user changes `/usr/bin/python3` to an older version of Python === The `-P` flag was introduced in Python 3.11. When an admin/user changes `/usr/bin/python3` to point to an older version of Python, e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would break the scripts. However, changing `/usr/bin/python3` to a different Python would ''brick'' a Fedora system even now. So we don't consider that an issue. See an example that changes `/usr/bin/python3` to Python 3.9 (don't try this at home): [root@086a2804411a /]# head -n1 /usr/bin/dnf #!/usr/bin/python3 [root@086a2804411a /]# dnf --version 4.12.0 ... [root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3 [root@086a2804411a /]# dnf --version Traceback (most recent call last): File "/usr/bin/dnf", line 61, in <module> from dnf.cli import main ModuleNotFoundError: No module named 'dnf' === Risks === As with any other change, there is a risk that this will break things. The change owners plan to test the change extensively via Copr before they deploy the change in Rawhide. If things go badly, they are prepared to delay or cancel the change. The [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ 201x-era Python RPM macros] set the shebang flags by a weird hack. As a result, it is not well-defined what scripts will be affected by this change. The change owners are aware that: # not all Python scripts in `/usr/bin` will have the `-P` flag automatically # some scripts '''not''' in `/usr/bin` might gain the `-P` flag as well The first point is an acceptable gradual deployment of the default flag. The second point is not very dangerous because we don't except users to directly execute Python scripts via shebangs when such scripts are not in `$PATH`. If a problematic package is found, it can opt-out easily. If this causes too much friction, we will only change the flag used in the `%pyproject_install` macro, leaving packages with the "legacy" macros intact. == Feedback == * Relevant upstream discussion: [https://discuss.python.org/t/13896 Should console_scripts entry points exclude the scripts directory from sys.path?] * Adding `-P` to Python: [https://github.com/python/cpython/issues/57684 GitHub issue], [https://mail.python.org/archives/list/python-dev@xxxxxxxxxx/thread/IU5Q2AXAURFVDPRWNU3BDFVKV2QX5NOR/ email thread] * [https://github.com/rpm-software-management/dnf/pull/1815#pullrequestreview-891177878 dnf maintainers response for doing this explicitly in dnf first] Generally, people seem to think that this is a good thing. They are afraid to change the default behavior of Python but welcome using this flag for system-installed scripts. == Benefit to Fedora == Python programs in `/usr/bin` will be less fragile to other random files being present in `/usr/bin`. Real hard-to-debug issues like [https://bugzilla.redhat.com/2057340 python3-notebook: ImportError: bad magic number in six] and [https://github.com/benjaminp/six/issues/359 bad magic number in six] will not happen. == Scope == * Proposal owners: ** Test everything in Copr ** If everything works, change the flags either before the Python 3.11 rebuild or before the Fedora 37 Mass Rebuild ** Provide guidance to packagers, fix bugs if needed * Other developers: ** Observe their packages, find and report bugs, opt-out if needed ** Volunteerily opt-in by calling the `%py3_shebang_fix` macro and/or by converting their packages to `%pyproject_install` * Release engineering: [https://pagure.io/releng/issue/10784 #10784] * Policies and guidelines: The new flag needs to be documented in the Python packaging guidelines (old and new) * Trademark approval: not needed for this Change * Alignment with Objectives: no == Upgrade/compatibility impact == No impact is anticipated. == How To Test == * Low level: Examine the value of the changed RPM macros, it should contain the `-P` flag * Middle level: Examine the shebang lines of RPM-installed Python scripts in `/usr/bin`, it should contain the `-P` flag * High level: Tests that RPM-installed Python scripts still behave as expected but don't try to import stuff from `/usr/bin` == User Experience == Users of RPM-installed scripts should get a safer experience by default. Users of Python should not observe a difference, the behavior is not a new default: the `-P` flag needs to be explicitly used. == Dependencies == We need [[Changes/Python3.11]] first. == Contingency Plan == * Contingency mechanism: defer to F38; only add `-P` to shebangs in `%pyproject_install` to keep backward compatibility of the old macros; revert and rebuild * Contingency deadline: 1 week before the beta freeze * Blocks release? No == Documentation == * This page * TBD Updated Python guidelines * https://docs.python.org/3.11/using/cmdline.html#cmdoption-P * https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights (Security improvements) == Release Notes == TBD -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
