F37 proposal: Python: Add -P to default shebangs (System-Wide Change proposal)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



https://fedoraproject.org/wiki/Changes/PythonSafePath

== Summary ==
The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P`
flag] will be added to the Python shebang macros
(`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that
adhere to those macros will change their Python shebangs from `#!
/usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result,
will no longer have the directory of the script (such as `/usr/bin`)
in `sys.path`. An opt-out mechanism exists.

== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Vstinner|Victor Stinner]]
* Email: python-maint@xxxxxxxxxx

== Detailed Description ==
All Python 3 shebang RPM macros will be changed to contain one more
flag: `-P`. Previously, they contained `-s`, now they will contain
`-sP`.

>From the [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
documentation for the `-P` option]:

:Don’t prepend a potentially unsafe path to `sys.path`:
:
:* `python -m module` command line: Don’t prepend the current working directory.
:* `python script.py` command line: Don’t prepend the script’s
directory. If it’s a symbolic link, resolve symbolic links.
:* `python -c code` and `python` (REPL) command lines: Don’t prepend
an empty string, which means the current working directory.

In shebangs, only the middle option (''don’t prepend the script’s
directory'') is relevant.

Consider the following executbale script installed as
`/usr/bin/let-there-be-fun`:

 #! /usr/bin/python3 -s
 import abc
 ...

When the script is directly executed (e.g. by running
`let-there-be-fun` from the console), the script's directory
(`/usr/bin`) is prepended to `sys.path`. Python tries to locate an
importable `abc` module in `/usr/bin` first. This can cause real
issues: [https://bugzilla.redhat.com/2057340 python3-notebook:
ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six].

When the shebang includes `-P`:

 #! /usr/bin/python3 -sP
 import abc
 ...

...the script's directory (`/usr/bin`) is '''not''' prepended to
`sys.path`. The change owners consider this approach safer for the
majority of Fedora's RPM packages.

By default, '''all standardly RPM-packaged Python packages with
scripts in `/usr/bin` will gain the `-P` flag in their shebang''',
assuming the software is packaged in a way that respects the Python
shebang RPM macros (see below for opt-out and explicit opt-in
mechanisms). Due to the variety of ways such scripts can be
created/packaged, there will likely be packages that will not be
affected by the change automatically. (In other words, the change is
applied on RPM macro level, no added mechanics to force the flag, such
as BRP scripts, are planned as part of this change.)

=== List of RPM macros that will gain `-P` ===

* `%{py3_shbang_opts}`
* `%{py3_shbang_opts_nodash}`
* `%{py3_shebang_flags}`
* `%{py_shbang_opts}`
* `%{py_shbang_opts_nodash}`
* `%{py_shebang_flags}`

=== Opting out ===

If the new behavior is not desirable to your package amend the macros
(e.g. with `sed`) to remove the `P` flag.

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, use:

 # Don't add -P to Python shebang
 # This package only works when /usr/bin is in sys.path (use your own
rationale here)
 %global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, use:

 # Don't add -P to Python shebang
 # This package only works when /usr/bin is in sys.path (use your own
rationale here)
 %global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)

(The only difference is the name of the macro.)

=== Opting in ===

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, the standard set of Python shebang flags is
applied to all files with Python shebangs installed in `/usr/bin/`.

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, the standard set of Python shebang flags might be
applied to some files and not applied to others depending on the exact
structure of the packaged software.

If you wish to explicitly apply the standard set of Python shebang
flags on a certain file that is not handled automatically, use
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#py3_shebang_fix
the `%py3_shebang_fix` macro].

=== What if the packager changes `%__python3` to an older version of Python ===

The `-P` flag was introduced in Python 3.11. When `%__python3` is
redefined to an older Python version, e.g. `/usr/bin/python3.10`,
including the `-P` flag in shebangs would break the scripts. Hence,
the flag will be included conditionally, presumably somehow like this:

 <nowiki>%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P'
if hasattr(sys.flags, 'safe_path') else '')")</nowiki>

=== What if the admin/user changes `/usr/bin/python3` to an older
version of Python ===

The `-P` flag was introduced in Python 3.11. When an admin/user
changes `/usr/bin/python3` to point to an older version of Python,
e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would
break the scripts.

However, changing `/usr/bin/python3` to a different Python would
''brick'' a Fedora system even now. So we don't consider that an
issue. See an example that changes `/usr/bin/python3` to Python 3.9
(don't try this at home):

 [root@086a2804411a /]# head -n1 /usr/bin/dnf
 #!/usr/bin/python3

 [root@086a2804411a /]# dnf --version
 4.12.0
 ...

 [root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

 [root@086a2804411a /]# dnf --version
 Traceback (most recent call last):
   File "/usr/bin/dnf", line 61, in <module>
     from dnf.cli import main
 ModuleNotFoundError: No module named 'dnf'

=== Risks ===

As with any other change, there is a risk that this will break things.
The change owners plan to test the change extensively via Copr before
they deploy the change in Rawhide. If things go badly, they are
prepared to delay or cancel the change.

The [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python RPM macros] set the shebang flags by a weird hack. As
a result, it is not well-defined what scripts will be affected by this
change. The change owners are aware that:

# not all Python scripts in `/usr/bin` will have the `-P` flag automatically
# some scripts '''not''' in `/usr/bin` might gain the `-P` flag as well

The first point is an acceptable gradual deployment of the default
flag. The second point is not very dangerous because we don't except
users to directly execute Python scripts via shebangs when such
scripts are not in `$PATH`. If a problematic package is found, it can
opt-out easily. If this causes too much friction, we will only change
the flag used in the `%pyproject_install` macro, leaving packages with
the "legacy" macros intact.

== Feedback ==
* Relevant upstream discussion: [https://discuss.python.org/t/13896
Should console_scripts entry points exclude the scripts directory from
sys.path?]
* Adding `-P` to Python:
[https://github.com/python/cpython/issues/57684 GitHub issue],
[https://mail.python.org/archives/list/python-dev@xxxxxxxxxx/thread/IU5Q2AXAURFVDPRWNU3BDFVKV2QX5NOR/
email thread]
* [https://github.com/rpm-software-management/dnf/pull/1815#pullrequestreview-891177878
dnf maintainers response for doing this explicitly in dnf first]

Generally, people seem to think that this is a good thing. They are
afraid to change the default behavior of Python but welcome using this
flag for system-installed scripts.

== Benefit to Fedora ==
Python programs in `/usr/bin` will be less fragile to other random
files being present in `/usr/bin`.
Real hard-to-debug issues like [https://bugzilla.redhat.com/2057340
python3-notebook: ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six]
will not happen.

== Scope ==
* Proposal owners:
** Test everything in Copr
** If everything works, change the flags either before the Python 3.11
rebuild or before the Fedora 37 Mass Rebuild
** Provide guidance to packagers, fix bugs if needed

* Other developers:
** Observe their packages, find and report bugs, opt-out if needed
** Volunteerily opt-in by calling the `%py3_shebang_fix` macro and/or
by converting their packages to `%pyproject_install`

* Release engineering: [https://pagure.io/releng/issue/10784 #10784]
* Policies and guidelines: The new flag needs to be documented in the
Python packaging guidelines (old and new)
* Trademark approval: not needed for this Change
* Alignment with Objectives: no


== Upgrade/compatibility impact ==
No impact is anticipated.

== How To Test ==
* Low level: Examine the value of the changed RPM macros, it should
contain the `-P` flag
* Middle level: Examine the shebang lines of RPM-installed Python
scripts in `/usr/bin`, it should contain the `-P` flag
* High level: Tests that RPM-installed Python scripts still behave as
expected but don't try to import stuff from `/usr/bin`

== User Experience ==
Users of RPM-installed scripts should get a safer experience by default.
Users of Python should not observe a difference, the behavior is not a
new default: the `-P` flag needs to be explicitly used.

== Dependencies ==
We need [[Changes/Python3.11]] first.

== Contingency Plan ==
* Contingency mechanism: defer to F38; only add `-P` to shebangs in
`%pyproject_install` to keep backward compatibility of the old macros;
revert and rebuild
* Contingency deadline: 1 week before the beta freeze
* Blocks release? No

== Documentation ==
* This page
* TBD Updated Python guidelines
* https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
* https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights
(Security improvements)

== Release Notes ==
TBD


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux