Fedora 32 Self-Contained Change proposal: Build Python with -fno-semantic-interposition for better performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup

Simplified version of another change proposal|This change was
originally proposed for [[Releases/32|Fedora 32]] as
[[Changes/PythonStaticSpeedup]], however based on
[https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/NWPVQSKVWDKA75PDEIJNJIFL5C5SJXB2/
community feedback], it has been significantly reduced.

== Summary ==
We add the <code>-fno-semantic-interposition</code> compiler/linker
flag when building Python interpreters, as it provides significant
performance improvement, up to 27% depending on the workload. Users
will no longer be able to use LD_PRELOAD to override a symbol from
libpython, which we consider a good trade off for the speedup.

== Owner ==
* Name: [[User:Cstratak| Charalampos Stratakis]], [[User:Vstinner|
Victor Stinner]], [[User:Churchyard| Miro Hrončok]]
* Email: python-maint@xxxxxxxxxx
* Shout-out: [[User:Jankratochvil|Jan Kratochvíl]] for first
suggesting this instead of the original proposal, followed by
[[User:Kkofler|Kevin Kofler]]. [[User:Fweimer|Florian Weimer]] for
providing answers to our questions. David Gray for originally
suggesting to link Python statically to gain performance.

== Detailed Description ==

When we build the Python interpreter with the
<code>-fno-semantic-interposition</code> compiler/linker flag, we can
achieve a performance gain of 5% to 27% depending on the workload.
Link time optimizations and profile guided optimizations also have a
greater impact when python3 is built this way.

As a negative side effect, it disables the LD_PRELOAD feature: it's no
longer possible to override symbols in libpython with LD_PRELOAD.

Interposition is enabled by default in compilers like GCC: function
calls to a library goes through a "Procedure Linkage Table" (PLT).
This indirection is required to allow a library loaded by LD_PRELOAD
environment variable to override a function. The indirection puts more
pressure on the CPU level 1 cache (instruction cache). In term of
performance, the main drawback is that function calls from a library
to the same library cannot be inlined, to respect the interposition
semantics. Inlining is usually a big win in term of performance.

Disabling interposition for libpython removes the overhead on function
calls by avoiding the PLT indirection, and allows to inline more
function calls. We're describing function calls from libpython to
libpython, something which is very common in Python: almost all
function calls are calls from libpython to libpython.

If Fedora users need to use LD_PRELOAD to override symbols in
libpython, the recommend way is to build a custom Python without
<code>-fno-semantic-interposition</code>.

It is still possible to use LD_PRELOAD to override symbols in other
libraries (for example in glibc).

=== Affected Pythons ===

Primarily, we will change the interpreter in the {{package|python3}}
package, that is Python 3.8 in Fedora 32 and any later version of
Python in future Fedora releases.

Impact on other Python packages (and generally software using Python)
is not anticipated (other than the possible speedup).

We will also change the
[https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.html
alternate Python interpreters] where possible and useful, primarily
the upstream supported versions of CPython, such as
{{package|python39}} (if already packaged), {{package|python37}} and
{{package|python36}}.

=== Affected Fedora releases ===

This is a Fedora 32 change and it will be implemented in Rawhide
(Fedora 32) only. Any future versions of Fedora will inherit the
change until it is reverted for some reason.

If it turns out that there are absolutely no issues, we might consider
backporting the speedup to already released Fedora versions (for
example Fedora 31). Such action would be separately coordinated with
[https://docs.fedoraproject.org/en-US/fesco/ FESCo].

== Benefit to Fedora ==
Python's performance will increase significantly depending on the
workload. Since many core components of the OS also depend on Python
this could lead to an increase in their performance as well, however
individual benchmarks will need to be conducted to verify the
performance gain for those components.

[https://pyperformance.readthedocs.io/ pyperformance] results,
ignoring differences smaller than 5%:

(See change proposal)

== Scope ==
* Proposal owners:
** Review and merge the
[https://src.fedoraproject.org/rpms/python3/pull-request/151 pull
request with the implementation].
** Monitor Koschei for significant problems.
** Backport the change to alternate Python versions.
* Other developers are encouraged to check if their package works as expected
* Release engineering: N/A (not needed for this Change) -- this change
does not require a mass rebuild nor any other special releng work
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==
Python package maintainers should verify that their packages work as
expected and the only impact the end users should see is a performance
increase for workloads relying on Python.

== How To Test ==
Test that everything Python related in Fedora works as usual.

=== Was the flag applied test ===

You can test whether the <code>-fno-semantic-interposition</code> flag
was applied for your Python build:

<pre>
>>> import sysconfig
>>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_CFLAGS') + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
True
>>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_LDFLAGS') + sysconfig.get_config_var('PY_LDFLAGS_NODIST'))
True
</pre>

Before the change, you would see <code>False</code>, <code>False</code>.

=== Performance test ===

The performance speedup can be measured using the official Python
benchmark suite [https://pyperformance.readthedocs.io/ pyperformance]:
see [https://pyperformance.readthedocs.io/usage.html#run-benchmarks
Run benchmarks].

== User Experience ==
Python based workloads should see a performance gain of up to 27%.

== Dependencies ==
This change is not dependent on anything else.

== Contingency Plan ==
* Contingency mechanism: If issues appear that cannot be fixed in a
timely manner the change can be easily reverted and will be considered
again for the next fedora release.
* Contingency deadline: Before the beta freeze of Fedora 32 (2020-02-25)
* Blocks release? Yes
* Blocks product? None

== Documentation ==
This change proposal has all the documentation.

See the [[Changes/PythonStaticSpeedup|previous change proposal]] and
the [https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/NWPVQSKVWDKA75PDEIJNJIFL5C5SJXB2/
thread about it on the devel mailing list] for more relevant
information about what we are not doing

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux