On 22. 01. 22 18:22, Kai A. Hiller wrote:
Hello everyone,
Hello Kai.
just a very quick and short reply, as I don't have time for a long and
complicated answer. See below.
I’m having trouble deciding on what is the best way to perform tests in Python
packages. Here are my thoughts and the solution I find most correct, but which
is kind of verbose and ugly to read. I’d be happy about comments. TL;DR and
code at the bottom.
For a Python package test to be reliable and representative of what will be
installed on a real Fedora system, I think it should be
* performed by the correct Python binary, i.e., %{python3}
* using the system-installed Python modules
* easy to read in the specfile
I found that running tests can go wrong in subtle ways:
* The wrong Python binary is used
* Python modules in the main build directory are used, not the ones installed
by the %install step – even a mix of both is possible
EXAMPLE
I’m assuming this common (for our use case unhelpful) packaging structure where
the Python modules and the test module are side-by-side in the main directory:
package/
|- module/
|- tests/
|- setup.py
|- tox.ini
|- …
This example uses tox to call the twisted trial test runner, which can be
called as Python module or as a standalone script. It should be similar to
other test runners. Here are approaches to running tests which are NOT working
properly with this simple %check section:
%check
sed -i 's|python -m twisted\.trial|…|' tox.ini
%tox
Broken 1: tox calling `python -m twisted.trial` (upstream default)
Used Python: .tox/py/bin/python :(
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES
Broken 2: tox calling `%{python3} -m twisted.trial`
Used Python: /usr/bin/python3
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES
Broken 3: tox calling `trial-3`
Used Python: /usr/bin/python3
sys.path[0]: /usr/bin
Uses system modules: YES
Finds tests module: NO
A first, still fragile, solution is choosing the calling method and PYTHONPATH
in a way that the order of module discovery is “first system modules, then cwd”:
%check
# Call test runner by its script, thus using the correct python and *avoiding*
# sys.path[0] being the cwd.
sed -i 's|python -m twisted\.trial|trial-3' tox.ini
# Prepend pwd to have access to the test module.
PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}:$PWD %tox
This works, but the modules in the main directory are only shadowed. Thus, they
might interfere with the tests if, e.g., the installation skips modules. This
can be fixed by
* moving to a new directory, copying only the test module folder over, or
* removing all modules in the current working dir (except the tests).
I arbitrarily chose removing for the following solution.
TL;DR
With all previously said, I’d arrive at using the following lines for running
my Python tests:
%check
# Make sure we use the correct python and and sys.path[0] is the cwd (so the
# test module *is* found).
sed -i 's|python|%{python3}' tox.ini
So, this is *not* actually needed. %tox will call (a symbolic link to)
%{python3} where tox configuration calls "python".
# Remove all Python modules from cwd, except the test module, to avoid their use.
find . -maxdepth 2 -name '__init__.py' -exec dirname {} \; | grep -v ^\./tests | xargs rm -rf
# Handles populating PYTHONPATH with system paths.
%tox
If you have some influence over upstream, you might want them to change this
scheme:
package/
|- src/
| | - module/
|- tests/
|- setup.py
|- tox.ini
That way, you don't have to do that.
Are my raised concerns valid? Are there already solutions I’m missing? Should
there be documentation/rpm macros to help with this?
Definitively valid, but I'll try to answer next week with more details.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
