Hello everyone,
I’m having trouble deciding on what is the best way to perform
tests in Python packages. Here are my thoughts and the solution I
find most correct, but which is kind of verbose and ugly to read.
I’d be happy about comments. TL;DR and code at the bottom.
For a Python package test to be reliable and representative of what will be installed on a real Fedora system, I think it should be
- performed by the correct Python binary, i.e., %{python3}
- using the system-installed Python modules
- easy to read in the specfile
I found that running tests can go wrong in subtle ways:
- The wrong Python binary is used
- Python modules in the main build directory are used, not the
ones installed by the %install step – even a mix of both is
possible
EXAMPLE
I’m assuming this common (for our use case unhelpful) packaging structure where the Python modules and the test module are side-by-side in the main directory:
package/
|- module/
|- tests/
|- setup.py
|- tox.ini
|- …
This example uses tox to call the twisted trial test runner, which can be called as Python module or as a standalone script. It should be similar to other test runners. Here are approaches to running tests which are NOT working properly with this simple %check section:
%check sed -i 's|python -m twisted\.trial|…|' tox.ini %tox
Broken 1: tox calling `python -m twisted.trial` (upstream
default)
Used Python: .tox/py/bin/python :(
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES
Broken 2: tox calling `%{python3} -m twisted.trial`
Used Python: /usr/bin/python3
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES
Broken 3: tox calling `trial-3`
Used Python: /usr/bin/python3
sys.path[0]: /usr/bin
Uses system modules: YES
Finds tests module: NO
A first, still fragile, solution is choosing the calling method and PYTHONPATH in a way that the order of module discovery is “first system modules, then cwd”:
%check
# Call test runner by its script, thus using the correct python and *avoiding*
# sys.path[0] being the cwd.
sed -i 's|python -m twisted\.trial|trial-3' tox.ini
# Prepend pwd to have access to the test module.
PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}:$PWD %tox
This works, but the modules in the main directory are only
shadowed. Thus, they might interfere with the tests if, e.g., the
installation skips modules. This can be fixed by
- moving to a new directory, copying only the test module folder over, or
- removing all modules in the current working dir (except the tests).
I arbitrarily chose removing for the following solution.
TL;DR
With all previously said, I’d arrive at using the following lines for running my Python tests:
%check
# Make sure we use the correct python and and sys.path[0] is the cwd (so the
# test module *is* found).
sed -i 's|python|%{python3}' tox.ini
# Remove all Python modules from cwd, except the test module, to avoid their use.
find . -maxdepth 2 -name '__init__.py' -exec dirname {} \; | grep -v ^\./tests | xargs rm -rf
# Handles populating PYTHONPATH with system paths.
%tox
Are my raised concerns valid? Are there already solutions I’m missing? Should there be documentation/rpm macros to help with this?
Thanks for your input!
Kai
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
