On Wed, Apr 14, 2021 at 5:18 AM Miro Hrončok <mhroncok@xxxxxxxxxx> wrote:
>
> Hello Pythonistas.
>
> I'd like to be able to automatically handle Python "namespace" packages from our
> packaging macros.
>
> The problem:
>
> Several Python packages share a "namespace", let's take an artificial example
> with food.spam and food.eggs Python packages.
>
> 1. the Python packages both have site-packages/food
> 2. sometimes such packages also both have site-packages/food/__init__.py
> (usually empty or mostly empty, but with different mtimes etc.)
>
> On RPM level, this means:
>
> 1. %{python3_sitelib}/food can be co-owned
> OR it can be in an artificial python3-food(-filesystem) package [0]
> OR it can be in an existing package that is always present [1]
> 2. %{python3_sitelib}/food/__init__.py and
> %{python3_sitelib}/food/__pycache__/__init__.*.pyc
> will conflict if present in multiple packages,
> they need to be removed or shared from the python3-food(-filesystem) package
>
> I want to solve this once for all, define the best practice, document it in the
> packaging guidelines and possible automate this in %pyproject_save_files [2].
>
>
> My current idea is:
>
> - sharing directories is safe and easy,
> let's do that instead of artificial packages (those are hard to automate)
> - namespace packages should not need __init__py with modern Python 3,
> let's discourage that
This, unfortunately, needs to be done upstream, at least some of the
time. There are three different ways to do namespace packages in
Python. The modern Python 3 version does not require __init__.py
files but the other two (from before Python 3.3 added namespace
packages to the core interpreter. One is implemented via pkgutil from
the python stdlib and the other is implemented via a setuptools
feature) have logic in the __init__.py to turn the directory into a
namespace. The Python 3.3+ and pkgutil methods of namespace packaging
are largely compatible (Enough so I think your idea to convert
pkgutil-based packages to Python-3.3+ versions will work) but the
setuptools version is incompatible.[1]_
The problem with us trying to change the setuptools using python
modules that we package to use the modern Python 3 occurs when a user
installs a different package in the namespace from upstream. The user
then has two packages which implement the namespace in incompatible
ways. My testing shows that this will result in all of our packages
failing to be found by python.[2]_
You could modify your proposal to deal with setuptools based
namespaces in a different manner than the other two namespaces. This
might cause more mistakes (as packagers will have to figure out if
they're in the special case scenario of a setuptools based namespace)
but it does simplify packaging in the other two cases.
.. [1]_: https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package
.. [2]_: Here's the procedure to test compatibility:
mkdir -p site-3.3/food/spam site-pkgutil/food/eggs site-setuptools/food/potato
echo "print('spam')" > site-3.3/food/spam/__init__.py
echo "__path__ = __import__('pkgutil').extend_path(__path__,
__name__)" > site-pkgutil/food/__init__.py
echo "print('eggs')" > site-pkgutil/food/eggs/__init__.py
echo "__import__('pkg_resources').declare_namespace(__name__)" >
site-setuptools/food/__init__.py
echo "print('potato')" > site-pkgutil/food/eggs/__init__.py
# These are both compatible
PYTHONPATH=site-3.3:site-pkgutil python3 -c 'import food.spam, food.eggs'
PYTHONPATH=site-pkgutil:site-3.3 python3 -c 'import food.spam, food.eggs'
# The setuptools namespace makes it so Python does not register the
3.3-style namespace at all
PYTHONPATH=site-3.3:site-setuptools python3 -c 'import food.spam, food.potato'
PYTHONPATH=site-setuptools:site-3.3 python3 -c 'import food.spam, food.potato'
# The setuptools namespace causes the pkgutil namespace to silently fail
PYTHONPATH=site-setuptools:site-pkgutil python3 -c 'import food.eggs,
food.potato'
PYTHONPATH=site-setuptools:site-pkgutil python3 -c 'import food.eggs,
food.potato'
-Toshio
_______________________________________________
packaging mailing list -- packaging@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to packaging-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/packaging@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
