Namespace package plugins

[rhattersley]

Environment: Ubuntu 19.10; Python 3.7.5; pip 18.1

References:

• https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages
• https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages

I'm attempting to provide a plugin system that only needs to work for Python 3.3+. Namespaces seem like a good fit as they easily allow different types of plugin to be separated out, e.g. myapp.io_plugins and myapp.viz_plugins. Given Python 3.3+ then native namespaces seem the way to go, but ...

I am unable to recreate the suggested myapp.plugins structure from PyPA using pip.

The problems are:

1. When installing myapp I can't get pip to create an empty myapp.plugins namespace package. Probably not a deal breaker (e.g. I can catch the ImportError), but somewhat inconvenient at best.
2. When installing a plugin, pip states "Successfully installed...", but then goes on to say "Target directory .../myapp already exists..." and fails to install the plugin.

NB. The code in the native namespace example project didn't help. It is equivalent to making myapp a namespace package, instead of myapp.plugins and changing it to fit the myapp.plugins guide broke the behaviour.

I've tried a fair few permutations but the one that seemed closest to the PyPA guide is:

myapp/
├── myapp
│   ├── __init__.py
│   ├── foo.py
│   └── plugins [empty directory]
└── setup.py

Where setup.py contains either packages=['myapp'] or packages=['myapp', 'myapp.plugins']. Either way, the myapp.plugins directory doesn't get created.

python -m pip install -t /path/to/target . results in (ignoring byte code and dist-info):

myapp/
├── __init__.py
└── foo.py

The plugin is defined as:

myplugin/
├── myapp
│   └── plugins
│       └── myplugin
│           └── __init__.py
└── setup.py

Where setup.py contains packages=['myapp.plugins.myplugin'].

python -m pip install -t /path/to/target . results in (ignoring byte code and dist-info):

Processing /path/to/myplugin
Building wheels for collected packages: MyPlugin
  Running setup.py bdist_wheel for MyPlugin ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-...
Successfully built MyPlugin
Installing collected packages: MyPlugin
Successfully installed MyPlugin-1
Target directory /path/to/target/myapp already exists. Specify --upgrade to force replacement.

Despite the Successfully ... the only change to the installed files is the addition of the MyPlugin dist-info files - the myapp/plugins/myplugin/__init__.py file is not installed.

[ncoghlan]

Does it work when installing with the latest pip, and/or building with the latest setuptools? Does using find_namespace_packages (as described https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages) change the behaviour?

The other thing to check would be whether it works correctly if the installation is to a virtual environment, rather than nominating a specific target directory with "-t".

[rhattersley]

Thanks for the suggestions @ncoghlan. 👍 Once you use the latest pip & setuptools it seems like the problems are restricted to using "-t".

1. Updating to pip 20.0.2 and setuptools 45.2.0 before installing myapp & myplugin with python -m pip install -t /path/to/target . doesn't help - it gives the same output & result. ❌
2. Creating a virtual environment with venv, and then installing both myapp & myplugin with python -m pip install . gives a working environment, but each install spits out an error error: invalid command 'bdist_wheel' before declaring a successful install. 😑
3. Creating a virtual environment with venv, upgrading pip & setuptools as above, and then installing both myapp & myplugin with python -m pip install . works. ✔️
4. Similarly, creating a separate conda environment with Python 3.7 and the latest versions of pip & setuptools, and then installing myapp & myplugin with python -m pip install . also works. ✔️

The use of explicit package names vs. find_namespace_packages didn't make any difference either way.

So thanks again for the suggestion to try venv - I'd got too obsessed with trying to figure out the right options in setup.py so it didn't occur to me that the -t behaviour might not be orthogonal. Does this mean there is a bug in the behaviour of -t?

[uranusjr]

Sounds like more issues to add to pypa/pip#4390.