PathFinder does not find namespace packages children

[FFY00]

BPO	45591
Nosy	@brettcannon, @ncoghlan, @ericsnowcurrently, @FFY00

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-25.22:17:37.304>
created_at = <Date 2021-10-23.21:28:44.889>
labels = ['invalid']
title = 'PathFinder does not find namespace packages children'
updated_at = <Date 2021-10-27.22:30:08.484>
user = 'https://github.com/FFY00'

bugs.python.org fields:

activity = <Date 2021-10-27.22:30:08.484>
actor = 'FFY00'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-25.22:17:37.304>
closer = 'FFY00'
components = []
creation = <Date 2021-10-23.21:28:44.889>
creator = 'FFY00'
dependencies = []
files = []
hgrepos = []
issue_num = 45591
keywords = []
message_count = 7.0
messages = ['404912', '404994', '405006', '405050', '405051', '405122', '405137']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'ncoghlan', 'eric.snow', 'FFY00']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45591'
versions = []

[FFY00]

$ tree namespace
namespace/
└── a.py

0 directories, 1 file

$ ./python
Python 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.machinery import PathFinder
>>> PathFinder.find_
PathFinder.find_distributions(  PathFinder.find_module(         PathFinder.find_spec(
>>> PathFinder.find_spec('namespace')
ModuleSpec(name='namespace', loader=None, submodule_search_locations=_NamespacePath(['/home/anubis/git/cpython/namespace']))
>>> PathFinder.find_spec('namespace.a')

Currently, it is unable to find namespace.a, but it should:

Python 3.11.0a1+ (heads/main:9e05da6224, Oct 23 2021, 20:36:14) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.machinery import PathFinder
>>> spec = PathFinder.find_spec('namespace')
>>> spec.loader.load_module('namespace.a')
<frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
<module 'namespace.a' (<_frozen_importlib_external.NamespaceLoader object at 0x7f81b944f480>)>

I can make a PR, but just wanted to make sure the current behavior is not intended.

[brettcannon]

You didn't specify the path to search in to find a. https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder.find_spec says that PathFinder only has class methods, which means find_spec() won't know where 'namespace' is, so the search will fail to find anything, hence returning None.

[FFY00]

Ah, this was not obvious to me! I did not specify a path, but as it defaults to sys.path and namespace.a is available there, I was expecting it to find it.
One of the things that threw me off was the first arguments being called fullname, which I assumed implied that PathFinder was able to do a recursive search.

This indeed works:

$ ./python
Python 3.11.0a1+ (heads/main:9e05da6224, Oct 23 2021, 20:36:14) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.machinery import PathFinder
>>> spec = PathFinder.find_spec('namespace')
>>> PathFinder.find_spec('a', path=spec.submodule_search_locations)
ModuleSpec(name='a', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f245d0d3e80>, origin='/home/anubis/git/cpython/namespace/a.py')

I should have validated if normal packages also had the same behavior, which they do, before opening this -.-

I am curious, what is fullname supposed to mean then? "full" in what sense?

[brettcannon]

I am curious, what is fullname supposed to mean then? "full" in what sense?

You can still specify the full name of the module you're trying to import, but you also need to pass in the location information for that module. So the code does fullname.rpartition(".")[-1] to get the name of the module that you're looking for. So PathFinder.find_spec('namespace.a', path=spec.submodule_search_locations) should also work.

[FFY00]

Thank you for clarifying, that does work. This is surprising behavior to me, do you recall what was the reasoning for this design? Or is there some discussion you can point me to? And sorry for bothering, I know I am asking too many questions, but I'd like to understand the importlib design 😅

[brettcannon]

The full name argument approach comes from PEP-302.

But you want the full name as you are otherwise missing potentially key information for the finder. For instance, if you manipulate __path__, then it's just some random directory you're searching in. But searching for what? What if you want custom logic based on what package you're searching under?

Because the import system is designed to be flexible enough to let you import from a URL or SQLite database as well as DSL files, providing all of the available information becomes important.

[FFY00]

Okay, I think that makes sense to me. Thank you!!