bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

hroncok · 2021-11-03T11:37:14Z

Consider the following directory structure:

.
└── PATH1
    └── namespace
        └── sub1
            └── __init__.py

And both PATH1 and PATH2 in sys path:

$ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> import namespace.sub1
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace'])
>>> ...

While this interpreter still runs, PATH2/namespace/sub2 is created:

.
├── PATH1
│   └── namespace
│       └── sub1
│           └── __init__.py
└── PATH2
    └── namespace
        └── sub2
            └── __init__.py

The newly created module cannot be imported:

>>> ...
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'

Calling importlib.invalidate_caches() now newly allows to import it:

>>> import importlib
>>> importlib.invalidate_caches()
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace'])
>>> import namespace.sub2
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace'])

This was not previously possible.

https://bugs.python.org/issue45703

Automerge-Triggered-By: GH:encukou

Doc/library/importlib.rst

…ches() Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

vstinner · 2021-11-10T10:22:26Z

cc @jaraco

pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

miss-islington · 2021-12-07T14:45:05Z

Thanks @hroncok for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2021-12-07T14:45:07Z

Sorry @hroncok, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker ae1965ccb4b1fad63fab40fe8805d1b8247668d3 3.10

miss-islington · 2021-12-07T14:45:09Z

Thanks @hroncok for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. (cherry picked from commit ae1965c) Co-authored-by: Miro Hrončok <miro@hroncok.cz>

bedevere-bot · 2021-12-07T14:45:25Z

GH-29964 is a backport of this pull request to the 3.9 branch.

…date_ca… (pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. (cherry picked from commit ae1965c) Co-authored-by: Miro Hrončok <miro@hroncok.cz>

pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

bedevere-bot · 2022-01-26T15:43:18Z

GH-30922 is a backport of this pull request to the 3.10 branch.

…date_cache (GH-29384) (GH-30922) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

…ate_cache (pythonGH-29384) (pythonGH-30922) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

…ate_cache (GH-29384) (GH-30922) (GH-31076) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. Co-Authored-By: Miro Hrončok <miro@hroncok.cz> Automerge-Triggered-By: GH:encukou

…ate_cache (pythonGH-29384) (pythonGH-30922) (pythonGH-31076) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. Co-Authored-By: Miro Hrončok <miro@hroncok.cz> Automerge-Triggered-By: GH:encukou

the-knights-who-say-ni added the CLA signed label Nov 3, 2021

hroncok force-pushed the invalidate_namespacepath_cache branch 2 times, most recently from d5e7171 to 397a07e Compare November 3, 2021 12:50

AlexWaygood reviewed Nov 3, 2021

View reviewed changes

Doc/library/importlib.rst Outdated Show resolved Hide resolved

hroncok force-pushed the invalidate_namespacepath_cache branch from 397a07e to 09ff56e Compare November 3, 2021 13:55

hroncok force-pushed the invalidate_namespacepath_cache branch from 09ff56e to f79ced4 Compare November 3, 2021 13:55

This comment has been minimized.

Sign in to view

hroncok marked this pull request as ready for review November 3, 2021 14:25

encukou approved these changes Nov 23, 2021

View reviewed changes

bedevere-bot added the awaiting merge label Nov 23, 2021

encukou added the 🤖 automerge label Nov 23, 2021

miss-islington merged commit ae1965c into python:main Nov 23, 2021

bedevere-bot removed the awaiting merge label Nov 23, 2021

hroncok deleted the invalidate_namespacepath_cache branch November 23, 2021 16:01

encukou added needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Dec 7, 2021

miss-islington self-assigned this Dec 7, 2021

bedevere-bot removed the needs backport to 3.9 only security fixes label Dec 7, 2021

bedevere-bot removed the needs backport to 3.10 only security fixes label Jan 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

hroncok commented Nov 3, 2021 •

edited by miss-islington

Loading

This comment has been minimized.

vstinner commented Nov 10, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

bedevere-bot commented Dec 7, 2021

bedevere-bot commented Jan 26, 2022

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

Conversation

hroncok commented Nov 3, 2021 • edited by miss-islington Loading

This comment has been minimized.

vstinner commented Nov 10, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

bedevere-bot commented Dec 7, 2021

bedevere-bot commented Jan 26, 2022

hroncok commented Nov 3, 2021 •

edited by miss-islington

Loading