Overlapping PYTHONPATH may cause import already imported module

[aklajnert]

BPO	46806
Nosy	@ericsnowcurrently, @aklajnert

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-20.12:34:47.401>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10', '3.7']
title = 'Overlapping PYTHONPATH may cause import already imported module'
updated_at = <Date 2022-02-27.10:45:32.995>
user = 'https://github.com/aklajnert'

bugs.python.org fields:

activity = <Date 2022-02-27.10:45:32.995>
actor = 'aklajnert'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2022-02-20.12:34:47.401>
creator = 'aklajnert'
dependencies = []
files = []
hgrepos = []
issue_num = 46806
keywords = []
message_count = 7.0
messages = ['413584', '413713', '413714', '413716', '413772', '413813', '414146']
nosy_count = 3.0
nosy_names = ['docs@python', 'eric.snow', 'aklajnert']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46806'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[aklajnert]

I'm not 100% sure whether it is a bug or intentional behavior but looks like a bug to me. I couldn't find anything about it here or anywhere else.

Sample project structure:

.
├── main.py
└── src
    ├── __init__.py
    ├── common_object.py
    ├── user_1.py
    ├── user_2.py
    └── user_3.py

__init__.py is an empty file.

# src/common_object.py
OBJECT = object()

# src/user_1.py
from .common_object import OBJECT

# src/user_2.py
from src.common_object import OBJECT

# src/user_3.py
from common_object import OBJECT

# main.py
import sys

sys.path.append("src")

from src import user_1, user_2, user_3


if __name__ == '__main__':
    print(user_1.OBJECT is user_2.OBJECT) # True
    print(user_1.OBJECT is user_3.OBJECT) # False

Since src package is added to PYTHONPATH, it is possible to import common_object by calling from src.common_object or from common_object.
Both methods work, but using import without src. makes Python load the same module again instead of using the already loaded one.

If you extend main.py with the following code, you'll see a bit more:

modules = [
    module
    for name, module in sys.modules.items()
    if "common_object" in name
]
print(len(modules)) # 2
print(modules[0].__file__ == modules[1].__file__) # True

In the sys.modules dict there will be two separate modules - one called common_object and another named src.common_object.
If you compare the __file__ value for both modules you'll see that they are the same. It seems that python gets the module name wrong.

[ericsnowcurrently]

When you run a Python script, the directory the script is in is automatically added to the beginning of sys.path. This is the fundamental issue you've run into.

Basically, "src.common_object" is imported relative to the "src" that was imported relative to that automatically added sys.path entry. However, "common_object" is a distinct module imported relative to the sys.path entry you explicitly added.

In general, adding a package's directory to sys.path is a bad idea.

[ericsnowcurrently]

Here is more detail on what happens when "from src import user_1, user_2, user_3" is executed in main.py:

1. the "src" module is imported
   a. not found in sys.modules
   b. file found on a sys.path entry (the directory main.py is in)
   c. the "src" module is created with __path__ set to the src directory
   d. the module is added to sys.modules
   e. src/init.py is executed
2. the "src.user_1" module is imported
   a. not found in sys.modules
   b. file found relative to src.__path__
   c. module created
   d. added to sys.modules
   e. executed
3. "from .common_object import OBJECT" is resolved to "from src.common_object import OBJECT"
4. "src.common_object" is imported (see 2)
5. src.common_object.OBJECT is created
6. "src.user_2" is imported (see 2)
7. "src.common_object" is already found in sys.modules and used
8. "src.user_3" is imported (see 2)
9. "common_object" is imported
   a. not found in sys.modules
   b. file found on a sys.path entry (the one you added in main.py)
   c. module created
   d. added to sys.modules
   e. executed
10. common_object.OBJECT is created

So the module created at (4) is different than the one at (9), even though they are imported from the same file. Consequently, the OBJECT in each is likewise distinct.