Parallel import of modules causes a ModuleNotFoundError

Bug report

Bug description:

Using the free-threading build parallel import of modules causes a ModuleNotFoundError. The module that cannot be found can differ. A minimal reproducer:

import sys
from threading import Thread, Barrier
from importlib import import_module, reload
import time


# Setup threads
t0 = time.perf_counter()
number_of_threads = 8
barrier = Barrier(number_of_threads)

pmods = []


def work(ii):
    # Method doing the actual import
    barrier.wait()
    while True:
        try:
            m = pmods.pop()
            mod = import_module(m)
            # print(f'  {ii}: {m} done')
        except IndexError:
            return


worker_threads = []
for ii in range(number_of_threads):
    worker_threads.append(Thread(target=work, args=[ii]))

dt = time.perf_counter() - t0
print(f"setup threads {dt*1e3:.2f}")


def parallel_import(modules: list[str]):
    global pmods
    pmods += modules
    for t in worker_threads:
        t.start()
    for t in worker_threads:
        t.join()


def seq_import(modules: list[str]):
    for m in modules:
        mod = import_module(m)


import_method = seq_import
import_method = parallel_import

mods = [
    "abc",
    "functools",
    "weakref",
    "linecache",
    "glob",
    "annotationlib",
    "argparse",
    "fnmatch",
    "itertools",
    "operator",
    "string",
    "re",
    "collections",
    "sqlite3",
    "pathlib",
    "urllib",
    "typing",
    "csv",
    "uuid",
]

mods += [
    "setuptools",
    "sympy",
    "django",
    "boto3",
]  # these are some packages that need to be installed. without these the reproduction is the issue is much harder

t0 = time.perf_counter()
import_method(mods)
dt = time.perf_counter() - t0
print(f"{import_method.__name__} {dt*1e3:.2f}")

Note: this script fails on my system about 1 in 4 times. Changing the modules imported (or the order, or the number of threads) can effect this.

Modules that have given issues: collections.abs, glob, sympy, django.utils.regex_helper. This suggests it is a general issue, and not a particular package.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

I'm seeing failures in the GIL-enabled build too (3.14.0a5 from uv). I made some slight modifications to the script to exit on error and set sys.setswitchinterval to a small value:

https://gist.github.com/colesbury/90493ef5b6f818bbbfde6922ffc8ea8a

setup threads 0.05
  4: sympy failed: SymPy now depends on mpmath as an external library. See https://docs.sympy.org/latest/install.html#mpmath for more information.
Traceback (most recent call last):
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/sympy/__init__.py", line 22, in <module>
    import mpmath
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/__init__.py", line 5, in <module>
    from .ctx_fp import FPContext
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/ctx_fp.py", line 1, in <module>
    from .ctx_base import StandardBaseContext
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/ctx_base.py", line 3, in <module>
    from .libmp.backend import xrange
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/libmp/__init__.py", line 40, in <module>
    from .libhyper import (NoConvergence, make_hyp_summator,
    ...<3 lines>...
      mpf_ellipk, mpc_ellipk, mpf_ellipe, mpc_ellipe)
ModuleNotFoundError: No module named 'mpmath.libmp.libhyper'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/raid/sgross/tmp/parallel_imports.py", line 25, in work
    mod = import_module(m)
  File "/private/home/sgross/.local/share/uv/python/cpython-3.14.0a5-linux-x86_64-gnu/lib/python3.14/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1386, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1359, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1330, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 762, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/sympy/__init__.py", line 24, in <module>
    raise ImportError("SymPy now depends on mpmath as an external library. "
    "See https://docs.sympy.org/latest/install.html#mpmath for more information.")
ImportError: SymPy now depends on mpmath as an external library. See https://docs.sympy.org/latest/install.html#mpmath for more information.

One race condition is in setuptools, which can add and removes entries in sys.meta_path. If that happens concurrently with the iteration of sys.meta_path, we can end up skipping entries and failing to import a module:

https://github.com/pypa/setuptools/blob/a3cc40f4b8fc90e8f57e3a59a54980fb1f408207/_distutils_hack/__init__.py#L226-L234

cpython/Lib/importlib/_bootstrap.py

Lines 1254 to 1262 in 05e89c3

    
           for finder in meta_path: 
        
               with _ImportLockContext(): 
        
                   try: 
        
                       find_spec = finder.find_spec 
        
                   except AttributeError: 
        
                       continue 
        
                   else: 
        
                       spec = find_spec(name, path, target) 
        
               if spec is not None:

We should probably make a copy of sys.meta_path before iterating over it

pythongh-130094: Fix race condition in `_find_spec`

@eendebakpt

@eendebakpt - can you try the linked PR (#130101) and verify that it fixes the issue? I haven't seen any import errors locally with it anymore.

gh-130094: Fix race conditions in `importlib` (gh-130101)

This should be fixed in main now.

[3.13] pythongh-130094: Fix race conditions in `importlib` (pythongh-…

[3.13] gh-130094: Fix race conditions in `importlib` (gh-130101) (gh-…

[CONTINT-3239] Revert "[automated] Upgrade embedded Python patch vers…

[Backport 7.84.x] [CONTINT-3239] Revert "[automated] Upgrade embedded…

[Backport 7.85.x] [CONTINT-3239] Revert "[automated] Upgrade embedded…

Navigation Menu

Repository navigation

Parallel import of modules causes a ModuleNotFoundError #130094

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	for finder in meta_path:
	with _ImportLockContext():
	try:
	find_spec = finder.find_spec
	except AttributeError:
	continue
	else:
	spec = find_spec(name, path, target)
	if spec is not None:

Parallel import of modules causes a ModuleNotFoundError #130094

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

colesbury commented on Feb 14, 2025

colesbury commented on Feb 14, 2025

colesbury commented on Feb 14, 2025

colesbury commented on Feb 18, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions