test_interpreters: test_create_many_threaded() failed on FreeBSD: log: RuntimeError: interpreter creation failed

[vstinner]

test_create_many_threaded (test.test_interpreters.StressTests.test_create_many_threaded) ...

Warning -- Uncaught thread exception: RuntimeError
Exception in thread Thread-83 (task):
RuntimeError: error evaluating path

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/threading.py", line 996, in run
    self._target(*self._args, **self._kwargs)
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/test_interpreters.py", line 483, in task
    interp = interpreters.create()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/support/interpreters.py", line 25, in create
    id = _interpreters.create(isolated=isolated)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: interpreter creation failed

ok

build: https://buildbot.python.org/all/#/builders/1223/builds/187

Linked PRs

• gh-109700: Improve stress tests for interpreter creation #109946
• [3.13] gh-109700: Improve stress tests for interpreter creation (GH-109946) #133391

[vstinner]

Bug also seen on aarch64 RHEL8 LTO + PGO 3.x: https://buildbot.python.org/all/#/builders/78/builds/5402

[serhiy-storchaka]

Important part in output:

0:05:34 load avg: 1.66 [424/463/1] test_interpreters failed (env changed) -- running (1): test_threading (2 min 42 sec)
Exception ignored error evaluating path:
Traceback (most recent call last):
  File "<frozen getpath>", line 356, in <module>
ValueError: embedded null byte

[vstinner]

ValueError: embedded null byte

It comes from the FreeBSD build.

---

The aarch64 RHEL8 LTO + PGO 3.x build has a different error: TypeError: descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object.

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/python'
  isolated = 0
  environment = 0
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 1
  stdlib dir = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/Lib'
  sys._base_executable = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/python'
  sys.base_prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
  sys.base_exec_prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
  sys.platlibdir = 'lib'
  sys.executable = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/python'
  sys.prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
  sys.exec_prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
  sys.path = [
    '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target/lib/python313.zip',
    '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/Lib',
    '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/build/lib.linux-aarch64-3.13',
  ]
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1004, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1100, in get_code
  File "<frozen importlib._bootstrap_external>", line 1199, in get_data
TypeError: descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object

[vstinner]

I failed to reproduce the issue on Linux just with this command:

./python -m test test_interpreters -v --forever -j25 --fail-env-changed

[serhiy-storchaka]

'/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/build/lib.linux-aarch64-3.13',

This looks very suspicious. Why two /build/s in a row?

[vstinner]

This looks very suspicious. Why two /build/s in a row?

That's how aarch64 RHEL8 LTO + PGO 3.x is configured.

• Python source code: /home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build
• Main test process run in: /home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/build/test_python_worker_2806656æ

Yes, there are two build/build/ sub-directories, but I just think that it's a choice of the buildbot configuration, not a Python bug.

[serhiy-storchaka]

It may be related to #109615. What if run tests locally in such configuration?

[serhiy-storchaka]

This test consumes a lot of memory. 200 threads need more than 6 GB of memory if run them simultaneous (and if not, then what is the point of using so many threads?). Perhaps there is a leak, because if run tests repeatedly with limited memory, they finally crash.

$ (ulimit -v 7000000; ./python -m test -vuall test_interpreters -m test_create_many_threaded --forever)
...
0:00:11 load avg: 37.22 [  5] test_interpreters
test_create_many_threaded (test.test_interpreters.StressTests.test_create_many_threaded) ... MemoryErrorMemoryErrorMemoryErrorFatal Python error: Segmentation fault

Current thread 0x00007fc450032640 (most recent call first):
  <no Python frame>

200 live interpreters also consume sufficient amount of memory.

All normal tests only need 600-700 MB of memory. Tests which need more are marked with @bigmemtest decorator and perform a dry run or skipped by default. It seems that not all buildbots have so much memory.

@ericsnowcurrently What is the purpose of this test? Can it use less threads, for example 10 threads, each sequentially creating 20 interpreters? Can 5 threads be enough?

[serhiy-storchaka]

test_create_many_sequential also crashes due to leaks.

$ (ulimit -v 700000; ./python -m test -vuall test_interpreters -m test_create_many_sequential --forever --fail-env-changed)
...
0:00:02 load avg: 0.62 [  2] test_interpreters
test_create_many_sequential (test.test_interpreters.StressTests.test_create_many_sequential) ... Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/site.py", line 73, in <module>
  File "/home/serhiy/py/cpython/Lib/os.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1004, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1137, in get_code
  File "<frozen importlib._bootstrap_external>", line 766, in _compile_bytecode
Fatal Python error: Segmentation fault

Current thread 0x00007f60db746740 (most recent call first):
  <no Python frame>

If you double the limit, it will crash on the 7th iteration instead of the 2nd. If triple -- on 11th. So it leaks 1.5-2 MB per interpreter. These two tests ran sequentially can leak up to 6 GB of memory.

[vstinner]

Wow, so now it's possible to leak a whole interpreter?

Maybe we need something like threading_setup() / threading_cleanup() which uses _thread._count(), to count how many interpreters we have before/after running tests?

[serhiy-storchaka]

These two tests ran sequentially can leak up to 6 GB of memory.

Actually, only up to 600 MB, but it is much anyway. I do not know how it is now, but several years ago some of buildbots had only few hundreds of MBs of physical memory, often failed after hours of swapping due to timeout. I suppose buildbots on which test_create_many_threaded fails also have very limited RAM.

[serhiy-storchaka]

More precisely, 1.4 MB are leaked in every subinterpreter.