Running the Python test suite leaks .pem files in /tmp

[vstinner]

Running the Python test suite has two issues:

• A test leaks 3 .pem files in /tmp
• The test suite is not marked as failed when it leaks temporary files (in /tmp)

Moreover, test_tools enters an unlimited loop and fills the $TMPDIR directory if the $TMPDIR is a sub-directory of the Python source code directory. Example:

• /home/vstinner/python/main/ : Python source code
• /home/vstinner/python/main/TMP/ : Temporary directory ($TMPDIR)

Running test_freeze_simple_script() of test_tools copies TMP/ into TMP/TMP/ and then into TMP/TMP/TMP/, etc. Quickly, it fills TMP/ with a "loop" of files :-)

Linked PRs

• gh-93353: regrtest checks for leaked temporary files #93776

[asottile]

related: pypa/pip#10753

[serhiy-storchaka]

Add parameter _remove=os.remove in def _tempfile() and use it instead of os.remove. The leak will gone.

I suppose that the finally code is executed after the module os has been cleared, so os.remove raises an exception. You do not get a report because the standard streams have already been cleared too. Perhaps the logging module creates reference loops and keeps reference to this module dict.

It is the answer of the puzzle. In general, you should not rely on the code executed at the interpreter shutdown stage (except registered with atexit). It may be much worse in PyPy.

[vstinner]

I created the PR #93377 to fix importlib.resources.

[vstinner]

related: pypa/pip#10753

Oh, it's known in pip. I didn't want to report the issue to pip until I found the root issue. Well, pip, requests and certifi are fine. The root issue is a Python bug (in importlib.resources).

[vstinner]

Add parameter _remove=os.remove in def _tempfile() and use it instead of os.remove. The leak will gone.

Right, see my PR #93377

I suppose that the finally code is executed after the module os has been cleared, so os.remove raises an exception.

When the _tempfile() context manager finalizer is called, os.remove is None.

You do not get a report because the standard streams have already been cleared too.

Right.

Perhaps the logging module creates reference loops and keeps reference to this module dict.

pip init_logging() function calls logging.setLoggerClass(VerboseLogger). Since logging calls os.register_at_fork(), the module namespace is kept alive until the last GC collection. Beause of logging.setLoggerClass(VerboseLogger), it seems like a large part of pip is also kept alive indirectly!

A common reference leak is... defining a function at the module level: the function refers to the module namespace, the module namespace refers to the function. Enjoy :-) The GC can break such cycle, it's designed for that.

But yeah, a combo of issues keeps lot of objects alive way longer "than expected".

In general, you should not rely on the code executed at the interpreter shutdown stage (except registered with atexit). It may be much worse in PyPy.

Well, that's a design choice of certifi. If you are concerned, report the issue to them ;-)

importlib.resources is not really faulty here, the API is designed to be used like:

with as_file() as filename: ...

But certifi calls __enter__() method and never calls the __exit__() method explicitly, but rely on Python GC to call the finally: block of _tempfile() context manager.

[serhiy-storchaka]

Should not ResourceWarning be emitted when collect not closed generators? But I afraid it can add a lot of noise in existing "working" code.

[tiran]

PR GH-93776 broke testing on WASI:

_PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=/build_oot/host/build/lib.wasi-wasm32-3.12:/Lib --mapdir /::../.. --' _PYTHON_PROJECT_BASE=/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host/build/lib.wasi-wasm32-3.12:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_d_wasi_wasm32-wasi ../build/python  ../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml 
== CPython 3.12.0a0 (heads/main-dirty:a338e106b6, Jun 14 2022, 13:20:18) [GCC 9.4.0]
== Linux-5.17.13-300.fc36.x86_64-x86_64-with-glibc2.31 little-endian
== cwd: /buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host/build/test_python_111802æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 6754872
0:00:00 load avg: 5.20 Run tests in parallel using 2 child processes (timeout: 15 min, worker timeout: 20 min)
Warning -- regrtest worker thread failed: Traceback (most recent call last):
Warning --   File "/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/Lib/test/libregrtest/runtest_mp.py", line 325, in run
Warning --     mp_result = self._runtest(test_name)
Warning --                 ^^^^^^^^^^^^^^^^^^^^^^^^
Warning --   File "/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/Lib/test/libregrtest/runtest_mp.py", line 280, in _runtest
Warning --     os.mkdir(tmp_dir)
Warning --     ^^^^^^^^^^^^^^^^^
Warning -- FileExistsError: [Errno 17] File exists: '/buildmaster-config/master/workers/local-worker/3.x.local-worker.wasi/build/build_oot/host/build/test_python_111802æ_tmpdir'
Kill <TestWorkerProcess #1 running test=test_typing pid=111806 time=2 ms> process group

[tiran]

The commit e566ce5 relies on an environment variable to create unique temp directories per instance. The approach does not work on WASI for two reasons:

1. WASI runtimes do not pass environment variables to WASM programs. python.wasm process does not see the TMPDIR variable.
2. wasmtime uses an explicit, sandboxed mapping of host directories to runtime directories (sort of a chroot, but better). The TMPDIR outside would not be the same as the TMPDIR inside.

[vstinner]

WASI runtimes do not pass environment variables to WASM programs. python.wasm process does not see the TMPDIR variable.

Ah, so this feature is not supported on WASI. Can you please test #93810 on WASI?

[vstinner]

The initial issue (leaked PEM files) has been fixed.

regrtest now detects leaked temporary files in the main branch. I don't want to backport this change since it causes some CI issues, and I'm still fixing regrtest. I have a few more changes to enhance regrtest which will use this issue number, but IMO the issue can be closed since the initial issue has been fixed.

[vstinner]

Follow-up: issue #93919 "On Windows, test_distutils leaks a temporary file: Microsoft".