ResourceWarning on env's setup

[stanislavlevin]

Issue
There are ResourceWarnings on virtual env creation.

Environment

• pip list of the host python where virtualenv is installed:

(integration) [builder@localhost pyproject_installer]$ pip list
Package        Version
-------------- -------
attrs          22.2.0
distlib        0.3.6
exceptiongroup 1.1.0
filelock       3.8.2
iniconfig      1.1.1
packaging      22.0
pip            22.3.1
platformdirs   2.6.0
pluggy         1.0.0
pytest         7.2.0
setuptools     65.6.3
tomli          2.0.1
virtualenv     20.17.1
wheel          0.38.4

Output of the virtual environment creation

(integration) [builder@localhost pyproject_installer]$ python -W error::ResourceWarning -m virtualenv --no-setuptools --no-wheel -vvv --with-traceback test
280 setup logging to NOTSET [DEBUG report:35]
280 created app data folder /usr/src/.local/share/virtualenv [DEBUG __init__:40]
284 find interpreter for spec PythonSpec(path=/usr/src/RPM/BUILD/pyproject_installer/.tox/integration/bin/python) [INFO builtin:56]
284 discover exe for PythonInfo(spec=CPython3.10.8.final.0-64, exe=/usr/src/RPM/BUILD/pyproject_installer/.tox/integration/bin/python, platform=linux, version='3.10.8 (main, Dec 17 2022, 11:34:59) [GCC 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt2)]', encoding_fs_io=utf-8-utf-8) in /usr [DEBUG py_info:437]
284 filesystem is case-sensitive [DEBUG info:24]
285 Attempting to acquire lock 140183178084464 on /usr/src/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:172]
285 Lock 140183178084464 acquired on /usr/src/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:176]
286 get interpreter info via cmd: /usr/bin/python3.10 /usr/src/RPM/BUILD/pyproject_installer/.tox/integration/lib/python3/site-packages/virtualenv/discovery/py_info.py XjXJ3A7syvkfleqZinswfzCYUKpcUjxL deGvXS8ZsucZgjoK64iWvsew6As8FDQp [DEBUG cached_py_info:108]
390 wrote python info of /usr/bin/python3.10 at /usr/src/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.json [DEBUG via_disk_folder:155]
390 Attempting to release lock 140183178084464 on /usr/src/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:209]
391 Lock 140183178084464 released on /usr/src/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:212]
391 proposed PythonInfo(spec=CPython3.10.8.final.0-64, system=/usr/bin/python3.10, exe=/usr/src/RPM/BUILD/pyproject_installer/.tox/integration/bin/python, platform=linux, version='3.10.8 (main, Dec 17 2022, 11:34:59) [GCC 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt2)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:63]
391 accepted PythonInfo(spec=CPython3.10.8.final.0-64, system=/usr/bin/python3.10, exe=/usr/src/RPM/BUILD/pyproject_installer/.tox/integration/bin/python, platform=linux, version='3.10.8 (main, Dec 17 2022, 11:34:59) [GCC 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt2)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:65]
428 create virtual environment via CPython3Posix(dest=/usr/src/RPM/BUILD/pyproject_installer/test, clear=False, no_vcs_ignore=False, global=False) [INFO session:48]
429 write /usr/src/RPM/BUILD/pyproject_installer/test/pyvenv.cfg [DEBUG pyenv_cfg:30]
429     home = /usr/bin [DEBUG pyenv_cfg:34]
429     implementation = CPython [DEBUG pyenv_cfg:34]
429     version_info = 3.10.8.final.0 [DEBUG pyenv_cfg:34]
429     virtualenv = 20.17.1 [DEBUG pyenv_cfg:34]
429     include-system-site-packages = false [DEBUG pyenv_cfg:34]
429     base-prefix = /usr [DEBUG pyenv_cfg:34]
429     base-exec-prefix = /usr [DEBUG pyenv_cfg:34]
429     base-executable = /usr/bin/python3.10 [DEBUG pyenv_cfg:34]
429 remove file /usr/src/RPM/BUILD/pyproject_installer/test/bin/python [DEBUG _sync:22]
429 symlink /usr/bin/python3.10 to /usr/src/RPM/BUILD/pyproject_installer/test/bin/python [DEBUG _sync:28]
430 create virtualenv import hook file /usr/src/RPM/BUILD/pyproject_installer/test/lib/python3/site-packages/_virtualenv.pth [DEBUG api:89]
430 create /usr/src/RPM/BUILD/pyproject_installer/test/lib/python3/site-packages/_virtualenv.py [DEBUG api:92]
430 ============================== target debug ============================== [DEBUG session:50]
430 debug via /usr/src/RPM/BUILD/pyproject_installer/test/bin/python /usr/src/RPM/BUILD/pyproject_installer/.tox/integration/lib/python3/site-packages/virtualenv/create/debug.py [DEBUG creator:197]
430 {
  "sys": {
    "executable": "/usr/src/RPM/BUILD/pyproject_installer/test/bin/python",
    "_base_executable": "/usr/src/RPM/BUILD/pyproject_installer/test/bin/python",
    "prefix": "/usr/src/RPM/BUILD/pyproject_installer/test",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/usr/src/RPM/BUILD/pyproject_installer/test",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib64/python310.zip",
      "/usr/lib64/python3.10",
      "/usr/lib64/python3.10/lib-dynload",
      "/usr/src/RPM/BUILD/pyproject_installer/test/lib64/python3/site-packages",
      "/usr/src/RPM/BUILD/pyproject_installer/test/lib/python3/site-packages"
    ],
    "meta_path": [
      "<class '_distutils_hack.DistutilsMetaFinder'>",
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.8 (main, Dec 17 2022, 11:34:59) [GCC 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt2)]",
  "makefile_filename": "/usr/lib64/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib64/python3.10/os.py'>",
  "site": "<module 'site' from '/usr/lib64/python3.10/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib64/python3.10/datetime.py'>",
  "math": "<module 'math' from '/usr/lib64/python3.10/lib-dynload/math.cpython-310.so'>",
  "json": "<module 'json' from '/usr/lib64/python3.10/json/__init__.py'>"
} [DEBUG session:51]
512 add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/usr/src/.local/share/virtualenv) [INFO session:55]
515 wrote embed update of distribution pip at /usr/src/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [DEBUG via_disk_folder:155]
517 triggered periodic upgrade of pip==22.3.1 (for python 3.10) via background process having PID 3708911 [INFO periodic_update:218]
Exception ignored in: <function Popen.__del__ at 0x7f7ef1a500d0>
Traceback (most recent call last):
  File "/usr/lib64/python3.10/subprocess.py", line 1072, in __del__
    _warn("subprocess %s is still running" % self.pid,
ResourceWarning: subprocess 3708911 is still running
Exception ignored in: <_io.FileIO name=3 mode='rb' closefd=True>
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/pyproject_installer/.tox/integration/lib/python3/site-packages/virtualenv/seed/wheels/periodic_update.py", line 70, in handle_auto_update
    trigger_update(distribution, for_py_version, wheel, search_dirs, app_data, periodic=True, env=env)
ResourceWarning: unclosed file <_io.BufferedReader name=3>
Exception ignored in: <_io.FileIO name=5 mode='rb' closefd=True>
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/pyproject_installer/.tox/integration/lib/python3/site-packages/virtualenv/seed/wheels/periodic_update.py", line 70, in handle_auto_update
    trigger_update(distribution, for_py_version, wheel, search_dirs, app_data, periodic=True, env=env)
ResourceWarning: unclosed file <_io.BufferedReader name=5>
...

[gaborbernat]

PR welcome.

[gaborbernat]

Seems no longer an issue as no one reported it in 1.5 years.

[sigma67]

I just reproduced this issue with OP's command.

Python 3.10.6 virtualenv 20.24.2 Ubuntu 22.04

 » python -W error::ResourceWarning -m virtualenv --no-setuptools --no-wheel -vvv --with-traceback test                                                    ~/PycharmProjects/myproject  
40 setup logging to NOTSET [DEBUG report:37]
46 find interpreter for spec PythonSpec(path=/home/myusername/PycharmProjects/myproject/.venv/bin/python) [INFO builtin:58]
46 discover exe for PythonInfo(spec=CPython3.10.6.final.0-64, exe=/home/myusername/PycharmProjects/myproject/.venv/bin/python, platform=linux, version='3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) in /usr [DEBUG py_info:441]
46 filesystem is case-sensitive [DEBUG info:26]
58 Attempting to acquire lock 140519554152144 on /home/myusername/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:212]
59 Lock 140519554152144 acquired on /home/myusername/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:215]
59 got python info of %s from (PosixPath('/usr/bin/python3.10'), PosixPath('/home/myusername/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.json')) [DEBUG via_disk_folder:131]
59 Attempting to release lock 140519554152144 on /home/myusername/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:244]
60 Lock 140519554152144 released on /home/myusername/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.lock [DEBUG _api:247]
60 proposed PythonInfo(spec=CPython3.10.6.final.0-64, system=/usr/bin/python3.10, exe=/home/myusername/PycharmProjects/myproject/.venv/bin/python, platform=linux, version='3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) [INFO builtin:65]
60 accepted PythonInfo(spec=CPython3.10.6.final.0-64, system=/usr/bin/python3.10, exe=/home/myusername/PycharmProjects/myproject/.venv/bin/python, platform=linux, version='3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:67]
76 create virtual environment via CPython3Posix(dest=/home/myusername/PycharmProjects/myproject/test, clear=False, no_vcs_ignore=False, global=False) [INFO session:50]
76 create folder /home/myusername/PycharmProjects/myproject/test/bin [DEBUG _sync:12]
76 create folder /home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages [DEBUG _sync:12]
77 write /home/myusername/PycharmProjects/myproject/test/pyvenv.cfg [DEBUG pyenv_cfg:32]
77      home = /usr/bin [DEBUG pyenv_cfg:36]
77      implementation = CPython [DEBUG pyenv_cfg:36]
77      version_info = 3.10.6.final.0 [DEBUG pyenv_cfg:36]
77      virtualenv = 20.24.2 [DEBUG pyenv_cfg:36]
77      include-system-site-packages = false [DEBUG pyenv_cfg:36]
77      base-prefix = /usr [DEBUG pyenv_cfg:36]
77      base-exec-prefix = /usr [DEBUG pyenv_cfg:36]
77      base-executable = /usr/bin/python3.10 [DEBUG pyenv_cfg:36]
77 symlink /usr/bin/python3.10 to /home/myusername/PycharmProjects/myproject/test/bin/python [DEBUG _sync:32]
77 create virtualenv import hook file /home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages/_virtualenv.pth [DEBUG api:91]
77 create /home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages/_virtualenv.py [DEBUG api:94]
77 ============================== target debug ============================== [DEBUG session:52]
77 debug via /home/myusername/PycharmProjects/myproject/test/bin/python /home/myusername/PycharmProjects/myproject/.venv/lib/python3.10/site-packages/virtualenv/create/debug.py [DEBUG creator:200]
77 {
  "sys": {
    "executable": "/home/myusername/PycharmProjects/myproject/test/bin/python",
    "_base_executable": "/home/myusername/PycharmProjects/myproject/test/bin/python",
    "prefix": "/home/myusername/PycharmProjects/myproject/test",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/myusername/PycharmProjects/myproject/test",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python310.zip",
      "/usr/lib/python3.10",
      "/usr/lib/python3.10/lib-dynload",
      "/home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]",
  "makefile_filename": "/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.10/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.10/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.10/datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/usr/lib/python3.10/json/__init__.py'>"
} [DEBUG session:53]
93 add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/home/myusername/.local/share/virtualenv) [INFO session:57]
96 got embed update of distribution %s from ('pip', PosixPath('/home/myusername/.local/share/virtualenv/wheel/3.10/embed/3/pip.json')) [DEBUG via_disk_folder:131]
98 wrote embed update of distribution %s at ('pip', PosixPath('/home/myusername/.local/share/virtualenv/wheel/3.10/embed/3/pip.json')) [DEBUG via_disk_folder:151]
98 triggered periodic upgrade of pip==23.2.1 (for python 3.10) via background process having PID 6871 [INFO periodic_update:224]
Exception ignored in: <function Popen.__del__ at 0x7fcd42de5b40>
Traceback (most recent call last):
  File "/usr/lib/python3.10/subprocess.py", line 1070, in __del__
    _warn("subprocess %s is still running" % self.pid,
ResourceWarning: subprocess 6871 is still running
Exception ignored in: <_io.FileIO name=3 mode='rb' closefd=True>
Traceback (most recent call last):
  File "/home/myusername/PycharmProjects/myproject/.venv/lib/python3.10/site-packages/virtualenv/seed/wheels/periodic_update.py", line 78, in handle_auto_update
    trigger_update(distribution, for_py_version, wheel, search_dirs, app_data, periodic=True, env=env)
ResourceWarning: unclosed file <_io.BufferedReader name=3>
Exception ignored in: <_io.FileIO name=5 mode='rb' closefd=True>
Traceback (most recent call last):
  File "/home/myusername/PycharmProjects/myproject/.venv/lib/python3.10/site-packages/virtualenv/seed/wheels/periodic_update.py", line 78, in handle_auto_update
    trigger_update(distribution, for_py_version, wheel, search_dirs, app_data, periodic=True, env=env)
ResourceWarning: unclosed file <_io.BufferedReader name=5>
98 got embed update of distribution %s from ('pip', PosixPath('/home/myusername/.local/share/virtualenv/wheel/3.10/embed/3/pip.json')) [DEBUG via_disk_folder:131]
99 using periodically updated wheel /home/myusername/.local/share/virtualenv/wheel/house/pip-23.1.2-py3-none-any.whl [DEBUG periodic_update:49]
99 install pip from wheel /home/myusername/.local/share/virtualenv/wheel/house/pip-23.1.2-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
100 Attempting to acquire lock 140519551946368 on /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any.lock [DEBUG _api:212]
101 Lock 140519551946368 acquired on /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any.lock [DEBUG _api:215]
101 Attempting to release lock 140519551946368 on /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any.lock [DEBUG _api:244]
101 Lock 140519551946368 released on /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any.lock [DEBUG _api:247]
101 copy directory /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any/pip to /home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages/pip [DEBUG _sync:40]
258 copy directory /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any/pip-23.1.2.dist-info to /home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages/pip-23.1.2.dist-info [DEBUG _sync:40]
261 copy /home/myusername/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.1.2-py3-none-any/pip-23.1.2.virtualenv to /home/myusername/PycharmProjects/myproject/test/lib/python3.10/site-packages/pip-23.1.2.virtualenv [DEBUG _sync:40]
262 generated console scripts pip pip3.10 pip3 pip-3.10 [DEBUG base:43]
262 add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [INFO session:63]
263 write /home/myusername/PycharmProjects/myproject/test/pyvenv.cfg [DEBUG pyenv_cfg:32]
263     home = /usr/bin [DEBUG pyenv_cfg:36]
263     implementation = CPython [DEBUG pyenv_cfg:36]
263     version_info = 3.10.6.final.0 [DEBUG pyenv_cfg:36]
263     virtualenv = 20.24.2 [DEBUG pyenv_cfg:36]
263     include-system-site-packages = false [DEBUG pyenv_cfg:36]
263     base-prefix = /usr [DEBUG pyenv_cfg:36]
263     base-exec-prefix = /usr [DEBUG pyenv_cfg:36]
263     base-executable = /usr/bin/python3.10 [DEBUG pyenv_cfg:36]
263 created virtual environment CPython3.10.6.final.0-64 in 224ms
  creator CPython3Posix(dest=/home/myusername/PycharmProjects/myproject/test, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/home/myusername/.local/share/virtualenv)
    added seed packages: pip==23.1.2
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]

[sigma67]

Thanks for reopening. This is caused directly by the periodic update feature, as the process that is being opened is never stopped (?)

It is not a problem if setting --no-periodic-update or when passing the env variable _VIRTUALENV_PERIODIC_UPDATE_INLINE=1 to cli_run with env.

[sigma67]

From my (user's) perspective this feature should be opt-in, not opt-out. But that's probably difficult to change at this point.

What would be the best way to go about fixing this? This is the relevant code.

virtualenv/src/virtualenv/seed/wheels/periodic_update.py

Lines 202 to 232 in 119c63e

	def trigger_update(distribution, for_py_version, wheel, search_dirs, app_data, env, periodic): # noqa: PLR0913
	wheel_path = None if wheel is None else str(wheel.path)
	cmd = [
	sys.executable,
	"-c",
	dedent(
	"""
	from virtualenv.report import setup_report, MAX_LEVEL
	from virtualenv.seed.wheels.periodic_update import do_update
	setup_report(MAX_LEVEL, show_pid=True)
	do_update({!r}, {!r}, {!r}, {!r}, {!r}, {!r})
	""",
	)
	.strip()
	.format(distribution, for_py_version, wheel_path, str(app_data), [str(p) for p in search_dirs], periodic),
	]
	debug = env.get("_VIRTUALENV_PERIODIC_UPDATE_INLINE") == "1"
	pipe = None if debug else PIPE
	kwargs = {"stdout": pipe, "stderr": pipe}
	if not debug and sys.platform == "win32":
	kwargs["creationflags"] = CREATE_NO_WINDOW
	process = Popen(cmd, **kwargs) # noqa: S603
	logging.info(
	"triggered periodic upgrade of %s%s (for python %s) via background process having PID %d",
	distribution,
	"" if wheel is None else f"=={wheel.version}",
	for_py_version,
	process.pid,
	)
	if debug:
	process.communicate() # on purpose not called to make it a background process

[gaborbernat]

From my (user's) perspective this feature should be opt-in, not opt-out.

https://discuss.python.org/t/how-should-virtualenv-behave-respective-of-seed-packages-e-g-pip-by-default/4146/18 this was a widely contested topic, with no clear winner, so this was the compromise.

What would be the best way to go about fixing this? This is the relevant code.

Up to whoever creates the PR 😊 as long as does the same outcome and fixes the warning I'm happy.

[sigma67]

I believe this should be as simple as calling p.stdout.close() and p.stderr.close(). Context: https://stackoverflow.com/a/58696973/5726546

The fact that setting _VIRTUALENV_PERIODIC_UPDATE_INLINE=1 doesn't produce the issue seems to confirm this:

"Note that most subprocess code examples don't do this, because they tend to use Popen.communicate(), which closes the file handles for you."