ValueError: I/O operation on closed file

[bnomis]

After all tests have finished, closing the test session, the faulthandler unconfigure function tries to get the fileno of stderr. If stderr is closed it throws a ValueError. When running in a CI it looks like tests failed since pytest exits with an error but actually all the tests have passed - just the teardown failed.

A quick fix would be to expand the scope of caught exceptions in the get_stderr_fileno() function in faulthandler.py. Currently ValueError is not caught.

Traceback

Traceback (most recent call last):
  File "/home/venv3.9/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 192, in console_main
    code = main()
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 169, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/main.py", line 313, in wrap_session
    config._ensure_unconfigure()
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1058, in _ensure_unconfigure
    self.hook.pytest_unconfigure(config=self)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/venv3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/faulthandler.py", line 43, in pytest_unconfigure
    faulthandler.enable(file=get_stderr_fileno())
  File "/home/venv3.9/lib/python3.9/site-packages/_pytest/faulthandler.py", line 48, in get_stderr_fileno
    fileno = sys.stderr.fileno()
ValueError: I/O operation on closed file

Platform

Python: 3.9.10
Platform: Linux-5.4.226-129.415.amzn2.x86_64-x86_64-with-glibc2.31

pytest: 7.4.2
pluggy: 1.3.0

Plugins:

Faker: 19.6.1
anyio: 4.0.0
cov: 4.1.0
csv: 3.0.0
django: 4.5.2
factoryboy: 2.5.1
flake8: 1.1.1
html: 4.0.2
icdiff: 0.7
instafail: 0.5.0
isort: 3.1.0
metadata: 3.0.0
mock: 3.11.1
mypy: 0.10.3
ordering: 0.6
randomly: 3.15.0
subprocess: 1.5.0
timeout: 2.1.0
xdist: 3.3.1
typeguard: 2.13.3

[RonnyPfannschmidt]

im cuious as to why faulthandler is enabled by default

i believe thats a missing case in our suite, in particualr since the ValueError comes in there

[nicoddemus]

im cuious as to why faulthandler is enabled by default

IIRC we decided it would be more useful if it was enabled by default, given it has low impact and when crashes happen it is extremely useful. Also probably users wouldn't even know about it unless it is enabled by default.

I'm curious though what caused the file handler to be closed in the first place? I suppose something in your test suite/production code doing something with that file handler?

Regardless, indeed seems like the solution is just to add ValueError to the exceptions handled here:

pytest/src/_pytest/faulthandler.py

Lines 46 to 58 in 6c2feb7

	def get_stderr_fileno() -> int:
	try:
	fileno = sys.stderr.fileno()
	# The Twisted Logger will return an invalid file descriptor since it is not backed
	# by an FD. So, let's also forward this to the same code path as with pytest-xdist.
	if fileno == -1:
	raise AttributeError()
	return fileno
	except (AttributeError, io.UnsupportedOperation):
	# pytest-xdist monkeypatches sys.stderr with an object that is not an actual file.
	# https://docs.python.org/3/library/faulthandler.html#issue-with-file-descriptors
	# This is potentially dangerous, but the best we can do.
	return sys.__stderr__.fileno()

[bnomis]

I tried to find a call to stderr.close() in our code. It must be happening somewhere so I am curious too but I could not find anything.

[RonnyPfannschmidt]

@nicoddemus no, what i meant was why the faulthandler was initially enabled, based on the error and the current code, the exception happens when trying to re-enable faulthandler in pytest shutdown based on having remembered that it was enabled

[RonnyPfannschmidt]

@bnomis its thinkable that output capture was not entirely disabled yet and thus the file is currently not valid