Assertion Error in fixture teardown when test failed is causing crash if '-x' option is used and more than one test executed

[roberfi]

Problem happens when using -x option and more than one test are selected to be executed. Then, if assertion fails for one test and another exception is raised in fixture teardown (in this case another AssertionError), pytest crashes and exception is thrown instead of giving test report.

This is the example code to reproduce the issue:

import pytest


@pytest.fixture(scope="session")
def temp_fixture():
    print("starting")

    yield "anything"

    print("ending")
    assert False, "Failing in fixture"


def test_temp1(temp_fixture):
    print(f"running test: {temp_fixture}")

    assert False, "failing in test"


def test_temp2(temp_fixture):
    print(f"running test: {temp_fixture}")

Output of test execution

pytest -x .

[...]

================================================= test session starts =================================================
platform win32 -- Python 3.7.0, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: D:\Users\username\Desktop\tmp
collected 2 items

test_tmp.py Fending
Traceback (most recent call last):
  File "d:\users\username\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\users\username\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\Users\username\.virtualenvs\username-Q45eS3Uo\Scripts\pytest.exe\__main__.py", line 7, in <module>
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\config\__init__.py", line 185, in console_main
    code = main()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\config\__init__.py", line 163, in main
    config=config
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\main.py", line 305, in wrap_session
    session=session, exitstatus=session.exitstatus
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\terminal.py", line 803, in pytest_sessionfinish
    outcome.get_result()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\runner.py", line 103, in pytest_sessionfinish
    session._setupstate.teardown_all()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\runner.py", line 412, in teardown_all
    self._pop_and_teardown()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\runner.py", line 387, in _pop_and_teardown
    self._teardown_with_finalization(colitem)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\runner.py", line 405, in _teardown_with_finalization
    self._callfinalizers(colitem)
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\runner.py", line 402, in _callfinalizers
    raise exc
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\runner.py", line 395, in _callfinalizers
    fin()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\fixtures.py", line 1034, in finish
    raise exc
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\fixtures.py", line 1027, in finish
    func()
  File "d:\users\username\.virtualenvs\username-q45es3uo\lib\site-packages\_pytest\fixtures.py", line 941, in _teardown_yield_fixture
    next(it)
  File "D:\Users\username\Desktop\tmp\test_tmp.py", line 11, in temp_fixture
    assert False, "Failing in fixture"
AssertionError: Failing in fixture
assert False

If executed without -x option, this is the output:

pytest .

[...]

------------------------------------------------ Captured stdout setup ------------------------------------------------
starting
------------------------------------------------ Captured stdout call -------------------------------------------------
running test: anything
=============================================== short test summary info ===============================================
FAILED test_tmp.py::test_temp1 - AssertionError: failing in test
ERROR test_tmp.py::test_temp2 - AssertionError: Failing in fixture
======================================== 1 failed, 1 passed, 1 error in 0.10s =========================================

pip list:

Package            Version
------------------ -------
atomicwrites       1.4.0
attrs              21.2.0
colorama           0.4.4
importlib-metadata 4.9.0
iniconfig          1.1.1
packaging          21.3
pip                20.0.2
pluggy             1.0.0
py                 1.11.0
pyparsing          3.0.6
pytest             6.2.5
setuptools         46.0.0
toml               0.10.2
typing-extensions  4.0.1
wheel              0.34.2
zipp               3.6.0

Versions:

• OS: Windows Server 2016 (also observed in Ubuntu 20.04)
• Python: 3.7.0 (also observed in Python 3.7.9)
• Pytest: 6.2.5

Checklist:

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

[RonnyPfannschmidt]

i believe the "quick fix" could be to create a session scope test report for that tear-down, but only pass it to reporting on failure

[bluetech]

This is fixed by #11721 I believe, will be part of pytest 8.0.0.