unittest.TestCase cleanup functions not invoked on test failure

[cecilbot]

stdlib unittest style cleanup functions registered with unittest.TestCase.addCleanup are not invoked when a test fails. It appears this issue was introduced in pytest version 5.4.0, examples below for version 5.4.1.

System Info

• Ubuntu 18.04.3 LTS
• Python 3.6.8
• pytest 5.4.1

Example test and pytest output

import unittest

def cleanup():
    raise Exception('cleanup')

class Test(unittest.TestCase):
    def setUp(self):
        print('setup')
        self.addCleanup(cleanup)

    def tearDown(self):
        print('teardown')

    def test_no_cleanup(self):
        assert False

    def test_cleanup(self):
        assert True

(venv-3.6.8) cecil@python36-vm:~$ pytest ceciltest.py
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.6.8, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/cecil
plugins: cov-2.8.1
collected 2 items

ceciltest.py FF                                                                                                                                                                                  [100%]

=============================================================================================== FAILURES ===============================================================================================
__________________________________________________________________________________________ Test.test_cleanup ___________________________________________________________________________________________

    def cleanup():
>       raise Exception('cleanup!')
E       Exception: cleanup!

ceciltest.py:4: Exception
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
setup
teardown
_________________________________________________________________________________________ Test.test_no_cleanup _________________________________________________________________________________________

self = <ceciltest.Test testMethod=test_no_cleanup>

    def test_no_cleanup(self):
>       assert False
E       assert False

ceciltest.py:16: AssertionError
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
setup
--------------------------------------------------------------------------------------- Captured stdout teardown ---------------------------------------------------------------------------------------
teardown
======================================================================================= short test summary info ========================================================================================
FAILED ceciltest.py::Test::test_cleanup - Exception: cleanup!
FAILED ceciltest.py::Test::test_no_cleanup - assert False
========================================================================================== 2 failed in 0.12s ===========================================================================================

Trying pytest 5.3.5 (works as expected)

(venv-3.6.8) cecil@python36-vm:~$ pytest ceciltest.py
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.6.8, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/cecil
plugins: cov-2.8.1
collected 2 items

ceciltest.py FFE                                                                                                                                                                                 [100%]

================================================================================================ ERRORS ================================================================================================
______________________________________________________________________________ ERROR at teardown of Test.test_no_cleanup _______________________________________________________________________________

    def cleanup():
>       raise Exception('cleanup!')
E       Exception: cleanup!

ceciltest.py:4: Exception
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
setup
teardown
=============================================================================================== FAILURES ===============================================================================================
__________________________________________________________________________________________ Test.test_cleanup ___________________________________________________________________________________________

    def cleanup():
>       raise Exception('cleanup!')
E       Exception: cleanup!

ceciltest.py:4: Exception
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
setup
teardown
_________________________________________________________________________________________ Test.test_no_cleanup _________________________________________________________________________________________

self = <ceciltest.Test testMethod=test_no_cleanup>

    def test_no_cleanup(self):
>       assert False
E       AssertionError: assert False

ceciltest.py:16: AssertionError
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
setup
teardown
====================================================================================== 2 failed, 1 error in 0.12s ======================================================================================

pip list

(venv-3.6.8) cecil@python36-vm:~$ pip list
Package            Version
------------------ ----------
astroid            2.3.3
attrs              19.3.0
boto               2.49.0
certifi            2019.11.28
chardet            3.0.4
coverage           5.0.4
httpretty          0.8.10
idna               2.9
importlib-metadata 1.5.0
isort              4.3.21
Jinja2             2.11.1
lazy-object-proxy  1.4.3
MarkupSafe         1.1.1
mccabe             0.6.1
more-itertools     8.2.0
moto               0.4.31
packaging          20.3
parameterized      0.7.0
pep8               1.6.2
pip                18.1
pluggy             0.13.1
py                 1.8.1
pylint             2.4.4
pyparsing          2.4.6
pytest             5.4.1
pytest-cov         2.8.1
python-dateutil    2.8.1
pytz               2019.3
requests           2.23.0
setuptools         40.6.2
six                1.14.0
typed-ast          1.4.1
urllib3            1.25.8
wcwidth            0.1.8
Werkzeug           1.0.0
wrapt              1.11.2
xmltodict          0.12.0
zipp               3.1.0

[mdmintz]

Here's a unit test to add to https://github.com/pytest-dev/pytest/blob/master/testing/test_unittest.py when the fix is ready. (Passes on 5.3.5, Fails on 5.4.0)

def test_outcome_errors(testdir):
    testpath = testdir.makepyfile(
        """
        import unittest
        class MyTestCase(unittest.TestCase):
            def test_fail(self):
                raise Exception("FAIL!")
            def tearDown(self):
                print(self._outcome.errors)
    """
    )
    reprec = testdir.inline_run(testpath)
    passed, skipped, failed = reprec.countoutcomes()
    assert failed == 1, failed

If pytest does TDD, I can just create a pull-request for this test right now if you want, and it'll be failing until the issue is fixed.

[nicoddemus]

@mdmintz are you sure your test is related to this issue? See #7049 (comment).

If not, could you please open a separate issue? Thanks!