pytest --pdb does not run tearDown for skipped tests

[lanzz]

test.py:

import unittest


class Tests(unittest.TestCase):

    def setUp(self):
        print 'setUp'

    def test_skip(self):
        print 'skip'
        self.skipTest('skip')

    def tearDown(self):
        print 'tearDown'

Output without --pdb:

$ pytest -s test.py 
=========================== test session starts ============================
platform darwin -- Python 3.6.8, pytest-5.2.1, py-1.8.0, pluggy-0.12.0
collected 1 item                                                           

test.py setUp
skip
tearDown
s

============================ 1 skipped in 0.01s ============================

Output with --pdb:

$ pytest -s --pdb test.py 
=========================== test session starts ============================
platform darwin -- Python 3.6.8, pytest-5.2.1, py-1.8.0, pluggy-0.12.0
collected 1 item                                                           

test.py setUp
skip
s

============================ 1 skipped in 0.01s ============================

When --pdb is not specified, tearDown is run as expected. With --pdb, is isn't run. I've mostly tracked this down to exception handling in unittest.TestCase.run() that will intercept the SkipTest exception raised by TestCase.skipTest() and still run tearDown, but TestCase.debug() does not handle any exceptions, including SkipTest, so it will not run tearDown. This could be argued to be an upstream bug in unittest itself, but I'm not even sure if TestCase.debug() is actually the intended mechanism to be used for this scenario.

This basically means that after skipped tests there's no way to tear down stuff that has been set up in setUp when running with --pdb.

pip list:

Package            Version
------------------ -------
atomicwrites       1.3.0  
attrs              19.3.0 
importlib-metadata 0.23   
more-itertools     7.2.0  
packaging          19.2   
pip                19.3.1 
pluggy             0.13.0 
py                 1.8.0  
pyparsing          2.4.2  
pytest             5.2.1  
setuptools         41.4.0 
six                1.12.0 
wcwidth            0.1.7  
wheel              0.33.6 
zipp               0.6.0  

[blueyed]

The reason for this is:

pytest/src/_pytest/unittest.py

Lines 205 to 212 in 408e53c

	def runtest(self):
	if self.config.pluginmanager.get_plugin("pdbinvoke") is None:
	self._testcase(result=self)
	else:
	# disables tearDown and cleanups for post mortem debugging (see #1890)
	if self._handle_skip():
	return
	self._testcase.debug()

(#1890, see my comments at #1890 (comment))

I think it is bad to do it like that still - since this gets done regardless of pdb was really invoked after all, but just when --pdb is used as an option already.

I think it should rather not do anything special, and then pdb should be invoked before any teardown (on test failure), but when continuing the tests (or quitting from there) it should still run the teardown.
/cc @mbyt

[blueyed]

unittest's debug does this:
https://github.com/python/cpython/blob/24ddd9c2d6ab61cbce7e68d6de36d4df9bd2c3fb/Lib/unittest/case.py#L721-L728

    def debug(self):
        """Run the test without collecting errors in a TestResult"""
        self.setUp()
        getattr(self, self._testMethodName)()
        self.tearDown()
        while self._cleanups:
            function, args, kwargs = self._cleanups.pop(-1)
            function(*args, **kwargs)

I.e. it will not call self.tearDown() itself when there is a SkipTest exception.

And Django does this in its TestCase: https://github.com/django/django/blob/1711c509faf3111bdf5a3a860b2cd01c0dc5d233/django/test/testcases.py#L246-L279
Initially added in django/django@1711c50

[blueyed]

At its core it appears to be a Python issue, but then it is maybe a misuse of using debug() in the first place here (and seems to only be used as a workaround for --pdb in pytest, while e.g. Django only implements it to help users using .debug() explicitly).

[lanzz]

Ordinarily I would not use --pdb by default, but I'm troubleshooting a large legacy codebase where some tests leak side effects between each other and I'm trying to spot cases like this and fix them. Running without --pdb is not sufficient, because a failing test might not fail in isolation, and I won't have access to its state if I don't get a debugger immediately, as part of the full run of the entire test suite.

[blueyed]

Please try #5996.