Description
Currently looking into pytest to port over a unittest-based test system with quite a bunch of customisations, we have a number of base "test superclass" for opt-in capabilities and setup/teardown code reuse.
While in the future we'll probably phase them out for regular fixtures & markers, right now I'm trying to "softly" add pytest and these superclasses seemed like a nice seam for a few default markers, fixtures and the like, and then things started breaking down completely: class-level markers (and fixtures since I understand fixtures are enabled through markers? I may be mistaken) are shared between classes rather than only copied/shared from superclass to subclass.
More precisely, if a class has a marker, any marker apparently set on a subclass will actually be set on the superclass (and on all "sibling" subclasses"):
@pytest.mark.a
class Base(unittest.TestCase):
pass
@pytest.mark.b
class TestBar(Base):
def test_foo(self):
pass
class TestBaz(Base):
def test_qux(self):
self.fail("should not execute")Selecting using -m b, one would expect only test_foo would be collected…
> py.test -m b test_foo.py --collect-only
= test session starts =
platform darwin -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/masklinn/projects/scratchpad/test, inifile:
collected 2 items
<Module 'test_foo.py'>
<UnitTestCase 'TestBar'>
<TestCaseFunction 'test_foo'>
<UnitTestCase 'TestBaz'>
<TestCaseFunction 'test_qux'>
= in 0.02 seconds =
Digging in the code, the problem is in MarkDecorator:
if hasattr(func, '__bases__'):
if hasattr(func, 'pytestmark'):
l = func.pytestmark
if not isinstance(l, list):
func.pytestmark = [l, self]
else:
l.append(self)
else:
func.pytestmark = [self]so if func is a class and it has a pytestmark attribute which is already a list then append the new MarkDecorator to the list. This is problematic if the existing pytestmark comes from a superclass rather than being set on the current class.
I see a number of other possible problems with the code in case pytestmark is set explicitly (rather than implicitly via mark decorators)
- if pytestmark is an iterable but not a list (tuples and sets are seem easy/possible) things may completely blow up
- pytestmarks from multiple bases aren't merged, one is picked depending on the order of bases
- if a pytestmark is defined explicitly on the subclass, superclass pytestmarks are ignored
These seem like somewhat less problematic issue though as they're explicit and follow fairly normal Python inheritance concerns. The oddity with MarkDecorator is it's completely implicit, and seems desirable in no situation whatsoever.
Also why look for __bases__ instead of seeing if func is an instance of type or types.ClassType?