Test deselected if marker name matches parametrize id

[dimaspivak]

I've noticed a confusing situation that occurs when I write a test that is decorated with a marker that has the same name as an argvalue I pass in to the parametrize marker for that test when run with -m <marker>. Hard to explain, but super easy to reliably demonstrate. I'm seeing this on pytest 3.0.7 (Python 3.5.2, Mac OS X 10.12.4):

# test_thing.py
import pytest

@pytest.mark.cluster
@pytest.mark.parametrize('mode', ('cluster', 'standalone'))
def test_one(mode):
    assert True

Running pytest -vs --collect-only shows both tests selected. On the other hand, running pytest -vs --collect-only -m cluster will deselect the test_one[cluster] item and leave me only with the test_one[standalone].

To try to debug what's happening, I used the pytest_collection_modifyitems hook in a local conftest.py to print the keywords.keys() attribute for each item collected. That is,

# conftest.py
def pytest_collection_modifyitems(items):
    for item in items:
        print(item.keywords.keys())

The output when re-running my test (whether with or without the -m cluster shows the following keys:

['ex', 'parametrize', 'test_thing.py', 'cluster', 'test_one[cluster]']
['standalone', 'test_one[standalone]', 'parametrize', 'cluster', 'ex', 'test_thing.py']

In other words, it looks as if duplicate marker/parametrize ids are being deleted at some point, which causes the test to be deselected.

I've tracked down that this is happening at the marker and id level as the following snippet works around the error:
@pytest.mark.parametrize('mode', ('cluster', 'standalone'), ids=('cluster_', 'standalone')).

[RonnyPfannschmidt]

the problem seems to relate to how "keywords" are composed

mark matching doesnt use marks, but "keywords" which are used to store marks, this is yet another mark bug thats hard to fix

https://github.com/pytest-dev/pytest/blob/master/_pytest/python.py#L369 adds the parameter id to the node keywords

and matchmarks will match keywords instead of marks

[dimaspivak]

Dang. Well thanks for the explanation, @RonnyPfannschmidt; I really appreciate you taking the time to respond (and to be awesome on the project in general 😄 ).

[RonnyPfannschmidt]

that was the least i could do, fixing the inner details of mark to resolve such strange edge-cases will take quite a while - so atm all we can do is explain the issues as issues and fix them