fixture scope is ignored when using metafunc.parametrize()

[pytestbot]

Originally reported by: Nikolaus Rath (BitBucket: nikratio, GitHub: nikratio)

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/519

---

Please take a look at the following example:

import pytest

def pytest_generate_tests(metafunc):
    if 'arg1' in metafunc.fixturenames:
        metafunc.parametrize("arg1", [ 'arg1v1', 'arg1v2' ], scope='module')

    if 'arg2' in metafunc.fixturenames:
        metafunc.parametrize("arg2", [ 'arg2v1', 'arg2v2' ], scope='function')

@pytest.yield_fixture(scope='module')
def fix1(arg1):
    print('running fix1 with', arg1)
    yield 'fix1-' + arg1

@pytest.yield_fixture(scope='function')
def fix2(fix1, arg2):
    print('running fix2 with', fix1, arg2)
    yield 'fix2-' + arg2 + fix1

def test_one(fix2):
    assert True

def test_two(fix2):
    assert True

As I understand, test_one and test_two should be executed 4 times each. The fix2 function should be called once for every test execution (because it has function scope), and the fix1 function once for every value of arg1 (because it has module scope).

However, in practice fix1 seems to be called for every test function as well, as if it had function scope:

$ python3 ~/clones/pytest/pytest.py -v -s test_bug.py
============================= test session starts ==============================
platform linux -- Python 3.3.5 -- py-1.4.20 -- pytest-2.6.0.dev1 -- /usr/bin/python3
collected 8 items 

test_bug.py@20::test_one[arg1v1-arg2v1] running fix1 with arg1v1
running fix2 with fix1-arg1v1 arg2v1
PASSED
test_bug.py@23::test_two[arg1v1-arg2v1] running fix2 with fix1-arg1v1 arg2v1
PASSED
test_bug.py@20::test_one[arg1v1-arg2v2] running fix1 with arg1v1
running fix2 with fix1-arg1v1 arg2v2
PASSED
test_bug.py@23::test_two[arg1v1-arg2v2] running fix2 with fix1-arg1v1 arg2v2
PASSED
test_bug.py@20::test_one[arg1v2-arg2v1] running fix1 with arg1v2
running fix2 with fix1-arg1v2 arg2v1
PASSED
test_bug.py@23::test_two[arg1v2-arg2v1] running fix2 with fix1-arg1v2 arg2v1
PASSED
test_bug.py@20::test_one[arg1v2-arg2v2] running fix1 with arg1v2
running fix2 with fix1-arg1v2 arg2v2
PASSED
test_bug.py@23::test_two[arg1v2-arg2v2] running fix2 with fix1-arg1v2 arg2v2
PASSED

=========================== 8 passed in 0.01 seconds ===========================

Looks like a bug to me.

[ceridwen]

I have a test suite where I have to run some very expensive configuration before running all the tests, and the set of possible configurations is the Cartesian product of modes that are fixed with versions that I need to be able to pass in at run-time. I'm currently using a session-scoped fixture that's parametrized only by the modes, and I need to extend it to be parametrized by the versions too. The only way I've found to do parametrization of a fixture dependent on command-line arguments is pytest_generate_tests, but this bug breaks it for session fixtures. Thus, I see three possible routes:

1. Fix this bug. I'm guessing, though, that it would be hard to impossible. Where is the code that's responsible for handling fixture scoping? Does anyone know the cause of this bug?
2. Work around it somehow. I don't know to how to do this. Someone on IRC suggested adding pytest.mark.parametrize('session_fixture', params, indirect=True, scope='session') to a test, but for this test suite there's no one test that will always be run, and I don't understand exactly how this is supposed to work. Are there any other workarounds for this bug?
3. Move the session-level configuration out of pytest, which has its own disadvantages.

[nicoddemus]

@ceridwen I think the proper way to define fixtures dynamically based on command-line variables is demonstrated in #2959 (comment). Does that solve it for you?

[ceridwen]

Unfortunately, not quite, see #3161. I'm starting to suspect there's a more fundamental bug lurking here.