Indirect session fixture reordering not working as expected

[nicoddemus]

There seems to be a problem with our reordering code that optimizes higher-level fixtures' execution order.

import pytest

@pytest.fixture(scope="session")
def prepare(request):
    print("\nprepare setup with {!r}".format(request.param))

@pytest.mark.parametrize('prepare', ["dina"], indirect=True, scope="session")
def test_1(prepare):
    print("Test1 done\n")

@pytest.mark.parametrize('prepare', ["mor"], indirect=True, scope="session")
def test_2(prepare):
    print("Test2 done\n")

@pytest.mark.parametrize('prepare', ["dina"], indirect=True, scope="session")
def test_3(prepare):
    print("Test3 done\n")

I get this output:

λ pytest .tmp\test_fix_session.py -sq

prepare setup with 'dina'
Test1 done

.
prepare setup with 'mor'
Test2 done

.
prepare setup with 'dina'
Test3 done

.
3 passed in 0.02s

We would expect it to reorder the tests to [test_1, test_3, test_2] so the fixture would execute only twice, instead of 3 times as it is doing now, so definitely looks like a bug (unless I'm missing something).

Thanks @dina809 for the original report (#8328).

Originally posted by @nicoddemus in #8328 (comment)

[Formartha]

Subscribed

[symonk]

I'm gonna have a look into this one next on my hit list, I've read the other thread for all the context and will start debugging soon

[haxtibal]

@nicoddemus This issue hit me as well, hopefully I found the root cause and a possible fix, see this commit

Reordering test items only took parameter indexes into account. This
works fine for direct parametrization. But for indirect parametrization,
the parameter index is always 0, and therefore every item went into the
same argkey group.

Now also add the parameter value to the argkey. This is enough
information to establish different argkey groups and make the grouping
algorithm work.

The output from your minimal reproducer is now as expected

prepare setup with 'dina'
Test1 done
Test3 done
prepare setup with 'mor'
Test2 done

but the change makes 3 tests fail (test_parameters_without_eq_semantics, test_parameters_without_eq_semantics, test_parametrize_issue634). I've not yet looked at them in detail.

Could you please give some short feedback? If you think I'm on the right track, I'll try to understand and fix the regression and open a PR here.

[nicoddemus]

Hi @haxtibal, that's great! I looked at the code and it makes sense.

The failing tests are about making sure that pytest works even if the argvalues can't compare with == (raise an error for example).

I think a solution is to return a wrapper instead of the actual object, something like:

@attr.s(autoattribs=True, eq=False)
class SafeEqWrapper:

    obj: object

    def __eq__(self, other: object) -> Any:
        try:
            res = self.obj == other
            bool(res)
            return res
        except Exception:
            # perhaps return False?
            return id(self.obj) == id(other)

And then use it like this:

key: _Key = (argname, param_index, SafeEqWrapper(param_value))

Feel free to open a PR even if incomplete, we can discuss this over the code. 👍