AUTO cases cannot be used from test files with non-test prefix, such as bench_*.py. Even if the latter are correct test files according to python_files. #326

@mroeschke found this following #320

pytest introduced (in version 3 I guess) a python_files configuration option allowing users to change the naming patterns pytest is using to collect tests. (https://docs.pytest.org/en/stable/reference/reference.html#confval-python_files)

in #320 we became stricter on allowing AUTO cases to execute only on "valid test files" but we do not take into account the python_files glob patterns for this. Besides, our current definition of AUTO cases naming only holds for tests named test_*, and therefore will fail even for tests named *_test.py (which are considered valid by default in pytest).

See also https://docs.pytest.org/en/7.1.x/example/pythoncollection.html#changing-naming-conventions

• We should probably try to leverage one of pytest internal functions to check if a file is a test file. That would ensure that the config options are used correctly and exactly the same way than inside pytest.
• concerning the AUTO cases naming pattern, the above makes it hard to be "smart". The only thing I can think of (but it is ugly), would be to repeat the information explicitly to define a mapping. So

[pytest]
python_files =
    test_*.py
    check_*.py
    example_*.py

[pytest-cases]
auto_cases_files =
    test_*.py -> cases_*.py
    check_*.py -> cases_*_check.py
    example_*.py -> example_*_cases.py

The convention would be to have <source_pattern> -> <dest_pattern>, and it would be restricted to a single star acting as a capturing group.

Thoughts ?