I'm sad and tire of the way PyTest handles sys.path

[Ark-kun]

The sun has long set, but I'm still sitting in my chair after all the hours fighting with PyTest. PyTest surely is magical, but when the magic does not work as intended, it can drive you mad like a company's "voice assistant" that does not help you, but also does not let you talk to a human. Sometimes I really long for Python's unittest way of doing what the user asks.

Our tests system is complex. Everything happens remotely. I'm not running pytest, the system does. The system runs pytest program and allows me to specify CLI args.

Here is our dir structure

third_party/
├─ databricks_cli/
│  ├─ __init__.py
│  ├─ ...
│  ├─ tests/
│  │  ├─ __init__.py
│  │  ├─ conftest.py
│  │  ├─ utils.py
│  │  ├─ group_foo/
│  │  │  ├─ __init__.py
│  │  │  ├─ test_foo.py

test_foo.py:

from tests import utils
E   ModuleNotFoundError: No module named 'tests'

Also

    from tests import conftest
E   ModuleNotFoundError: No module named 'tests'

Also

    import conftest
E   ModuleNotFoundError: No module named 'conftest'

When I do the equivalent of cd "third_party/databricks_cli" && pytest, the sys.path only has third_party, not third_party/databricks_cli.

When I do the equivalent of cd "third_party/databricks_cli/tests" && pytest, the sys.path only has third_party, not third_party/databricks_cli.

unittest has --start-directory and --top-level-directory and changing them results in actual changes.

Wherever I run, there are only walls.
I know that using python -m pytest instead of pytest would solve the issue. But I cannot replace the pytest command executable.
I see people trying to use rootdir to affect sys.path. The documentation now tells in bold letters that's wrong. #4594
I was hopeful when I saw that a PR added a pythonpath option. #9134 I tried pytest --override-ini='pythonpath=foo', but this option is not really working for me and some other users: #11311 And it looks like pythonpath is not really PYTHONPATH. Back to square one.

I feel powerless. There seems to be nothing I can do to change the PyTests's sys.path behaviors.

A simple thing. Adding current directory to sys.path. Impossible.

I'm sad and tired. Thank you for listening. You can close this bug as "works as intended" now.

[RonnyPfannschmidt]

Since you have all the dunder init files, of course pytest is going to go forward with the parent folders

Using relative imports should solve all of the issues you see.

If pytest added every folder under the sun import name conflicts would create hell.

This behavior is documented.

The impression I get of you based on this is

• too lazy to read the docs
• too proud to ask for help when still possible
• just spiteful enough to passive aggressive rant afterwards

[The-Compiler]

Looks like we already have #11311 for the --override-ini issue. Other than that, as you say yourself, I don't see anything actionable in your post.

[nicoddemus]

@Ark-kun I emphasize with the frustration, and we maintainers should also try to look at this post and emphasize as well -- we all felt frustrated after trying for hours to do something we felt should be simple, but kept hitting a wall.

I emphasize after spending the last 3 hours investigating #11475 and realizing there's no simple solution, due to the complexities of Python imports and the pytest needs regarding importing files.

As mentioned, the issue is caused because you have __init__.py files, so everything seems under a package, however the code is trying to use relative paths to import the files:

from tests import utils
E   ModuleNotFoundError: No module named 'tests'

Changing this to local imports should fix it:

from .tests import utils

This is also I think the recommended way in general, not only to appease pytest.