[red-knot] PEP 561 compliant module resolver

[carljm]

(Or intentional choices to diverge from PEP 561 compliance.)

Subtasks:

• Vendor typeshed's stubs: [red-knot] Vendor typeshed's stdlib #11340
• Add automation for keeping typeshed's stubs up to date: Add automation for updating our vendored typeshed stubs #11427
• Implement --custom-typeshed-dir internally: [red-knot] Encapsulate module resolution logic in module.rs #11767
• Implement the typing spec's module resolution order:
  • [red-knot] Encapsulate module resolution logic in module.rs #11767
  • Proposal for tweaking the spec posted at https://discuss.python.org/t/pep-561-s-module-resolution-order-seems-incorrect/55048/1
• Make vendored typeshed stubs available from the built Ruff binary: [red-knot] Include vendored typeshed stubs as a zipfile in the Ruff binary #11779
• Use vendored typeshed stubs at runtime to resolve symbols to stdlib definitions if --custom-typeshed-dir is not passed
  • Add a VendoredFileSystem implementation: [red-knot]: Add a VendoredFileSystem implementation #11863
  • Use the VendoredFileSystem implementation to resolve stdlib stubs: [red-knot] Use vendored typeshed stubs for stdlib module resolution #12224
• Add support for stubs packages suffixed with -stubs installed into site-packages (must take priority over other packages installed into site-packages)
• Add support for typeshed's VERSIONS file: we shouldn't fallback to a typeshed module if the VERSIONS file specifies it does not exist with the --target-version the user passes.
  • Add a parser for VERSIONS: [red-knot] Add a parser for typeshed's VERSIONS file #11836
  • Use the VERSIONS file when resolving stdlib modules: [red-knot] Respect typeshed's VERSIONS file when resolving stdlib modules #12141
• Add support for editable installs that use static ._pth files:
  • [red-knot] Add a read_directory() method to the ruff_db::system::System trait #12289
  • [red-knot] Add support for editable installs to the module resolver #12307
• Fix [red-knot] Module resolver has incorrect behaviour if a single-file module and potential namespace package coexist #12278
• Add understanding of py.typed files with partial in them, which are used to explicitly mark a package's annotations as partially complete
• Audit mypy's and pyright's module resolvers to see if there's anything that's not in the spec that still needs to be implemented

[MichaReiser]

One thing that we need to implement as part of this is a vendored filesystem. It's not a functionality but a task to complete on our way to get there.

[AlexWaygood]

One thing that we need to implement as part of this is a vendored filesystem. It's not a functionality but a task to complete on our way to get there.

Right. I was considering that part of

Use vendored typeshed stubs at runtime to resolve symbols to stdlib definitions if --custom-typeshed-dir is not passed

since, as you say, it's necessary to implement a vendored filesystem in order to be able to do that

[MichaReiser]

@AlexWaygood is there more work that needs doing or can we consider this done?

[AlexWaygood]

We can't call ourself PEP-561-compliant until we do this, which is still outstanding, and important. It's not huge, but also not trivial:

* [ ]  Add support for stubs packages suffixed with `-stubs` installed into `site-packages

But we decided to defer work on it for now because the type-inference work is more important.

We should also add support for py.typed files that explicitly mark type annotations as partial, which is specified by the PEP but not listed in the tasks above; I'll add that now.