Update check_dependencies to support markers

[anoadragon453]

This PR addresses an issue that came up during #19071. It allowed CI to pass on that PR's branch.

The Problem

@V02460 had added the following to pyprojects.toml:

[tool.poetry.dependencies]
pydantic = [
    { version = "~= 2.8", python = "< 3.14" },
    { version = "~= 2.12", python = ">= 3.14" },
]

then the tests-olddeps CI job started failing with:

Synapse 1.141.0rc1 needs pydantic==2.12; python_version == "3.14", but got pydantic==2.8.0
Missing Requirements: "pydantic"
To install run:
    pip install --upgrade --force "pydantic"

tests-olddeps uses Python 3.9 and pydantic==2.8.0, which should have been fine according to pyproject.toml. However, the check_dependencies.check_requirements method raised the above. This is because it didn't take python into account at all, and failed when it read version = "~= 2.12".

Python Packaging Background

The python bit in the snippet above is actually poetry syntax. It translates to the following PEP 508 environment "marker" syntax:

pydantic >=2.8,<3.0 ; python_version < "3.14"
pydantic>=2.12,<3.0; python_version >= "3.14"

python_version < "3.14" is a "marker", and a specific dependency (i.e. requirement) can have multiple (i.e. (sys_platform != 'win32') and (python_version >= '3.10')`).

In addition to markers, we have "extras", which we're more familiar with. You can specify an extra identifier, such as "postgres" when installing pip install matrix-synapse[postgres] and it'll install extra dependencies specifically for using Synapse with postgres.

Markers can also include extras as a conditional, stating that a particular requirement only applies when the user has asked for a particular marker. For example:

  # Only install if the user asks for the "socks" extra
  # AND they're on CPython
  "pysocks>=1.7; extra == 'socks' and platform_python_implementation == 'CPython'",

You'll notice we have some boolean logic here (and, or, etc.). The python packaging code represents a "marker" on a requirement as a Marker Tree. For example:

requests>=2.32 ; python_version >= "3.9" and (sys_platform == "win32" or platform_system == "Darwin")

becomes:

AND
├── comparison: python_version >= "3.9"
└── OR
    ├── comparison: sys_platform == "win32"
    └── comparison: platform_system == "Darwin"

requirement.marker in the code is actually an object representing this tree. i.e. it includes all the conditions for a given requirement.

A "marker environment" is the environment that a particular marker is evaluated in. Parts of the environment are: what version of Python is the user using, what OS are they on, what extras did they ask for, etc. Markers are "evaluated" for whether they apply according to this environment.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[MadLittleMods]

This looks similar to the logic in _marker_applies_for_any_extra(...)

Although we have an extra extra in DEV_EXTRAS condition to satisfy. And we can't just separate things.

Just a note and I think this is fine ⏩

[anoadragon453]

Yes, it's a bit of a shame. But the only ideal way to combine them would be adding a boolean argument to the function signature, and that feels dirtier than the two (simple) methods IMO.

[MadLittleMods]

Nothing seems too crazy. And the background from the PR description makes sense. I haven't made sure all of the pieces here are aligning in terms of giving a complete picture but I assume the tests cover this.

[anoadragon453]

That, and the fact that the linked Pydantic v2 PR now passes CI won me over.

[anoadragon453]

Thanks for the review @MadLittleMods! 🦄

[anoadragon453]

Yes, it's a bit of a shame. But the only ideal way to combine them would be adding a boolean argument to the function signature, and that feels dirtier than the two (simple) methods IMO.

[anoadragon453]

That, and the fact that the linked Pydantic v2 PR now passes CI won me over.

[anoadragon453]

CI failures are due to flaky tests.