Skip to content

Conversation

realtyem
Copy link

Normally poetry installs a project into a randomiish directory of the sort:
/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.13
The tough part there is the hashy looking code and the python version change with each version of python. Since the cache is keyed by python version/extras installed/hash of poetry.lock file, it seems fine to just install the dependencies and project into a known 'never changing' spot, like the .venv directory inside the project.

Seems to save about 30 seconds to a minute in some 'ad-hoc' testing of installing during several synapse CI test runs

@realtyem realtyem requested a review from a team as a code owner July 14, 2025 09:32
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I support reusing a python environment if the poetry.lock file has changed. But we should not re-use a venv if the python version is different. If a native module is compiled against python 3.10, we shouldn't try to use it with python 3.11.

The hash (i.e. pswDeSvb) is a hash of the project directory's path, ensuring a different hash per project.

@realtyem
Copy link
Author

realtyem commented Jul 22, 2025

Oh! No, different versions of python are covered! Notice the cache key for this part:
poetry-venv-Linux-3.13.5-0f5a5a6beccff673210b8a43a3fb3601f6396194f8b94d52507a8f745e71429c-all

To be clear though, this would not restore the cache if the poetry.lock hash has changed. I am onboard with the idea of finding a way around that, but I suspect it will be more difficult to do safely(and without having unbounded growth of a given python version's cache entry). The cache would just be replaced by any updates/removals on job finish, so that may be fine. But it would remove the poetry.lock hash from the cache key.

(For full reference, the first cache is for the install of poetry and it's key looks something like:
poetry-install-cache-ubuntu-24.04-3.13.5-2.1.1)

I was able to test this on my repo, just to see what it looked like. Here was an example test run for synapse(this particular one is for the Typechecking job, but I did update all of them for this test. Setup Poetry was the step name to search in).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants