Skip to content

Dependencies cache key has to be specific to OS version #432

Description

@illia-v

Description:

A cache key looks like this setup-python-Linux-python-3.11.0-beta.3-pip-60d374041db12689c3d346e149aa43744f06aa781ab1c8cbbe70e0b09b98dd70.
It is specific to a Python version and OS type. It has to be specific to a version of an operating system too.

Justification:

PyPI provides source distributions of packages (archives with code) and built distributions (wheels). When a wheel suitable for an environment is not found on PyPI, a source distribution is usually downloaded and a wheel is built locally. The source may contain some code that needs to be compiled (C, Rust, etc.)

On Linux, a compiled wheel may depend on a specific version of a shared library. Different Ubuntu releases used on GitHub Actions may provide different versions of shared libraries. Therefore, a wheel compiled on Ubuntu 20.04 may be incompatible with Ubuntu 22.04. This makes the cache feature unusable when a workflow runs jobs on multiple Ubuntu versions or when an Ubuntu version is changed in a workflow.

https://github.com/urllib3/urllib3/runs/6817232938 is an example of a failed run.
The error is caused by a cffi wheel that was compiled and cached in Ubuntu 20.04 previously where it depended on libffi.so.7.
Ubuntu 22.04 (that provides libffi.so.8) restored that cache archive because it composed the same cache key, and cffi become unusable there (urllib3/urllib3#2626 (comment) provides a little bit more info.)

BTW, #188 may be a related bug report issue.

Are you willing to submit a PR?

No. Although, I guess it is a matter of changing the following lines of code to include a version of an OS in the key.

const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`;
const restoreKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}`;

Activity

  1. vsafonkin commented on Jun 13, 2022

    @vsafonkin

    Hi @illia-v, we will take a look at this, thank you.

  2. dsame commented on Jun 27, 2022

    @dsame
    Contributor

    @e-korolevskii implementing this feature should solve this issue and this

  3. panticmilos commented on Jul 19, 2022

    @panticmilos
    Contributor

    Hi @illia-v, we merged PR https://github.com/actions/setup-python/pull/467 that is solving this issue.

    For now, I will close this issue, if there are any questions or other issues please feel free to continue the conversation here, reopen the issue, or create a new one.

    Cheers

  4. illia-v commented on Jul 21, 2022

    @illia-v
    Author

    @panticmilos thank you!

  5. illia-v commented on Jul 21, 2022

    @illia-v
    Author

    @panticmilos what do you think about updating these values too?

    const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`;

    const primaryKey = `${this.CACHE_KEY_PREFIX}-${process.env['RUNNER_OS']}-python-${this.pythonVersion}-${this.packageManager}-${hash}`;

  6. panticmilos commented on Aug 10, 2022

    @panticmilos
    Contributor

    Hi @illia-v, sorry for the delayed answer. At first I have implemented that part as well. But then as I and my team discussed, we realized we are saving virtualenv for poetry and pipenv. For pip we are saving global cache and that's why we had to do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

feature requestNew feature or request to improve the current logic

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions