Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: introduce read-only and read-write PyTorch build caches #1546

Merged
merged 1 commit into from
Nov 2, 2022
Merged

ci: introduce read-only and read-write PyTorch build caches #1546

merged 1 commit into from
Nov 2, 2022

Conversation

ashay
Copy link
Collaborator

@ashay ashay commented Nov 2, 2022

Until recently, we had to either risk feature branches creating PyTorch
build caches (which were unusable by the main branch or other parallel
feature branches because of GitHub's rules around sharing caches among
branches) or we had to limit the PyTorch build caches to only the main
branch, causing CI runs on feature branches to be terribly slow because
they had to rebuild PyTorch each time.

This patch enables the best of both worlds, by using a fork
(github.com/ashay/cache) of the GitHub's cache action, where the fork
adds an option (called save) which, when set, uploads a new cache
entry. We thus set this save flag only when we're building PyTorch
from source in Torch-MLIR's main branch, whereas all other builds set
this save flag to false.

The ability to conditionally update the cache has been an oft-requested
feature on the original (github.com/actions/cache) repository and
multiple unmerged PRs exist to allow conditional cache updates, so it is
likely that using the fork is only a temporary solution.


Here is a run (on the main branch) that created a PyTorch build cache, and here is a run in a feature branch that skips the cache update.

Until recently, we had to either risk feature branches creating PyTorch
build caches (which were unusable by the main branch or other parallel
feature branches because of GitHub's rules around sharing caches among
branches) or we had to limit the PyTorch build caches to only the main
branch, causing CI runs on feature branches to be terribly slow because
they had to rebuild PyTorch each time.

This patch enables the best of both worlds, by using a fork
(github.com/ashay/cache) of the GitHub's cache action, where the fork
adds an option (called `save`) which, when set, uploads a new cache
entry.  We thus set this `save` flag only when we're building PyTorch
from source in Torch-MLIR's main branch, whereas all other builds set
this `save` flag to `false`.

The ability to conditionally update the cache has been an oft-requested
feature on the original (github.com/actions/cache) repository and
multiple unmerged PRs exist to allow conditional cache updates, so it is
likely that using the fork is only a temporary solution.
@ashay ashay requested a review from powderluv November 2, 2022 03:17
@powderluv
Copy link
Collaborator

This is awesome. Looks like upstream is adding support soon actions/cache#964 but we can have a fork for now.

@powderluv powderluv merged commit 031d127 into llvm:main Nov 2, 2022
@ashay ashay deleted the ashay/read-only-pytorch-build-cache branch November 2, 2022 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants