Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate CircleCI cache between main and pull or other branches #24886

Merged
merged 3 commits into from
Jul 18, 2023
Merged

Separate CircleCI cache between main and pull or other branches #24886

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
84caef6
fix
ydshieh Jul 18, 2023
9260203
fix
ydshieh Jul 18, 2023
70f5e49
fix
ydshieh Jul 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions .circleci/create_circleci_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,11 @@ def __post_init__(self):
def to_dict(self):
env = COMMON_ENV_VARIABLES.copy()
env.update(self.additional_env)

cache_branch_prefix = os.environ.get("CIRCLE_BRANCH", "pull")
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far I only worked on .circleci/create_circleci_config.py.

For .circleci/config.yml, it needs somehow a bit more work (as it's not a python file)

if cache_branch_prefix != "main":
cache_branch_prefix = "pull"

job = {
"working_directory": self.working_directory,
"docker": self.docker_image,
Expand All @@ -101,16 +106,21 @@ def to_dict(self):
{
"restore_cache": {
"keys": [
f"v{self.cache_version}-{self.cache_name}-pip-" + '{{ checksum "setup.py" }}',
f"v{self.cache_version}-{self.cache_name}-pip-",
# check the fully-matched cache first
f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-pip-" + '{{ checksum "setup.py" }}',
# try the partially-matched cache from `main`
f"v{self.cache_version}-{self.cache_name}-main-pip-",
Comment on lines +111 to +112
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to main vs pull, we don't want pr_2 being affected by pr_1.

We can be confident to use the cache used in main: the problem happens only when we use dev. version on main (say accelerate), and we decide to use the un-pinned release version in a PR (that is behind the pinned dev. version).

# try the general partially-matched cache
f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-pip-",
]
}
},
{
"restore_cache": {
"keys": [
f"v{self.cache_version}-{self.cache_name}-site-packages-" + '{{ checksum "setup.py" }}',
f"v{self.cache_version}-{self.cache_name}-site-packages-",
f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-site-packages-" + '{{ checksum "setup.py" }}',
f"v{self.cache_version}-{self.cache_name}-main-site-packages-",
f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-site-packages-",
]
}
},
Expand All @@ -119,15 +129,15 @@ def to_dict(self):
steps.append(
{
"save_cache": {
"key": f"v{self.cache_version}-{self.cache_name}-pip-" + '{{ checksum "setup.py" }}',
"key": f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-pip-" + '{{ checksum "setup.py" }}',
"paths": ["~/.cache/pip"],
}
}
)
steps.append(
{
"save_cache": {
"key": f"v{self.cache_version}-{self.cache_name}-site-packages-" + '{{ checksum "setup.py" }}',
"key": f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-site-packages-" + '{{ checksum "setup.py" }}',
"paths": ["~/.pyenv/versions/"],
}
}
Expand Down
Loading