Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce file_mounts_sync_continuously cluster option #9544

Merged
merged 13 commits into from
Jul 28, 2020
Merged

Introduce file_mounts_sync_continuously cluster option #9544

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
93ed55d
Separate out file_mounts contents hashing into its own separate hash
alanwguo Jul 16, 2020
26d186e
add test and default value for file_mounts_sync_continuously
alanwguo Jul 17, 2020
6c0ca89
format code
alanwguo Jul 17, 2020
59052a3
Update comments
alanwguo Jul 20, 2020
0d6ebd1
Add param to skip setup commands when only file_mounts content change…
alanwguo Jul 22, 2020
094262c
Refactor so that runtime_hash retains previous behavior
alanwguo Jul 22, 2020
d7f6461
fix issue with hashing a hash
alanwguo Jul 22, 2020
e0e1080
Merge branch 'master' into file-mounts-continous
alanwguo Jul 22, 2020
37dd9b6
fix bug where trying to set contents hash when it wasn't generated
alanwguo Jul 22, 2020
3c6f798
Merge branch 'master' into file-mounts-continous
alanwguo Jul 22, 2020
415a32e
Fix lint error
alanwguo Jul 23, 2020
108059d
clear out provider between tests to get rid of flakyness
alanwguo Jul 23, 2020
ca7b27d
reduce chance of race condition from node_launcher launching a node i…
alanwguo Jul 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update comments
  • Loading branch information
@alanwguo
alanwguo committed Jul 20, 2020
commit 59052a36fece464df04bad08dec7a388402e1b69
7 changes: 6 additions & 1 deletion python/ray/autoscaler/updater.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,11 @@ def do_update(self):

node_tags = self.provider.node_tags(self.node_id)
logger.debug("Node tags: {}".format(str(node_tags)))
# file_mounts_content_hash will only change whenever the user restarts
# or updates their cluster with `get_or_create_head_node` unless
# continuous mode is on. In that case, the file_mounts_content_hash
# will be recalculated at a regular interval while the cluster is
# running.
if node_tags.get(
TAG_RAY_RUNTIME_CONFIG) == self.runtime_hash and node_tags.get(
TAG_RAY_FILE_MOUNTS_CONTENTS
Expand All @@ -149,7 +154,7 @@ def do_update(self):
self.node_id, {TAG_RAY_NODE_STATUS: STATUS_SYNCING_FILES})
self.sync_file_mounts(self.rsync_up)

# Only run init/setup commands when runtime config has changed
# Only run init/setup commands when runtime config has changed.
if node_tags.get(TAG_RAY_RUNTIME_CONFIG) != self.runtime_hash:
# Run init commands
self.provider.set_node_tags(
Expand Down
2 changes: 1 addition & 1 deletion python/ray/autoscaler/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ def add_hash_of_file(fpath):
conf_str = (json.dumps(file_mounts, sort_keys=True).encode("utf-8") +
json.dumps(extra_objs, sort_keys=True).encode("utf-8"))

# Only hash the files once unless use_chaced_contents_hash is false.
# Only hash the files once, unless use_cached_contents_hash is false.
if not use_cached_contents_hash or conf_str not in _hash_cache:
config_hasher.update(conf_str)
for local_path in sorted(file_mounts.values()):
Expand Down