Auto-scale ray clusters based on GCS load metrics #1348

ericl · 2017-12-19T08:12:51Z

What do these changes do?

This adds (experimental) auto-scaling support for Ray clusters based on GCS load metrics. The auto-scaling algorithm is as follows:

Based on current (instantaneous) load information, we compute the approximate number of "used workers". This is based on the bottleneck resource, e.g. if 8/8 GPUs are used in a 8-node cluster but all the CPUs are idle, the number of used nodes is still counted as 8. This number can also be fractional.
We scale that number by 1 / target_utilization_fraction and round up to determine the target cluster size (subject to the max_workers constraint). The autoscaler control loop takes care of launching new nodes until the target cluster size is met.
When a node is idle for more than idle_timeout_minutes, we remove it from the cluster if that would not drop the cluster size below min_workers.

Note that we'll need to update the wheel in the example yaml file after this PR is merged.

AmplabJenkins · 2017-12-19T09:01:39Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-19T09:01:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2849/
Test PASSed.

robertnishihara · 2017-12-20T07:47:26Z

Just tried this with the default config

ray create_or_update ray/python/ray/autoscaler/aws/example.yaml

It successfully started a node, but then it printed

NodeUpdater: Applied config a6565145497ee0bcb66896453b9a61a488f47020 to node i-0d4016ff8eecf92ff
Head node up-to-date, IP address is: None
To monitor auto-scaling activity, you can run:

  ssh -i /Users/rkn/.ssh/ray-autoscaler_1.pem ubuntu@None 'tail -f /tmp/raylogs/monitor-*'

for some reason the IP address is None.

robertnishihara · 2017-12-20T22:21:02Z

python/ray/autoscaler/autoscaler.py

@@ -64,6 +74,93 @@
 # Max number of nodes to launch at a time.
 MAX_CONCURRENT_LAUNCHES = 10

+# Print debug string once every this many seconds
+DEBUG_INTERVAL_S = 5


These constants MAX_NUM_FAILURES, MAX_CONCURRENT_LAUNCHES , and DEBUG_INTERVAL_S should go in https://github.com/ray-project/ray/blob/master/src/common/state/ray_config.h. All constants are there to simplify debugging (these kinds of constants often need to be changed during debugging) so that you don't have to track down the constants throughout the whole codebase (these are not considered exposed to users).

If you want I can make this change.

How do you access them from Python? I'd prefer to keep Python constants in .py files, since it avoids recompiles.

You can see how it's done in #1192, it actually involves a number of files to expose it to Python.

Good point about recompilation. Maybe we should have two files. One for C constants and one for Python constants. What do you think about creating ray/python/ray/constants.py?

robertnishihara · 2017-12-21T00:25:30Z

I'm getting timeout errors on the current master

$ ray create_or_update ~/Workspace/ray/python/ray/autoscaler/aws/example.yaml 
Role not specified for head node, using arn:aws:iam::339530224232:instance-profile/ray-autoscaler
KeyName not specified for nodes, using ray-autoscaler_1
SubnetId not specified for head node, using subnet-8e489ed7 in us-west-2c
SubnetId not specified for workers, using subnet-8e489ed7 in us-west-2c
SecurityGroupIds not specified for head node, using ray-autoscaler-default
SecurityGroupIds not specified for workers, using ray-autoscaler-default
Launching new head node...
Updating files on head node...
NodeUpdater: Updating i-01596b463d5700cc1 to 8f0512ae89aaeb7304fb593120a29f313cce7691, logging to (console)
NodeUpdater: Waiting for IP of i-01596b463d5700cc1...
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: 
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: 
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Waiting for SSH to i-01596b463d5700cc1...
NodeUpdater: SSH not up, retrying: Command '['ssh', '-o', 'ConnectTimeout=5s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'uptime']' returned non-zero exit status 255.
NodeUpdater: Syncing /Users/rkn/.ssh/ray-autoscaler_1.pem to ~/ray_bootstrap_key.pem...
ssh: connect to host 35.162.178.19 port 22: Operation timed out
NodeUpdater: Error updating Command '['ssh', '-o', 'ConnectTimeout=60s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'mkdir -p ~']' returned non-zero exit status 255., see (console) for remote logs
Process NodeUpdaterProcess-1:
Traceback (most recent call last):
  File "/Users/rkn/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/Users/rkn/Workspace/ray/python/ray/autoscaler/updater.py", line 67, in run
    raise e
  File "/Users/rkn/Workspace/ray/python/ray/autoscaler/updater.py", line 54, in run
    self.do_update()
  File "/Users/rkn/Workspace/ray/python/ray/autoscaler/updater.py", line 132, in do_update
    "mkdir -p {}".format(os.path.dirname(remote_path)))
  File "/Users/rkn/Workspace/ray/python/ray/autoscaler/updater.py", line 158, in ssh_cmd
    ], stdout=redirect or self.stdout, stderr=redirect or self.stderr)
  File "/Users/rkn/anaconda3/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-o', 'ConnectTimeout=60s', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/rkn/.ssh/ray-autoscaler_1.pem', 'ubuntu@35.162.178.19', 'mkdir -p ~']' returned non-zero exit status 255.
Error: updating 35.162.178.19 failed

It seems a bit opaque to me. Any idea about this?

ericl · 2017-12-21T01:38:27Z

Hmm, when you run the SSH command manually does it work? It looks like it can't reach the instance for some reason.

robertnishihara · 2017-12-21T03:02:00Z

@ericl I just tried again and this time it succeeded. I'll let you know if I can reproduce the original problem.

robertnishihara · 2017-12-21T07:04:11Z

python/ray/autoscaler/autoscaler.py

@@ -4,8 +4,10 @@

 import json
 import hashlib
+import math


shall we just use numpy everywhere?

robertnishihara · 2017-12-28T23:41:35Z

Oh, I just noticed you removed the line

echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc

adding that back in along with

source ~/.bashrc

should fix it. I'm trying this out and will push the diff if it works

ericl · 2017-12-29T00:24:46Z

Oh ic, I removed that since I thought it was unneeded. Where does that path entry come from on login if not bashrc?

AmplabJenkins · 2017-12-29T00:36:14Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-29T00:36:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3001/
Test PASSed.

robertnishihara · 2017-12-29T04:24:25Z

I see, I'll remove the source thing then. The line echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc is usually run by the Anaconda installation script (it prompts you when run interactively and asks if you want to add it to your PATH). However, when running it non-interactively, it just doesn't happen.

AmplabJenkins · 2017-12-29T05:16:56Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-29T05:16:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3003/
Test PASSed.

robertnishihara · 2017-12-29T05:21:09Z

python/ray/autoscaler/aws/development-example.yaml

    # Install basics.
    - sudo apt-get update
    - sudo apt-get install -y cmake pkg-config build-essential autoconf curl libtool unzip python
+    # TODO(ekl): are these commands idempotent?


they are definitely not idempotent

I added some || trues which should fix it for now.

robertnishihara · 2017-12-29T05:33:22Z

@ericl Just ran out of disk space on the cluster while testing this. Is there a way to increase disk space through the yaml file?

robertnishihara · 2017-12-29T05:58:40Z

python/ray/services.py

@@ -850,6 +850,7 @@ def start_worker(node_ip_address, object_store_name, object_store_manager_name,
            default.
    """
    command = [sys.executable,
+               "-u",


great idea :)

robertnishihara · 2017-12-29T06:01:42Z

python/ray/autoscaler/autoscaler.py

+
+# The autoscaler will attempt to restart Ray on nodes it hasn't heard from
+# in more than this interval.
+HEARTBEAT_TIMEOUT_S = 30


I still think it'd be a good idea to have a separate file with all of the constants. That way people don't have to search through all the files to find all the constants they need to change while debugging. What do you think about putting these in a separate ray_constants.py file?

robertnishihara · 2017-12-29T06:08:40Z

python/ray/autoscaler/autoscaler.py

@@ -123,43 +247,59 @@ def update(self):
                raise e

    def _update(self):
+        # Throttle autoscaling updates to this interval to avoid exceeding
+        # rate limits on API calls.
+        if time.time() - self.last_update_time < self.update_interval_s:


throttling here is a great idea

ericl · 2017-12-29T06:23:31Z

Hmm I think you can attach EBS or something using the node config, but I'm not sure how exactly. It should be in the linked API docs.

AmplabJenkins · 2017-12-30T05:18:55Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-30T05:18:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3014/
Test PASSed.

robertnishihara

Looks good to me. Note there seem to be some linting errors on travis.

AmplabJenkins · 2017-12-30T08:54:04Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-30T08:54:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3021/
Test PASSed.

AmplabJenkins · 2017-12-30T22:55:47Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-30T22:55:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3025/
Test PASSed.

AmplabJenkins · 2017-12-31T07:33:41Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-31T07:33:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3041/
Test PASSed.

ericl added 10 commits December 16, 2017 18:40

wip gc metrics autoscaling

a8eb626

add load metrics debug string

4c09b68

add ray ip

c6a9c2c

wire it end to end

cc1d722

wip dev

6732223

fix bug

419ca59

update

a8240b8

Sun Dec 17 14:19:03 PST 2017

c49a35a

wip

9a69baf

wip

03f47de

robertnishihara reviewed Dec 20, 2017

View reviewed changes

robertnishihara reviewed Dec 21, 2017

View reviewed changes

ericl added 8 commits December 20, 2017 23:40

add update throttling; reorg init commands

97dfa7f

Wed Dec 20 23:44:56 PST 2017

3ccb5fc

Wed Dec 20 23:51:37 PST 2017

d805f90

Wed Dec 20 23:58:18 PST 2017

13436dd

Thu Dec 21 00:03:19 PST 2017

f6d96d1

Thu Dec 21 00:04:33 PST 2017

7fe7ec7

Thu Dec 21 00:05:32 PST 2017

60a487b

Thu Dec 21 00:15:11 PST 2017

68df74a

ericl force-pushed the load-metrics branch from 3910ac5 to b8211b5 Compare December 21, 2017 08:31

Thu Dec 21 00:30:54 PST 2017

ead1984

ericl force-pushed the load-metrics branch 2 times, most recently from fd9e762 to 6172f78 Compare December 21, 2017 08:41

Fix path for development-example.yaml

4280e53

Remove unnecessary line.

1969bcf

robertnishihara reviewed Dec 29, 2017

View reviewed changes

fix idempotent

bef8e29

robertnishihara approved these changes Dec 30, 2017

View reviewed changes

Update autoscaler.py

4093407

fix nondterministic test

b5df753

Update autoscaler.py

3237986

ericl merged commit b6c42f9 into ray-project:master Dec 31, 2017

robertnishihara deleted the load-metrics branch December 31, 2017 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-scale ray clusters based on GCS load metrics #1348

Auto-scale ray clusters based on GCS load metrics #1348

ericl commented Dec 19, 2017 •

edited

Loading

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

robertnishihara commented Dec 20, 2017 •

edited

Loading

robertnishihara Dec 20, 2017

ericl Dec 21, 2017 •

edited

Loading

robertnishihara Dec 21, 2017

robertnishihara commented Dec 21, 2017

ericl commented Dec 21, 2017

robertnishihara commented Dec 21, 2017

robertnishihara Dec 21, 2017

robertnishihara commented Dec 28, 2017 •

edited

Loading

ericl commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

robertnishihara commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

robertnishihara Dec 29, 2017

ericl Dec 30, 2017

robertnishihara commented Dec 29, 2017 •

edited

Loading

robertnishihara Dec 29, 2017

robertnishihara Dec 29, 2017

ericl Dec 30, 2017

robertnishihara Dec 29, 2017

ericl commented Dec 29, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

robertnishihara left a comment

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 31, 2017

AmplabJenkins commented Dec 31, 2017

Auto-scale ray clusters based on GCS load metrics #1348

Auto-scale ray clusters based on GCS load metrics #1348

Conversation

ericl commented Dec 19, 2017 • edited Loading

What do these changes do?

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

robertnishihara commented Dec 20, 2017 • edited Loading

Choose a reason for hiding this comment

ericl Dec 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertnishihara commented Dec 21, 2017

ericl commented Dec 21, 2017

robertnishihara commented Dec 21, 2017

Choose a reason for hiding this comment

robertnishihara commented Dec 28, 2017 • edited Loading

ericl commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

robertnishihara commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

AmplabJenkins commented Dec 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertnishihara commented Dec 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericl commented Dec 29, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

robertnishihara left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 30, 2017

AmplabJenkins commented Dec 31, 2017

AmplabJenkins commented Dec 31, 2017

ericl commented Dec 19, 2017 •

edited

Loading

robertnishihara commented Dec 20, 2017 •

edited

Loading

ericl Dec 21, 2017 •

edited

Loading

robertnishihara commented Dec 28, 2017 •

edited

Loading

robertnishihara commented Dec 29, 2017 •

edited

Loading