[autoscaler] Change the get behavior of node providers' _get_node

[hartikainen]

What do these changes do?

Change the get behavior of node providers' _get_node such that terminated nodes do not get filtered out.

Related issue number

#4072

@ericl @richardliaw

[hartikainen]

Noting that this still needs to be tested of GCP.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12231/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12229/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12232/
Test FAILed.

[hartikainen]

I don't know if this fully solves the problem. I ran an experiment with config where min_workers=0, max_workers=50, initial_workers=10. Here are some issues:

1. Ray attach hangs

• ray attach /home/kristian/github/haarnoja/softqlearning-private/config/ray-autoscaler-gce.yaml --cluster-name=20190222t142143-109935-halfcheetah-v2 --tmux
• This might be due to the recent stdout changes and unrelated to our changes.

2. I don't think the initial_workers ever took effect. My cluster never got scaled beyond 8 nodes.
3. I manually preempted one of my nodes. I think it somehow messed up the scaling since my node count constantly fluctuates between 7 and 8 nodes. Looking at the monitor logs I see:

2019-02-22 23:34:43,996 INFO autoscaler.py:610 -- LoadMetrics: NodeIdleSeconds=Min=0 Mean=156 Max=1409, NumNodesConnected=9, NumNodesUsed=8.0, ResourceUsage=64.0/72.0 b'CPU', 0.0/0.0 b'GPU', TimeSinceLastHeartbeat=Min=0 Mean=151 Max=1503

• The TimeSinceLastHeartbeat Max keeps increasing. I guess this is due to the preempted node. Is this expected?
• I don't see anything obviously wrong in the logs. I wonder why the scaling up doesn't happen since I still have 156 trials pending and max_workers would allow another 42 nodes to be launched.

[ericl]

Hm. One thing to try is to completely disable the caching and see if that matters.

For * TimeSinceLastHeartbeat Max keeps increasing, I think this is expected.

Can you post your logs? It probably won't scale up if your utilization isn't high enough.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12246/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12247/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12248/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12249/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12272/
Test FAILed.