reconnecting cluster slave nodes to the head node fails

[atumanov]

Steps to reproduce:
Start head node:
scripts/start_ray.sh --num-cpus 100 --num-gpus 0 --num-workers 100 --head
Login to the second (slave) node and start ray, pointing to the head node:
./scripts/start_ray.sh --redis-address <headnode_ip:redis_port>
Then stop Ray on the slave node :
./scripts/stop_ray.sh

And now try to start Ray on the slave node again:

./scripts/start_ray.sh --redis-address <headnode_ip:redis_port>
Waiting for redis server at <headnode_ip:redis_port> to respond...
Using IP address ####### for this node.
Traceback (most recent call last):
  File "/data/atumanov/ray/scripts/start_ray.py", line 109, in <module>
    check_no_existing_redis_clients(node_ip_address, args.redis_address)
  File "/data/atumanov/ray/scripts/start_ray.py", line 34, in check_no_existing_redis_clients
    raise Exception("This Redis instance is already connected to clients with this IP address.")
Exception: This Redis instance is already connected to clients with this IP address.

Takeaway: starting and stopping Ray on slave nodes is not idempotent and it should be.

[atumanov]

Every time I want to restart a slave node with different resource configuration, I have to restart the head node, due to this issue. This is also related to handling node failures. If the node just reboots and attempts to reconnect to the headnode on startup, it should be able to.

[robertnishihara]

Good point, this should probably be handled the same way we handle node failures (e.g., when you run ./scripts/stop_ray.sh it could be handled the same way as any other node failure). If you then rerun ./scripts/start_ray.sh ... then it should reconnect but the local scheduler will have a different DBClientID, so any tasks assigned to the old local scheduler will not be picked up by the new one (and ideally those will already have been cleaned up by the monitor process).

[robertnishihara]

I haven't tested it out, but this should be addressed in #347.

[atumanov]

still not working for me as of master commit 227c916

Steps to reproduce:

1. start head node:
   scripts/start_ray.sh --head --num-workers 144 --num-cpus 144
2. start slave node (on a different physical machine):
   ./scripts/start_ray.sh --redis-address IP:PORT --num-workers 0 --num-cpus 0
3. stop Ray on slave node:
   ./scripts/stop_ray.sh
4. sleep for minimum 15 seconds:
   sleep 15
5. Attempt to start ray on slave node again:
   ./scripts/start_ray.sh --redis-address IP:PORT --num-workers 0 --num-cpus 0

Error message:

Waiting for redis server at IP:PORT to respond...
Using IP address LOCAL_IP for this node.
Traceback (most recent call last):
  File "/Users/atumanov/devrepo/github/ray-mydev/scripts/start_ray.py", line 132, in <module>
    check_no_existing_redis_clients(node_ip_address, args.redis_address)
  File "/Users/atumanov/devrepo/github/ray-mydev/scripts/start_ray.py", line 52, in check_no_existing_redis_clients
    raise Exception("This Redis instance is already connected to clients "
Exception: This Redis instance is already connected to clients with this IP address.

[robertnishihara]

I just tried this out and it worked for me. Any additional tips for reproducing the problem?

Note, I do see the problem if I call start_ray.sh too soon after stop_ray.sh.

[atumanov]

Yes, I can consistently reproduce this on a master if I connect and then disconnect a driver on the local node after step 2 and before step 3 above:

python -c 'import ray; ray.init(redis_address="IP:PORT")'

Simplified set of steps to reproduce on master as of commit 1e84747

1. start head node scripts/start_ray.sh --head
2. start local node: ./scripts/start_ray.sh --redis-address IP:PORT
3. connect and disconnect a driver on the local node : python -c 'import ray; ray.init(redis_address="IP:PORT")'
4. stop ray on local node : scripts/stop_ray.sh
5. sleep 15 (tried sleeping minutes)
6. attempt to start ray on local node : ./scripts/start_ray.sh --redis-address IP:PORT

[stephanie-wang]

I'm not able to reproduce this with the additional step, @atumanov. Are you sure step 6 is not being run on the head node?

There does seem to be a bug in the Python worker code that prevents a driver from reconnecting to a node that was stopped and then restarted. The steps are the same as above. After the local node is restarted, the command python -c 'import ray; ray.init(redis_address="IP:PORT")' will fail with an error message like:

[ERROR] (/home/ubuntu/ray/src/common/io.cc:121: errno: Connection refused) Connection to socket failed for pathname /tmp/plasma_store84702062.

I'm fixing this now, along with the ability to immediately remove clients that are killed with the stop_ray.sh script (rather than waiting for the full heartbeat timeout).

[robertnishihara]

Closing because I think this has been resolved. If anyone comes across this problem again, please re-open the issue.

[danielsuo]

I was able to replicate this issue on eadb998, Python 3.6.3 MacOS locally, Python 3.5.2 Ubuntu 16.04 remotely.

Same repro steps as @atumanov (using ray start and ray stop with appropriate flags). The only way I can start ray on the local node again is if I restart the head node. I must be missing something really basic, but ray stop isn't clearing the local node's ip from the head node's redis.