[xray] Extraneous "Worker died" error message

[richardliaw]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
• Ray installed from (source or binary): Pip latest
• Ray version: Pip latest
• Python version: 3.6
• Exact command to reproduce:
  Use a g2.2xlarge with Ray installed on DL AMI.

import ray
import time

@ray.remote
class Test():
    def run(self):
        time.sleep(5)

if __name__ == '__main__':
    ray.init()
    test = Test.remote()
    objs = []
    objs += [test.run.remote()]
    objs += [test.__ray_terminate__.remote()]
    ray.wait(objs)

Describe the problem

export RAY_USE_XRAY=1
python ray/python/ray/tune/examples/hyperband_example.py

Will output the following lines (specifically the middle line):

Saved checkpoint for my_class_6_height=14,width=38 to ObjectID(010000006d44a100c121e0de5a29e386905eec05)
A worker died or was killed while executing task 0000000023bd9eec8029546da425b6ad13982871.
Created LogSyncer for /home/ubuntu/ray_results/hyperband_test/my_class_13_height=34,width=78_2018-07-25_20-43-013okdysr0 ->

[richardliaw]

Also, shutdown callback manager is extraneous output that occurs at the end of run.

[richardliaw]

It would also be great if something was returned from test.__ray_terminate__.remote(), so that the application will know with certainty that the actor is killed.

[robertnishihara]

@richardliaw Did the application run successfully? Or did the application fail?

[richardliaw]

Ran successfully.

[robertnishihara]

Assigning myself to review.

[robertnishihara]

Fixed by #2935.