Actor reconstruction doesn't work when the actor process is killed.

[raulchen]

I'm currently trying to implement actor reconstruction for xray, and did some research on how actor reconstruction works in legacy ray.

In the below script, when we try to get a missing result of a actor task, and the actor process is dead, I'd expect Ray to replay all tasks (including the actor creation task) and reconstruct the result.

import ray
import os
import signal
import time
import pyarrow.plasma as plasma

@ray.remote(checkpoint_interval=-1)
class MyActor(object):

    def __init__(self):
        print("Actor created" + str(os.getpid()))
        self.value = 0

    def increase(self):
        self.value += 1
        return self.value

    def get_pid(self):
        return os.getpid()

ray.init(use_raylet=False, num_workers=0)
actor = MyActor.remote()
ids = []
for _ in range(5):
    id = actor.increase.remote()
    ids.append(id)
    ray.get(id)

# Kill the actor process
id = actor.get_pid.remote()
ids.append(id)
pid = ray.get(id)
time.sleep(2)
print("killing %s" % pid)
os.kill(pid, signal.SIGKILL)

# Delete the previous results from object store.
ray.worker.global_worker.plasma_client.delete(
    [plasma.ObjectID(id.id()) for id in ids]
)
# Flush object store cache.
for _ in range(100):
    ray.put(1)
time.sleep(2)

print(ids)
print(ray.get(ids[-1]))

However, local scheduler fails with this error:

F0913 20:57:53.068881 2784486272 local_scheduler_algorithm.cc:1589]  Check failed: algorithm_state->local_objects.count(object_id) == 0 fce1281dd6a115e7b90d55d875e62fee930e3916
*** Check failure stack trace: ***
    @        0x101e7c62a  google::LogMessage::Fail()
    @        0x101e7a44e  google::LogMessage::SendToLog()
    @        0x101e7b2cf  google::LogMessage::Flush()
    @        0x101e7b109  google::LogMessage::~LogMessage()
    @        0x101e7b3c5  google::LogMessage::~LogMessage()
    @        0x101e6632d  ray::RayLog::~RayLog()
    @        0x101e22933  handle_object_available()
    @        0x101e0a78b  process_plasma_notification()
    @        0x101e41221  aeProcessEvents
    @        0x101e4150b  aeMain
    @        0x101e12bf7  start_server()
    @        0x101e136c2  main

[raulchen]

I'm still confused about the behavior of actor fault tolerance in legacy ray.
Ideally, I think the behavior could be like this:

1. When an actor accidentally dies (either when the process is killed, or when the whole node is down), ray should resubmit the actor creation task and reload state from a checkpoint (if there's one).
2. If a client ask for an object whose creating task is before the checkpoint, the get will fail. If an object after the checkpoint is required, tasks after the checkpoint will be replayed to reconstruct the object.

@robertnishihara @stephanie-wang @atumanov what do you think about this?

[ericl]

This is actually a feature -- if an actor is killed or otherwise exits with nonzero (iirc) exit code, it is considered an application level error. This prevents runaway re-execution of bad code, and also allows users to manually kill trials in tune.

[raulchen]

@ericl thanks for explaining.

if an actor is killed or otherwise exits with nonzero (iirc) exit code, it is considered an application level error

I think this doesn't always indicate an application level error. For example, if one actor dies because the node goes down, users would want Ray to automatically restart the actor, otherwise they have to implement monitoring actor health in their application code.

I think it makes sense to provide an option to let users decide whether an actor should be automatically restarted.

[ericl]

Perhaps we can choose a subset of the exit codes to treat as application level? For example, SIGINT=130.

…

On Thu, Sep 13, 2018, 7:51 PM Hao Chen ***@***.***> wrote: @ericl <https://github.com/ericl> thanks for explaining. if an actor is killed or otherwise exits with nonzero (iirc) exit code, it is considered an application level error I think this doesn't always indicate an application level error. For example, if one actor dies because the node goes down, users would want Ray to automatically restart the actor, otherwise they have to implement monitoring actor health in their application code. I think it makes sense to provide an option to let users decide whether an actor should be automatically restarted. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2868 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA6Sg-PaxKIYg1fKcc0tB8PloESIrfEks5uaxnFgaJpZM4WnOWR> .

[raulchen]

@ericl that might be a good idea, but I'm not sure which exits codes belong to application level and which don't.
btw, how do you manually kill trials today?

[ericl]

Today, we tell the user to kill the pid of the trial actor out of band if they want to live interrupt a trial.

I agree making auto-restart an option would be best. I think whether to reconstruct actors at all is intended to be an option (and defaults to false in xray). Tune can disable it for its actors then.

[robertnishihara]

@raulchen This feature needs more design. I think we should enumerate the scenarios that we're interested in and see what the ideal behaviors are.

The current behavior is the following.

1. An actor process dies, but the node does not die. This is currently treated as an application level failure and the actor method that failed (as well as future actor methods) are treated as if they had raised an exception.
2. A node (and all the actors on it) dies. With Legacy Ray, we attempt to reconstruct the actor and rerun the tasks that were executed on it (potentially loading from a checkpoint). With Xray, we currently raise an exception whenever someone tries to get the result of an actor method that can't be invoked because the actor is dead.

Some relevant scenarios.

1. You are training a model on a cluster with N machines (each with some GPUs). You create one actor per machine to do data parallel training. One node dies. In this case, the application probably wants to continue training with fewer machines (as opposed to trying to reconstruct the actor since there is nowhere to put it).
2. RLlib has some fairly complex scenarios, e.g., the application creates "manager" actors, which then hierarchically create "worker" actors. If a manager actor dies, we probably want to kill all of its corresponding worker actors as well.
3. In some applications (which ones?), you probably want transparent fault tolerance (that is, you want the computation to proceed exactly as it would have if there had been no failures).

Various things that can be done.

• Recreate actors and rerun methods for them.
• Recreate actors and load state from checkpoints.
• Require user-defined save/restore methods.
• Register a callback that will run when an actor dies.
• Raise exceptions when people try to get objects from dead actors.

This doesn't really cover the full space, so we should probably make a design doc for this and make sure we are covering the full space of important scenarios.

[raulchen]

@robertnishihara thanks for sharing these info!

Let me describe our scenario and requirements first. As you already know, in the streaming system, we have one master actor and a number of worker actors. Data transfer among worker actors is done through Ray Queue, but master and workers will still use remote tasks to send command messages to each other for coordination.

When any of these actors dies, we want it to automatically restart with the same actor id. So other actors can continue sending tasks to it via the old actor handle. Otherwise, if we monitor actor health at application level, we'll need to broadcast the new actor id to all actors. That makes the design complex, and the monitor will still be a single point of failure.

---

Recently, I was also trying to think about possible scenarios of using actors. However, I feel the requirements for actor fault tolerance may vary quite widely. Thus, I think the key problem for now is to figure out what APIs we should provide to let the users have enough flexibility to handle their cases.

I think the options you listed above look good to me. But I still have a couple of questions. I'll put up a doc later and link it here.

[raulchen]

hi @robertnishihara @ericl, I put up an informal doc here https://docs.google.com/document/d/1Nmrljn8WDGyDMlM_308-UsXtdYdJdBFxf4HmH6x-vpk/edit#. please let me know if you have any comments.

Also, it seems that the idea of having an option for whether the actor should be auto-recreated looks reasonable to you as well. I'll try to implement it first. Because it's an urgent requirement for us.