Reload failed actors from checkpoint

[stephanie-wang]

Describe the problem

Ability to restart a failed actor from its most recent checkpoint and replay all methods since then.

[raulchen]

I've already implemented this internally, mostly based on this design. I can move the code to github soon. Hope you didn't spend much time on this. Sorry for not informing you beforehand.

[stephanie-wang]

I see, @raulchen, can you open a PR ASAP? @ujvl was going to implement this in the backend, but maybe he can review instead.

[raulchen]

@stephanie-wang sure, I'll start working on this today. Apologize for the inconvenience.

[stephanie-wang]

No problem, thanks!

By the way, do you also have an implementation for reloading from a checkpoint without replaying the most recent tasks? If yes, it would be great if you could open a PR for that as well. Thank you!

[raulchen]

reloading from a checkpoint without replaying the most recent tasks

With my current API, this can be done by making should_checkpoint return True for every task, but don't really save the checkpoint to external storage in save_checkpoint. However, this includes small overhead of saving actor's state to GCS for each task.

[raulchen]

BTW, I didn't implement checkpoint storage as @ujvl wanted to do in #3825. But I agree that could be useful for some users.

[ujvl]

That's fine we can implement other storage options in another PR.

With my current API, this can be done by making should_checkpoint return True for every task, but don't really save the checkpoint to external storage in save_checkpoint.

Just to clarify, you mean save_checkpoint would still save checkpoint data on some calls, but would save actor state every call, right? I think there's probably a way to do it without that overhead but it might be better to discuss that after looking at your PR.

[raulchen]

Just to clarify, you mean save_checkpoint would still save checkpoint data on some calls, but would save actor state every call, right?

@ujvl yeah, that's right.

[stephanie-wang]

I see, I think it would be great to have an API explicitly for reloading an actor from a checkpoint without replay. Also, it would be useful to be able to create a new actor from an old actor's checkpoint.

[raulchen]

I see, I think it would be great to have an API explicitly for reloading an actor from a checkpoint without replay. Also, it would be useful to be able to create a new actor from an old actor's checkpoint.

Agreed. I have thought about how to do that though.

[stephanie-wang]

From a backend perspective, it seems like creating a new actor from an old actor's checkpoint would be pretty easy. I think it could actually be implemented entirely in the Python/Java client.

It's not entirely clear to me how we can natively support reloading an existing actor without replay. If the actor only has one caller, then it seems pretty easy to do, but not so sure about multiple callers. One proposal is that if an actor task gets resubmitted, then we just pretend that the task has been executed (e.g., by treating its dummy object return values as local), so that the next task can run immediately.

Regardless, I think it's best that we start with @raulchen's suggestion of doing a mock checkpoint on every call so that we implement and test the API, but it would be nice to remove the per-task overhead in the future.