TaskCancellation #7669

ijrsvt · 2020-03-20T01:22:36Z

Task Cancellation for locally submitted tasks. Cancellation for remote tasks and the full API will be added in follow-up PRs.

Why are these changes needed?

Related issue number

Closes #854

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested (please justify below)

AmplabJenkins · 2020-03-20T01:30:33Z

Can one of the admins verify this patch?

AmplabJenkins · 2020-03-20T02:12:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23407/
Test FAILed.

AmplabJenkins · 2020-03-25T21:20:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23709/
Test FAILed.

AmplabJenkins · 2020-03-26T00:40:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23717/
Test PASSed.

AmplabJenkins · 2020-03-31T02:57:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23979/
Test PASSed.

AmplabJenkins · 2020-03-31T05:23:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23986/
Test PASSed.

AmplabJenkins · 2020-04-07T04:30:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24331/
Test FAILed.

ijrsvt · 2020-04-07T04:37:30Z

python/ray/worker.py

    worker = ray.worker.global_worker
    worker.check_connected()
-    worker.core_worker.kill_actor(actor._ray_actor_id, False)
+
+    if isinstance(id, ray.actor.ActorHandle):


I'll move this to a new PR later

AmplabJenkins · 2020-04-07T05:39:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24341/
Test FAILed.

src/ray/core_worker/core_worker.cc

AmplabJenkins · 2020-04-07T08:05:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/24344/
Test PASSed.

simon-mo

Awesome work!

python/ray/_raylet.pyx

simon-mo · 2020-04-07T17:38:16Z

src/ray/core_worker/core_worker.cc

+  auto task_id = object_id.TaskId();
+  if (task_manager_->IsTaskPending(task_id)) {
+    auto task_spec = task_manager_->GetTaskSpec(object_id.TaskId());
+    if (!task_spec.IsActorCreationTask())


let's do a RAY_CHECK instead of if statement here because this is definitely checked by python code.

I don't think it actually is.

src/ray/core_worker/transport/direct_task_transport.cc

src/ray/protobuf/core_worker.proto

src/ray/core_worker/core_worker.h

src/ray/core_worker/core_worker.cc

simon-mo · 2020-04-07T17:49:56Z

src/ray/core_worker/core_worker.cc

+      boost::bind(&CoreWorker::TryKillTask, this, task_id, num_tries - 1));
+}
+
+void CoreWorker::HandleKillTask(const rpc::KillTaskRequest &request,


what happen if the task is inside worker's scheduling queue and timeout expired?

For direct tasks, the task is never entered into the scheduling queue, it is immediately executed.

What about for actor tasks? Is that not supported at all?

AmplabJenkins · 2020-04-23T02:30:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25075/
Test PASSed.

AmplabJenkins · 2020-04-23T19:41:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25107/
Test PASSed.

AmplabJenkins · 2020-04-23T22:07:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25108/
Test PASSed.

AmplabJenkins · 2020-04-24T00:38:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25109/
Test PASSed.

AmplabJenkins · 2020-04-24T21:58:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25139/
Test PASSed.

AmplabJenkins · 2020-04-24T23:32:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25142/
Test PASSed.

AmplabJenkins · 2020-04-24T23:51:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25143/
Test FAILed.

AmplabJenkins · 2020-04-25T01:01:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25146/
Test PASSed.

AmplabJenkins · 2020-04-25T06:02:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25151/
Test PASSed.

AmplabJenkins · 2020-04-25T20:59:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/25156/
Test PASSed.

python/ray/worker.py

simon-mo · 2020-05-26T04:24:16Z

Nice catch! I think this is a mistake. Actor handle can only be killed with ray.kill. Feel free to submit a quick doc fix!

…

On Mon, May 25, 2020 at 9:11 PM Mitar ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In python/ray/worker.py <#7669 (comment)>: > worker = ray.worker.global_worker worker.check_connected() worker.core_worker.kill_actor(actor._ray_actor_id, False) +def cancel(object_id, force=False): + """Kill a task forcefully. + + This will interrupt any running tasks on the actor, causing them to fail + immediately. Any atexit handlers installed in the actor will still be run. + + If this actor is reconstructable, it will be attempted to be reconstructed. + + Args: + id (ActorHandle or ObjectID): Handle for the actor to kill or ObjectID + of the task to kill. Why it says that ActorHandle is allowed here, but then it raises an exception if one passes one? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#7669 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBD7A7U2Q6G6IWOFWB3UXDRTM6O3ANCNFSM4LP3W54A> .

mitar · 2020-05-26T04:25:36Z

Not sure why there are two calls here though. It seems pretty easy to expose to the user only one? API proliferation?

ijrsvt · 2020-05-28T03:16:18Z

@mitar We may eventually combine them into one API. The main reason for separating them was that kill-ing a scheduled task semantically doesn't make sense--cancel-ing a task does. Similarly, cancel-ing an actor does not make as much sense as kill-ing an actor.

mitar · 2020-05-28T03:51:36Z

But canceling a running task still kills it, no?

edoakes · 2020-05-28T22:00:05Z

ray.kill kills the target process, ray.cancel doesn't by default

mitar · 2020-05-28T22:33:36Z

Yea, so we could have ray.cancel(id, force) for both actors and tasks, where one would send an interrupt without force to the actor as well?

Anyway, it works for me. I am just surprised with this design decisions. :-) Keep up with good work. :-) Ray is amazing.

Smol comment

169c540

ijrsvt changed the title ~~Local Mode is in C++~~ TaskCancellation Mar 20, 2020

ijrsvt added 2 commits March 25, 2020 12:08

Merge branch 'master' into TaskCancellation

682c5b5

WIP, not passing ray.init

2d020ba

Fixed small problem

1958e05

simon-mo self-assigned this Mar 27, 2020

wip

4fdeb5a

Pseudo interrupt things

40b2bb5

ijrsvt added 6 commits March 30, 2020 23:29

Basic prototype operational

d1295c3

Merge branch 'master' of github.com:ijrsvt/ray into TaskCancellation

269a3b1

correct proc title

028d9f7

Mostly done

a4b58e5

Cleanup

33ad6a1

cleaner raylet error

4f7eec7

ijrsvt marked this pull request as ready for review April 7, 2020 03:23

ijrsvt commented Apr 7, 2020

View reviewed changes

Cleaning up a few loose ends

cc3ca28

ijrsvt added 2 commits April 6, 2020 23:54

Fixing Race Conds

bd47066

Prelim testing

c0b5ab4

ijrsvt commented Apr 7, 2020

View reviewed changes

src/ray/core_worker/core_worker.cc Outdated Show resolved Hide resolved

simon-mo reviewed Apr 7, 2020

View reviewed changes

Moving exist check to CoreWorker

3270f92

Fix Actor Transport Test

794f146

Fixing task manager test

e43ea33

chaning clock repr

9beea80

Fix build

2a789f5

ijrsvt added 2 commits April 24, 2020 15:05

fix white space

8c75a83

lint fix

8f7bdfe

Updating to medium size

6f1ef56

Fixing Java test compilation issue

f7fb69f

lengthen bad timeouts

e8cd360

stephanie-wang merged commit 69ff7e3 into ray-project:master Apr 25, 2020

ijrsvt mentioned this pull request May 19, 2020

[Docs] Add Cancelation to main docs. #8508

Merged

6 tasks

mitar reviewed May 26, 2020

View reviewed changes

python/ray/worker.py Show resolved Hide resolved

ijrsvt deleted the TaskCancellation branch August 26, 2020 00:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TaskCancellation #7669

TaskCancellation #7669

ijrsvt commented Mar 20, 2020 •

edited

Loading

AmplabJenkins commented Mar 20, 2020

AmplabJenkins commented Mar 20, 2020

AmplabJenkins commented Mar 25, 2020

AmplabJenkins commented Mar 26, 2020

AmplabJenkins commented Mar 31, 2020

AmplabJenkins commented Mar 31, 2020

AmplabJenkins commented Apr 7, 2020

ijrsvt Apr 7, 2020 •

edited

Loading

AmplabJenkins commented Apr 7, 2020

AmplabJenkins commented Apr 7, 2020

simon-mo left a comment

simon-mo Apr 7, 2020

ijrsvt Apr 10, 2020

simon-mo Apr 7, 2020

ijrsvt Apr 8, 2020

edoakes Apr 21, 2020

ijrsvt Apr 21, 2020

AmplabJenkins commented Apr 23, 2020

AmplabJenkins commented Apr 23, 2020

AmplabJenkins commented Apr 23, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 25, 2020

AmplabJenkins commented Apr 25, 2020

AmplabJenkins commented Apr 25, 2020

simon-mo commented May 26, 2020 via email

mitar commented May 26, 2020

ijrsvt commented May 28, 2020 •

edited

Loading

mitar commented May 28, 2020

edoakes commented May 28, 2020

mitar commented May 28, 2020

TaskCancellation #7669

TaskCancellation #7669

Conversation

ijrsvt commented Mar 20, 2020 • edited Loading

Why are these changes needed?

Related issue number

Checks

AmplabJenkins commented Mar 20, 2020

AmplabJenkins commented Mar 20, 2020

AmplabJenkins commented Mar 25, 2020

AmplabJenkins commented Mar 26, 2020

AmplabJenkins commented Mar 31, 2020

AmplabJenkins commented Mar 31, 2020

AmplabJenkins commented Apr 7, 2020

ijrsvt Apr 7, 2020 • edited Loading

Choose a reason for hiding this comment

AmplabJenkins commented Apr 7, 2020

AmplabJenkins commented Apr 7, 2020

simon-mo left a comment

Choose a reason for hiding this comment

simon-mo Apr 7, 2020

Choose a reason for hiding this comment

ijrsvt Apr 10, 2020

Choose a reason for hiding this comment

simon-mo Apr 7, 2020

Choose a reason for hiding this comment

ijrsvt Apr 8, 2020

Choose a reason for hiding this comment

edoakes Apr 21, 2020

Choose a reason for hiding this comment

ijrsvt Apr 21, 2020

Choose a reason for hiding this comment

AmplabJenkins commented Apr 23, 2020

AmplabJenkins commented Apr 23, 2020

AmplabJenkins commented Apr 23, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 24, 2020

AmplabJenkins commented Apr 25, 2020

AmplabJenkins commented Apr 25, 2020

AmplabJenkins commented Apr 25, 2020

simon-mo commented May 26, 2020 via email

mitar commented May 26, 2020

ijrsvt commented May 28, 2020 • edited Loading

mitar commented May 28, 2020

edoakes commented May 28, 2020

mitar commented May 28, 2020

ijrsvt commented Mar 20, 2020 •

edited

Loading

ijrsvt Apr 7, 2020 •

edited

Loading

ijrsvt commented May 28, 2020 •

edited

Loading