Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TaskCancellation #7669

Merged
merged 50 commits into from
Apr 25, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
169c540
Smol comment
ijrsvt Mar 19, 2020
682c5b5
Merge branch 'master' into TaskCancellation
ijrsvt Mar 25, 2020
2d020ba
WIP, not passing ray.init
ijrsvt Mar 25, 2020
1958e05
Fixed small problem
ijrsvt Mar 25, 2020
4fdeb5a
wip
ijrsvt Mar 31, 2020
40b2bb5
Pseudo interrupt things
ijrsvt Mar 31, 2020
d1295c3
Basic prototype operational
ijrsvt Mar 31, 2020
269a3b1
Merge branch 'master' of github.com:ijrsvt/ray into TaskCancellation
ijrsvt Apr 1, 2020
028d9f7
correct proc title
ijrsvt Apr 2, 2020
a4b58e5
Mostly done
ijrsvt Apr 7, 2020
33ad6a1
Cleanup
ijrsvt Apr 7, 2020
4f7eec7
cleaner raylet error
ijrsvt Apr 7, 2020
cc3ca28
Cleaning up a few loose ends
ijrsvt Apr 7, 2020
bd47066
Fixing Race Conds
ijrsvt Apr 7, 2020
c0b5ab4
Prelim testing
ijrsvt Apr 7, 2020
58c8bed
Fixing comments and adding second_check for kill
ijrsvt Apr 8, 2020
bae435f
Working_new_impl
ijrsvt Apr 9, 2020
9ab039d
demo_ready
ijrsvt Apr 9, 2020
d85496d
Fixing my english
ijrsvt Apr 10, 2020
d0ba816
Merge branch 'master' into TaskCancellation
ijrsvt Apr 10, 2020
652a0fe
Fixing a few problems
ijrsvt Apr 10, 2020
daac610
Small problems
ijrsvt Apr 10, 2020
b050b28
Cleaning up
ijrsvt Apr 10, 2020
b0457a3
Response to changes
ijrsvt Apr 15, 2020
18b3dbc
Fixing error passing
ijrsvt Apr 15, 2020
b813faf
Merge branch 'master' into TaskCancellation
ijrsvt Apr 15, 2020
112d7d8
Merged to master
ijrsvt Apr 15, 2020
ff8bbd3
fixing lock
ijrsvt Apr 15, 2020
af35898
Cleaning up print statements
ijrsvt Apr 15, 2020
b015c51
Format
ijrsvt Apr 15, 2020
616f487
Fixing Unit test build failure
ijrsvt Apr 16, 2020
2361273
mock_worker fix
ijrsvt Apr 16, 2020
9dba915
java_fix
ijrsvt Apr 16, 2020
9a43056
Canel
ijrsvt Apr 16, 2020
68a6458
Switching to Cancel
ijrsvt Apr 17, 2020
46545e1
Responding to Review
ijrsvt Apr 21, 2020
7308225
FixFormatting
ijrsvt Apr 21, 2020
1f95492
Merge branch 'master' into TaskCancellation
ijrsvt Apr 21, 2020
9a0d120
Lease cancellation
ijrsvt Apr 22, 2020
82a6248
FInal comments?
ijrsvt Apr 22, 2020
3270f92
Moving exist check to CoreWorker
ijrsvt Apr 23, 2020
794f146
Fix Actor Transport Test
ijrsvt Apr 23, 2020
e43ea33
Fixing task manager test
ijrsvt Apr 23, 2020
9beea80
chaning clock repr
ijrsvt Apr 23, 2020
2a789f5
Fix build
ijrsvt Apr 24, 2020
8c75a83
fix white space
ijrsvt Apr 24, 2020
8f7bdfe
lint fix
ijrsvt Apr 24, 2020
6f1ef56
Updating to medium size
ijrsvt Apr 24, 2020
f7fb69f
Fixing Java test compilation issue
ijrsvt Apr 25, 2020
e8cd360
lengthen bad timeouts
ijrsvt Apr 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Switching to Cancel
  • Loading branch information
ijrsvt committed Apr 17, 2020
commit 68a6458f0b6f1a62cd89d8c7b0b23cb8b99c16b5
9 changes: 6 additions & 3 deletions python/ray/_raylet.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -975,10 +975,13 @@ cdef class CoreWorker:
def kill_task(self, ObjectID object_id, c_bool force_kill):
cdef:
CObjectID c_object_id = object_id.native()
CRayStatus status = CRayStatus.OK()

with nogil:
check_status(CCoreWorkerProcess.GetCoreWorker().KillTask(
c_object_id, force_kill))
status = CCoreWorkerProcess.GetCoreWorker().KillTask(
c_object_id, force_kill)

if not status.ok():
raise ValueError(status.message().decode())

def resource_ids(self):
cdef:
Expand Down
8 changes: 8 additions & 0 deletions python/ray/tests/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -414,3 +414,11 @@ py_test(
tags = ["exclusive"],
deps = ["//:ray_lib"],
)

py_test(
name = "test_cancel",
size = "small",
srcs = ["test_cancel.py],
tags = ["exclusive"],
deps = ["//:ray_lib"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@


@pytest.mark.parametrize("use_force", [True, False])
def test_kill_chain(ray_start_regular, use_force):
def test_cancel_chain(ray_start_regular, use_force):
"""A helper method for chain of events tests"""
signaler = SignalActor.remote()

Expand Down Expand Up @@ -50,7 +50,7 @@ def wait_for(t):


@pytest.mark.parametrize("use_force", [True, False])
def test_kill_multiple_dependents(ray_start_regular, use_force):
def test_cancel_multiple_dependents(ray_start_regular, use_force):
"""A helper method for multiple waiters on events tests"""
signaler = SignalActor.remote()

Expand Down Expand Up @@ -87,7 +87,7 @@ def wait_for(t):


@pytest.mark.parametrize("use_force", [True, False])
def test_single_cpu_kill(shutdown_only, use_force):
def test_single_cpu_cancel(shutdown_only, use_force):
ray.init(num_cpus=1)
signaler = SignalActor.remote()

Expand Down Expand Up @@ -162,25 +162,25 @@ def infinite_sleep(y):

sleep_or_no = [random.randint(0, 1) for _ in range(100)]
tasks = [infinite_sleep.remote(i) for i in sleep_or_no]
killed = set()
cancelled = set()
for t in tasks:
if random.random() > 0.5:
ray.cancel(t, use_force)
killed.add(t)
cancelled.add(t)

ray.cancel(first, use_force)
killed.add(first)
cancelled.add(first)

for done in killed:
for done in cancelled:
with pytest.raises((RayTaskError, RayCancellationError)):
ray.get(done, 10)

for indx in range(len(tasks)):
t = tasks[indx]
if sleep_or_no[indx]:
ray.cancel(t, use_force)
killed.add(t)
if t in killed:
cancelled.add(t)
if t in cancelled:
with pytest.raises((RayTaskError, RayCancellationError)):
ray.get(t, 10)
else:
Expand Down
5 changes: 3 additions & 2 deletions python/ray/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -1663,8 +1663,9 @@ def cancel(object_id, force=False):

if isinstance(object_id, ray.ObjectID) and object_id.task_id().actor_id(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to have this logic in the core worker instead of in Python, so that it's easier to extend it across other languages. In the core worker, you can also check if the actor ID is nil instead of checking whether it's in ray.actors(). I'm actually not sure what the semantics of ray.actors() is; checking the ID seems safer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay! I'll move that Actor logic to core worker. Should I also check isinstance in core_worker?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

).hex() not in ray.actors().keys():
worker.core_worker.kill_task(object_id, force)
return
if len(ray.wait([object_id], timeout=0)[1]) != 1:
return
return worker.core_worker.kill_task(object_id, force)
raise ValueError("ray.cancel() only supported for non-actor object IDs. "
"Got: {}.".format(type(object_id)))

Expand Down
7 changes: 5 additions & 2 deletions src/ray/core_worker/core_worker.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1200,11 +1200,14 @@ Status CoreWorker::SubmitActorTask(const ActorID &actor_id, const RayFunction &f
}

Status CoreWorker::KillTask(const ObjectID &object_id, bool force_kill) {
if (!object_id.CreatedByTask()) {
return Status::Invalid("Not created by a task.");
}
auto task_spec = task_manager_->GetTaskSpec(object_id.TaskId());
if (task_spec.has_value() && !task_spec.value().IsActorCreationTask()) {
RAY_RETURN_NOT_OK(direct_task_submitter_->KillTask(task_spec.value(), force_kill));
return direct_task_submitter_->KillTask(task_spec.value(), force_kill);
}
return Status::OK();
return Status::Invalid("Task is not locally submitted.");
}

Status CoreWorker::KillActor(const ActorID &actor_id, bool force_kill,
Expand Down