[ID Refactor] Refactor ActorID, TaskID and ObjectID #5286

jovany-wang · 2019-07-26T15:57:45Z

IDs format:

JobID is generated by GCS.
Embed JobID to ActorID, and ActorID has 4 random bytes.
Embed ActorID to TaskID, and TaskID has 6 random bytes.
ObjectID: The flags bytes is used to store some flags of this object, like is_task, object_type and transport_type.

is_task indicates whether this object is created from a task. It use 1 bit to store.
object_type indicates the type of this object, a put object or a return object. It only use 1 bit. 0(binary) shows it's a put object and 1(binary) for return object.
transport_type indicates the type of the transport that transfer this object. It uses 3 bits, so we can support 8 different types. Currently, 000(binary) represents for a STANDARD transport and 001(binary) represents for a DIRECT_ACTOR_CALL transport.
There're 11 bits unused and we can extends other flags in the future.

The object index bytes store the index of this object, put index or return index.

Related issue number

#5285

Linter

I've run scripts/format.sh to lint the changes in this PR.

AmplabJenkins · 2019-07-26T16:03:02Z

Can one of the admins verify this patch?

AmplabJenkins · 2019-07-26T18:53:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15682/
Test PASSed.

stephanie-wang

Thanks, I left a bunch of comments, but they're mostly minor!

.travis.yml

src/ray/common/id.h

src/ray/common/id.cc

src/ray/common/common_protocol.h

zhijunfu

Thanks! LGTM overall, I left a few minor comments.

java/api/src/main/java/org/ray/api/TransportType.java

java/api/src/main/java/org/ray/api/id/ActorId.java

java/api/src/main/java/org/ray/api/id/TaskId.java

src/ray/common/id.h

java/api/src/main/java/org/ray/api/id/ObjectId.java

AmplabJenkins · 2019-07-27T05:11:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15700/
Test PASSed.

AmplabJenkins · 2019-07-27T20:12:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15709/
Test FAILed.

AmplabJenkins · 2019-07-28T11:22:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15727/
Test PASSed.

AmplabJenkins · 2019-07-28T12:18:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15728/
Test PASSed.

jovany-wang · 2019-07-29T08:20:31Z

@zhijunfu @stephanie-wang
Thanks. All comments of the first round review were addressed or replied.

.travis.yml

src/ray/common/id.cc

src/ray/common/id.h

src/ray/common/id.cc

AmplabJenkins · 2019-07-29T10:52:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15742/
Test PASSed.

AmplabJenkins · 2019-07-29T11:05:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15743/
Test PASSed.

AmplabJenkins · 2019-07-29T18:48:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15747/
Test FAILed.

AmplabJenkins · 2019-07-29T20:11:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15746/
Test FAILed.

zhijunfu

Thanks. I don't have further comments.

Left a TODO comment WIP for ObjectID ADD test Fix Add java part Fix Java test Fix Refine test. Enable test in CI

Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>

AmplabJenkins · 2019-08-04T03:04:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15948/
Test PASSed.

java/api/src/main/java/org/ray/api/id/ActorId.java

java/api/src/main/java/org/ray/api/id/ObjectId.java

java/api/src/main/java/org/ray/api/id/TaskId.java

java/runtime/src/main/java/org/ray/runtime/util/IdUtil.java

python/ray/_raylet.pyx

src/ray/common/id.h

src/ray/common/id.cc

AmplabJenkins · 2019-08-04T05:02:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15946/
Test PASSed.

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

AmplabJenkins · 2019-08-04T08:28:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15949/
Test PASSed.

AmplabJenkins · 2019-08-04T08:53:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15950/
Test PASSed.

AmplabJenkins · 2019-08-05T07:18:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15955/
Test PASSed.

AmplabJenkins · 2019-08-05T15:31:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15979/
Test PASSed.

AmplabJenkins · 2019-08-05T15:40:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15980/
Test PASSed.

AmplabJenkins · 2019-08-05T16:56:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15982/
Test PASSed.

AmplabJenkins · 2019-08-06T13:29:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16029/
Test PASSed.

AmplabJenkins · 2019-08-06T16:14:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16032/
Test PASSed.

AmplabJenkins · 2019-08-06T16:22:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16031/
Test PASSed.

AmplabJenkins · 2019-08-06T18:43:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16039/
Test FAILed.

AmplabJenkins · 2019-08-06T19:50:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16044/
Test PASSed.

jovany-wang · 2019-08-07T02:39:45Z

failures in CI seems unrelated. I'll merge this.

* Refactor ActorID, TaskID on the Java side. Left a TODO comment WIP for ObjectID ADD test Fix Add java part Fix Java test Fix Refine test. Enable test in CI * Extra a helper function. * Resolve TODOs * Fix Python CI * Fix Java lint * Update .travis.yml Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Address some comments. Address some comments. Add id_specification.rst Reanme id_specification.rst to id_specification.md typo Address zhijun's comments. Fix test Address comments. Fix lint Address comments * Fix test * Address comments. * Fix build error * Update src/ray/design_docs/id_specification.md Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/design_docs/id_specification.md Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/design_docs/id_specification.md Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/design_docs/id_specification.md Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/design_docs/id_specification.md Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Address comments * Update src/ray/common/id.h Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/common/id.h Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/common/id.h Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu> * Update src/ray/design_docs/id_specification.md Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update src/ray/design_docs/id_specification.md Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comments. * Address comments. * Address comments. * Update C++ part to make sure task id is generated determantic * WIP * Fix core worker * Fix Java part * Fix comments. * Add Python side * Fix python * Address comments * Fix linting * Fix * Fix C++ linting * Add JobId() method to TaskID * Fix linting * Update src/ray/common/id.h Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/TaskId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/TaskId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/ActorId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comments * Add DriverTaskId embeding job id * Fix tests * Add python dor_fake_driver_id * Address comments and fix linting * Fix CI

jovany-wang mentioned this pull request Jul 26, 2019

ID Refactor #5285

Closed

9 tasks

jovany-wang requested review from raulchen, stephanie-wang, pcmoritz, robertnishihara, ericl and zhijunfu July 26, 2019 17:37

stephanie-wang self-assigned this Jul 26, 2019

stephanie-wang requested changes Jul 26, 2019

View reviewed changes

zhijunfu reviewed Jul 27, 2019

View reviewed changes

jovany-wang force-pushed the id_embeding branch from c65eb18 to 043cb0b Compare July 28, 2019 08:32

raulchen reviewed Jul 29, 2019

View reviewed changes

zhijunfu approved these changes Jul 30, 2019

View reviewed changes

jovany-wang and others added 6 commits July 30, 2019 21:02

Refactor ActorID, TaskID on the Java side.

ba07448

Left a TODO comment WIP for ObjectID ADD test Fix Add java part Fix Java test Fix Refine test. Enable test in CI

Extra a helper function.

707cd9a

Resolve TODOs

fbcf2aa

Fix Python CI

370bf2f

Fix Java lint

cccee0d

Update .travis.yml

253840d

Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>

raulchen reviewed Aug 4, 2019

View reviewed changes

jovany-wang and others added 4 commits August 4, 2019 13:05

Update src/ray/common/id.h

53b9155

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

Update java/api/src/main/java/org/ray/api/id/TaskId.java

e22074a

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

Update java/api/src/main/java/org/ray/api/id/TaskId.java

0db872d

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

Update java/api/src/main/java/org/ray/api/id/ActorId.java

1737867

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

Address comments

8579fbc

jovany-wang added 3 commits August 5, 2019 19:58

Add DriverTaskId embeding job id

1092d9c

Fix tests

1960a38

Add python dor_fake_driver_id

7842049

Address comments and fix linting

2d80a63

jovany-wang force-pushed the id_embeding branch from 9f935f1 to 2d80a63 Compare August 6, 2019 04:44

jovany-wang added 2 commits August 6, 2019 14:01

Merge branch 'master' into id_embeding

185b977

Fix CI

f9145b7

jovany-wang merged commit d372f24 into ray-project:master Aug 7, 2019

raulchen deleted the id_embeding branch August 7, 2019 06:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ID Refactor] Refactor ActorID, TaskID and ObjectID #5286

[ID Refactor] Refactor ActorID, TaskID and ObjectID #5286

jovany-wang commented Jul 26, 2019 •

edited

Loading

AmplabJenkins commented Jul 26, 2019

AmplabJenkins commented Jul 26, 2019

stephanie-wang left a comment

zhijunfu left a comment

AmplabJenkins commented Jul 27, 2019

AmplabJenkins commented Jul 27, 2019

AmplabJenkins commented Jul 28, 2019

AmplabJenkins commented Jul 28, 2019

jovany-wang commented Jul 29, 2019 •

edited

Loading

AmplabJenkins commented Jul 29, 2019

AmplabJenkins commented Jul 29, 2019

AmplabJenkins commented Jul 29, 2019

AmplabJenkins commented Jul 29, 2019

zhijunfu left a comment

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

jovany-wang commented Aug 7, 2019

[ID Refactor] Refactor ActorID, TaskID and ObjectID #5286

[ID Refactor] Refactor ActorID, TaskID and ObjectID #5286

Conversation

jovany-wang commented Jul 26, 2019 • edited Loading

Related issue number

Linter

AmplabJenkins commented Jul 26, 2019

AmplabJenkins commented Jul 26, 2019

stephanie-wang left a comment

Choose a reason for hiding this comment

zhijunfu left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 27, 2019

AmplabJenkins commented Jul 27, 2019

AmplabJenkins commented Jul 28, 2019

AmplabJenkins commented Jul 28, 2019

jovany-wang commented Jul 29, 2019 • edited Loading

AmplabJenkins commented Jul 29, 2019

AmplabJenkins commented Jul 29, 2019

AmplabJenkins commented Jul 29, 2019

AmplabJenkins commented Jul 29, 2019

zhijunfu left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 4, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 5, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

jovany-wang commented Aug 7, 2019

jovany-wang commented Jul 26, 2019 •

edited

Loading

jovany-wang commented Jul 29, 2019 •

edited

Loading