Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ID Refactor] Shorten the length of JobID to 4 bytes #5110

Merged
merged 43 commits into from
Jul 11, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
8a280dd
WIP
jovany-wang Jul 1, 2019
dff8c21
Fix
jovany-wang Jul 3, 2019
689ff32
Add jobid test
jovany-wang Jul 3, 2019
e586258
Fix
jovany-wang Jul 3, 2019
d37f3d0
Add python part
jovany-wang Jul 3, 2019
29f774d
Fix
jovany-wang Jul 3, 2019
0123990
Fix tes
jovany-wang Jul 3, 2019
7aec837
Remove TODOs
jovany-wang Jul 3, 2019
7cef169
Fix C++ tests
jovany-wang Jul 3, 2019
12e1b4c
Lint
jovany-wang Jul 3, 2019
a5598b0
Fix
jovany-wang Jul 4, 2019
aa94dfa
Fix exporting functions in multiple ray.init
jovany-wang Jul 4, 2019
e99206a
Fix java test
jovany-wang Jul 5, 2019
a04d755
Fix lint
jovany-wang Jul 5, 2019
ce63b84
Fix linting
jovany-wang Jul 5, 2019
c307934
Merge branch 'master' into short-jobid-to-4bytes
jovany-wang Jul 7, 2019
e0e34f4
Address comments.
jovany-wang Jul 7, 2019
13c4150
FIx
jovany-wang Jul 8, 2019
18bf5bf
Address and fix linting
jovany-wang Jul 8, 2019
2d047f1
Refine and fix
jovany-wang Jul 8, 2019
7656f28
Fix
jovany-wang Jul 8, 2019
f0ceceb
address
jovany-wang Jul 8, 2019
bc5de09
Address comments.
jovany-wang Jul 8, 2019
24b6a29
Fix linting
jovany-wang Jul 8, 2019
166ece8
Fix
jovany-wang Jul 8, 2019
e81714d
Address
jovany-wang Jul 8, 2019
cacd3c1
Address comments.
jovany-wang Jul 8, 2019
ea1d0eb
Address
jovany-wang Jul 8, 2019
9f89ed6
Address
jovany-wang Jul 8, 2019
68b0c26
Merge branch 'master' into short-jobid-to-4bytes
jovany-wang Jul 8, 2019
376eef0
Fix
jovany-wang Jul 8, 2019
d1e1e6b
Fix
jovany-wang Jul 8, 2019
81a7040
Fix
jovany-wang Jul 9, 2019
d2699e0
Fix lint
jovany-wang Jul 9, 2019
0892ea2
Fix
jovany-wang Jul 9, 2019
95627dc
Fix linting
jovany-wang Jul 9, 2019
bb1a0d4
Address comments.
jovany-wang Jul 10, 2019
30f4683
Fix linting
jovany-wang Jul 10, 2019
193e15e
Address comments.
jovany-wang Jul 10, 2019
01cd242
Fix linting
jovany-wang Jul 10, 2019
4889da8
Merge branch 'master' into short-jobid-to-4bytes
jovany-wang Jul 11, 2019
ebff61a
address comments.
jovany-wang Jul 11, 2019
e36061c
Fix
jovany-wang Jul 11, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address comments.
  • Loading branch information
jovany-wang committed Jul 7, 2019
commit e0e34f442852e0e47296bc37ca6722911a56e27a
1 change: 1 addition & 0 deletions python/ray/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -1738,6 +1738,7 @@ def connect(node,
else:
# This is the code path of driver mode.
if job_id is None:
# TODO(qwang): use `GcsClient::GenerateJobId()` here.
job_id = JobID.from_int(
int(worker.redis_client.incr("JobCounter")))
jovany-wang marked this conversation as resolved.
Show resolved Hide resolved
# When tasks are executed on remote workers in the context of multiple
Expand Down
3 changes: 0 additions & 3 deletions src/ray/common/constants.h
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@
/// Length of Ray full-length IDs in bytes.
constexpr size_t kUniqueIDSize = 20;

/// Length of Job ID in bytes.
constexpr int64_t kJobIDSize = 4;

/// An ObjectID's bytes are split into the task ID itself and the index of the
/// object's creation. This is the maximum width of the object index in bits.
constexpr int kObjectIdIndexSize = 32;
Expand Down
11 changes: 8 additions & 3 deletions src/ray/common/id.h
Original file line number Diff line number Diff line change
Expand Up @@ -74,20 +74,25 @@ class UniqueID : public BaseID<UniqueID> {

class JobID : public BaseID<JobID> {
public:
static constexpr int64_t length = 4;

// TODO(qwang): Move this to `WorkerID::JobId()`
// once we separate `WorkerID` from `UniqueID`.
//
// Note that this is only used for driver id since we only embed
// job id to driver id and not embed job id to worker id yet.
static JobID FromDriverId(const WorkerID &driver_id);

static JobID FromInt(int32_t value);
jovany-wang marked this conversation as resolved.
Show resolved Hide resolved

static size_t Size() { return kJobIDSize; }
static size_t Size() { return length; }

JobID() : BaseID() {}

WorkerID DriverId() const;

private:
uint8_t id_[kJobIDSize];
uint8_t id_[length];
};

class TaskID : public BaseID<TaskID> {
Expand Down Expand Up @@ -141,7 +146,7 @@ class ObjectID : public BaseID<ObjectID> {
int32_t index_;
};

static_assert(sizeof(JobID) == kJobIDSize + sizeof(size_t),
static_assert(sizeof(JobID) == JobID::length + sizeof(size_t),
"JobID size is not as expected");
static_assert(sizeof(TaskID) == kTaskIDSize + sizeof(size_t),
"TaskID size is not as expected");
Expand Down