Java and Python `UniqueID` generation does not match

[stephanie-wang]

Describe the problem

The Java and Python frontends currently use different algorithms to generate TaskIDs and ObjectIDs. Ideally, they should both use the functions in src/ray/id.h to generate UniqueIDs. This is the code that is currently shared between the Python frontend and the Raylet backend.

This problem manifests itself when the Raylet backend tries to compute the TaskID of the task that generated a given ObjectID. It does this, for instance, when trying to decide whether to reconstruct an object. As of #2526, the exact sequence of steps is:

1. The ReconstructionPolicy is asked to reconstruct an object_id, which was computed with the code in java/runtime-common/src/main/java/org/ray/core/UniqueIdHelper.java.
2. The ReconstructionPolicy computes the task ID from object_id, but gets the wrong task ID, since it uses the ComputeTaskId helper function in src/ray/id.h to do so. It begins to listen for task lease notifications for the wrong task ID.
3. The ReconstructionPolicy soon times out the task since no notifications were received and attempts to reconstruct the task. However, it can't find the TaskSpec since it has the wrong task ID.

[robertnishihara]

@stephanie-wang thanks for reporting this!

cc @kingchin1218 @salah-man

[jovany-wang]

Thanks @stephanie-wang . I am following this issue.

[raulchen]

Thanks. It's a known issue. IIRC, there're also some dependencies on the task ID generation. So it might not be a very straightforward change. we'll look into this.

[stephanie-wang]

Great, thanks! @kingchin1218 and @raulchen, as a first step, we should at least make sure that the function to compute the task ID that created an object matches in both frontends. That way, at least we can have reconstruction work for the Java frontend.

[imzhenyu]

It is best that we can come up a manual id logic version defined in Java/Python/Core, respectively. And let's check their consistency upon every startup.

[jovany-wang]

@robertnishihara
It's time to close this issue. :)