[GCS]Support gcs client job&actor&node table subscribe idempotent

[ffbin]

Why are these changes needed?

Design document:
https://docs.google.com/document/d/1Cuqxlw53abEZPVYVF-pUWpbnwrFEgVTel_sQGwt8DmU/edit#heading=h.s3ogmk8m7308
This PR implements GCS client subscribe idempotent.
Node table and object location table are special in that they contain several records, so they are sorted by timestamp in GCS, I will implement it in the next pr.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested (please justify below)

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28212/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28213/
Test PASSed.

[rkooo567]

What's the relationship to this PR btw? #9153

[ffbin]

What's the relationship to this PR btw? #9153
I plan to split #9153 into three parts: the first is this pr, the second is support gcs client node resource subscribe idempotent, and the third is support gcs client object location subscribe idempotent.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28437/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28438/
Test PASSed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28453/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28462/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/28463/
Test PASSed.

[stephanie-wang]

Actor table idempotence is already implemented in #9449 and the node table should already be idempotent. I'm not familiar with what the job table does, but it looks like it's just marking jobs as dead, which is idempotent too.

So do we actually need this PR?

[ffbin]

Actor table idempotence is already implemented in #9449 and the node table should already be idempotent. I'm not familiar with what the job table does, but it looks like it's just marking jobs as dead, which is idempotent too.

So do we actually need this PR?

New changes in notification handlers could easily introduce bugs without being noticed. In GCS client to ensure message processing idempotent, we can develop more simple and efficient.

[stephanie-wang]

Actor table idempotence is already implemented in #9449 and the node table should already be idempotent. I'm not familiar with what the job table does, but it looks like it's just marking jobs as dead, which is idempotent too.
So do we actually need this PR?

New changes in notification handlers could easily introduce bugs without being noticed. In GCS client to ensure message processing idempotent, we can develop more simple and efficient.

But there is no point in making message processing in the GCS idempotent if the specific table logic is already idempotent. The problem has already been fixed, so this is just adding unnecessary complexity.

I'm not worried about changes in the notification handlers because these can be easily unit tested so we would definitely notice any new bugs. I suggest we close this PR.

[stephanie-wang]

Closing this PR for now since I don't believe the PR is necessary and it's now been 1 month. Feel free to reopen if there are more specific changes that need to be made.