[GCS]refactor the GCS Client Task Interface

[micafan]

Refactoring the GCS Client Task Interface .
Only expose Ray-specific semantics (e.g., AsyncSubscribe, AsyncUpdate, etc), and hide the implementation details about the backed storage.

Create this PR because the old PR #5515 which is merged and reverted (reverted because UT failed and the wrong CHECK in code).

Why are these changes needed?

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.

[AmplabJenkins]

Can one of the admins verify this patch?

[ericl]

It's a bit unfortunate we have to add interfaces for the task table, which is only used in a very minimal way at this point but is quite complex. Would it be bad to leave it out? Once we enable the new scheduler it will be completely unused.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19897/
Test FAILed.

[micafan]

It's a bit unfortunate we have to add interfaces for the task table, which is only used in a very minimal way at this point but is quite complex. Would it be bad to leave it out? Once we enable the new scheduler it will be completely unused.

How about add the new interface to TaskInfoAccessor too by then? I think expose TaskTable is not safe, some people might use it instead of TaskInfoAccessor.

[ericl]

It shouldn't matter, as the code is deprecated and will be removed as soon as the new scheduler is ready. Direct calls doesn't use it.

How about we freeze changes to the task lineage and lease management until it is removed?

[micafan]

It doesn't matter, as the code is deprecated and will be removed as soon as the new scheduler is ready.

How about we freeze changes to the task lineage and lease management until it is removed?

If you really don't want to add new interface to TaskInfoAccessor, then we can expose the TaskTable for temporary. I don't think freeze this PR is a good idea, because it is not clear how long the PR is frozen.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19902/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/19903/
Test FAILed.

[raulchen]

Hi @ericl, we'd like to merge this PR first to unblock other work on GCS Service.
The changes to table table and lineage cache are pretty straightforward. We simply wrap old task_table code with new TaskInfoAccessor interface.
When we enable the new scheduler, it should be very easy to remove the task table.
Also, this PR refactors code about the new gcs client as well.
Please let us know if you still have concerns. Thanks.

[raulchen]

BTW, for the short term, I think we still need task table to store actor creation task spec.

[micafan]

@ericl Could you take a look at the auto scaling test failed issue 6582 .

[raulchen]

Test failure should be irrelevant.