[core][cgraph] Fix eager release if destruction out of order

[dayshah]

Why are these changes needed?

The current implementation of the CompiledDagRef destructor will release the buffer for everything up to the execution index of the dagref that was destructed. This means that calling get on a previous dagref will fail. Now we're assuring that the previous execution for previous dagrefs complete and the results get cached before we release.

Because Python doesn't guarantee destruction order, this has the possible pitfall of requiring finishing execution of other previous dagrefs that are also about to be destructed, but the dagref with the higher execution_index just got del called on it first. Opened issue #49782 for this.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ruisearch42]

why do we prefer using an extra method call?

[dayshah]

because we want to use execute_until for release buffer and we don't want to get there because we don't want to remove result from buffer

[kevin85421]

Discussed offline:

Calling _execute_until to consume all DAG references with a smaller execution_index compared to the caller of the destruction may cause the memory usage increases in the driver process.

Instead, we will try to remember the execution_index that should be released and release it only when all DAG references with smaller execution_index values have been consumed.