[WIP] Rebasing materialized dataset on iterator back-pressure is active upon materialization #50880

alexeykudinkin · 2025-02-25T02:52:11Z

Why are these changes needed?

TBA

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Added new param to py-doc Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin added 7 commits February 24, 2025 17:55

Revisited InputDataBuffer to support blocks fed in via iterator

64b3e85

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Fixed stats

e37b453

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Updated py-doc

d4f93cf

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Removed unused param;

762d454

Added new param to py-doc Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Support iter inside InputData Logical Op

bd36b61

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Wire in planner

a88159f

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Rebased materialize to pass in iter instead of a list of bundles

0ca8e15

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin requested a review from a team as a code owner February 25, 2025 02:52

alexeykudinkin added the go add ONLY when ready to merge, run all tests label Feb 25, 2025