Task Summary
Part of #5881 (design: #5880).
Let the scheduler reuse matched ports. Before CostBasedScheduleGenerator runs, remove the operators whose results are reused, leaving the run-skeleton; schedule the run-skeleton as today; then combine it with the skipped regions into one schedule. This is the core of the feature and needs the most review.
Scope
CacheReusePreSchedulingStep: the reuse planner and skeleton generator. Starting from the outputs the run needs (sinks plus any ports the user asked to view), it keeps an operator only if one of its needed outputs is not a matched port; if a needed output is matched, its saved result supplies it and the operators above it are removed. The result is the run-skeleton, the skipped (ToSkip) regions, and the saved result locations for matched inputs and outputs.PreSchedulingHints: a small carrier for those result-location overrides; empty by default.CostBasedScheduleGenerator: two new optional inputs (the hints and the skipped regions), both empty by default. When empty, the schedule is built exactly as today. When present, it schedules the run-skeleton and places the skipped regions ahead of the regions that run.Region gains a cached flag (default false). RegionExecutionCoordinator runs the normal path when false, and a skip path when true: it records a completed-from-cache result with no workers and passes the saved result locations downstream.- Supporting changes: thread the execution id through the coordinators, move the workflow-completion signal to the execution coordinator (a skipped region completes at once with no workers), and add the optional cached fields on the port config.
Why this is safe when there are no matched ports. When the cache lookup returns nothing, CacheReusePreSchedulingStep returns early: the run-skeleton is the whole plan, the hints are empty, and there are no skipped regions. CostBasedScheduleGenerator then builds the schedule as it does today, and every region has cached = false, so the coordinator runs the normal path.
The planning diagrams in the design Discussion show a cost estimator inside the scheduler; here the cost-based choice is not used (every matched port is reused). It is the existing estimator, run only on the run-skeleton.
Depends on. #5882 (the cachedOutputs field and cache types) and #5883 (the COMPLETED_FROM_CACHE state). Can be done while still inactive, since nothing reads or writes cache entries until #5885.
Out of scope. No cost-based decision about whether to reuse (full reuse only). Reading and writing cache entries at submission and completion (#5885).
Task Type
Task Summary
Part of #5881 (design: #5880).
Let the scheduler reuse matched ports. Before
CostBasedScheduleGeneratorruns, remove the operators whose results are reused, leaving the run-skeleton; schedule the run-skeleton as today; then combine it with the skipped regions into one schedule. This is the core of the feature and needs the most review.Scope
CacheReusePreSchedulingStep: the reuse planner and skeleton generator. Starting from the outputs the run needs (sinks plus any ports the user asked to view), it keeps an operator only if one of its needed outputs is not a matched port; if a needed output is matched, its saved result supplies it and the operators above it are removed. The result is the run-skeleton, the skipped (ToSkip) regions, and the saved result locations for matched inputs and outputs.PreSchedulingHints: a small carrier for those result-location overrides; empty by default.CostBasedScheduleGenerator: two new optional inputs (the hints and the skipped regions), both empty by default. When empty, the schedule is built exactly as today. When present, it schedules the run-skeleton and places the skipped regions ahead of the regions that run.Regiongains acachedflag (default false).RegionExecutionCoordinatorruns the normal path when false, and a skip path when true: it records a completed-from-cache result with no workers and passes the saved result locations downstream.Why this is safe when there are no matched ports. When the cache lookup returns nothing,
CacheReusePreSchedulingStepreturns early: the run-skeleton is the whole plan, the hints are empty, and there are no skipped regions.CostBasedScheduleGeneratorthen builds the schedule as it does today, and every region hascached = false, so the coordinator runs the normal path.The planning diagrams in the design Discussion show a cost estimator inside the scheduler; here the cost-based choice is not used (every matched port is reused). It is the existing estimator, run only on the run-skeleton.
Depends on. #5882 (the
cachedOutputsfield and cache types) and #5883 (theCOMPLETED_FROM_CACHEstate). Can be done while still inactive, since nothing reads or writes cache entries until #5885.Out of scope. No cost-based decision about whether to reuse (full reuse only). Reading and writing cache entries at submission and completion (#5885).
Task Type