cross-back-end process graph processing

[JeroenVerstraelen]

Follow up on #95 and #53. Develop a cross-back end process graph splitting implementation that is useful in the current aggregator setup. It should be based on the algorithm developed during the feasibility study.

[soxofaan]

It should be based on the algorithm developed during the feasibility study.

@JeroenVerstraelen can you point me to the details on this feasibility study?

[soxofaan]

There are still some todo's and plenty of idea's for future work, but I decided to merge the current PoC with 9b65520

[soxofaan]

As discussed in #156 the idea is currently to trigger this feature with a job option like

job_options={
    "split_strategy": {
        "crossbackend": {
            "method": "deep"
        },
    }
}

[soxofaan]

Follow up todo's and tasks (WIP)

• Validation warning with BackendLookupFailure in crossbackend mode #154
• include process availability (not only collection availability) -> cross-backend processing: include process availability #157
• debug tools:
  • How can the graph or job splitting be visualized to the user so that they can inspect how the graph is split?
  • think about how to handle failures given the added dimension of working across backends: if the primary job fails, or the secondary job fails: how to present that to the user? How can they be instructed clearly how to fix what where?
  • -> cross-backend processing: user oriented debugging and observability #158
• splitting now only is done on nodes that output raster data (because the load_stac bridge is currently limited to geotiff), also include vector cube producing nodes? -> cross-backend processing: expand output format aspect #159
• enable the feature by default (instead of requiring an opt-in flag) -> cross-backend processing: enable by default #160
• provide tuning parameters (as job options), e.g. prefered backend for main job -> Cross-backend processing: more tuning parameters #161

[soxofaan]

made an initial demo notebook at
https://gist.github.com/soxofaan/c374928c32212031eb1518c1260eb954

unfortunately the job metadata visualization (including process graph viz) does not work in that online notebook viewer, so the explanation might not always make sense