Repository navigation
Resolve pipeline stages per dataset #883
Description
Activity
Interesting proposal. I think it would be good to point out that this change would make the pipeline definition dynamic rather than static.
On one hand this is nice as it add flexibility and allows the implementer to use the same pipeline for datasets with different structures. Though it does also make it more difficult to reason and predict what steps the pipeline will be taking at any particular time.
Another (admittedly not ideal) approach could be to add a skip mechanic on a
Stagelevel instead (skipping a stage based on some contextual value);skip: (dataset: Dataset) => boolean
However this would be inefficient if there are many datasets bringing their own optional compensator step. And it would mean that the parent of a skipped substage should use the original dataset as input rather than the substage output. Which would be a bit of a weird quirk.
Problem
PipelineOptions.stagesis a fixedStage[]: every selected dataset runs the same stages. The LODA pipeline (netwerk-digitaal-erfgoed/loda-pipeline#17) transforms six schema.org datasets to EDM in one run. All six share a generic stage set. denbosch also needs a compensator stage that lifts its source to schema.org first.Today LODA runs one
Pipelineper dataset: one process each, with no writer run or reporter shared between them.Proposal
Accept a function of the dataset, as
Stage.sourceForalready does for the distribution:A plain array keeps its current meaning, so this is not a breaking change.
Open questions
chainingis set when a stage has sub-stages. With a function that check moves to the point where the stages are resolved.