Closed
Description
Background
This would allow running multiple ingestions at the same time, smaller ones backfilling larger ones from a parallel process.
Maybe an option:
--skip-locked
That would make the pipeline skip jobs when a lock is held by a different process, while continue running the rest of jobs. E.g. not considering it an error as per default behavior.
Feature
Allow skipping jobs that hold locks to output tables.
Example [Optional]
--
Proposed Solution [Optional]
--