feat: let a backfill class cap how many batches run in parallel - #31
Open
guilleov wants to merge 4 commits into
Open
feat: let a backfill class cap how many batches run in parallel#31guilleov wants to merge 4 commits into
guilleov wants to merge 4 commits into
Conversation
A run enqueues every batch the moment the dripper creates it, so a run with hundreds of batches occupies as many worker threads even when the batches serialize on one shared resource and simply wait. A backfill class can now declare self.max_parallel_batches: only that many children are enqueued or running at once, the rest stay pending, and the finishing child (completed or failed) enqueues the oldest pending one. A stopped run releases its pending batches as stopped. Default stays unlimited; the retry action respects the limit through the same enqueue path.
Deleting a pending run raised FrozenError because after_commit :enqueue also fired on the destroy commit and called enqueued! on the frozen record. A run due now is pushed with perform_later instead of set(wait_until:), which the inline adapter rejects; future runs are still scheduled.
Keeps main's commit-before-enqueue ordering in BackfillRun#enqueue and adds the branch's no-scheduling path for runs due now.
guilleov
marked this pull request as ready for review
September 4, 2026 09:14
… job at whole seconds The compiled stylesheet picked up main's classes only after the merge, and the scheduling spec compared a sub-microsecond Ruby time with the persisted value, which passes or fails depending on the Ruby build.
braislchao
approved these changes
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DataDrip enqueues every batch of a run as soon as the dripper creates it. For a backfill whose batches serialize on one shared resource (the Iris CH-native backfill takes a global advisory lock so only one batch touches ClickHouse at a time), a run with hundreds of batches means hundreds of children holding
within_24_hoursworker threads while they block on the lock.What
DataDrip::Backfill.max_parallel_batches(class method, defaultnil= unlimited, current behaviour).BackfillRunBatch#enqueueonly enqueues while fewer than that many siblings areenqueued/running; the rest staypending.DripperChildhands the slot to the oldest pending batch after it completes or fails; when the run was stopped it releases the pending batches asstoppedso nothing sits inpendingforever.enqueue, so it respects the limit.Tests
bundle exec rspec: 299 examples, 0 failures. New examples cover the enqueue gate (limit vs. no limit), slot hand-over on success and on failure, and the stopped-run release.rubocop: clean.First consumer:
Iris::ChBackfillEntityin the monorepo (factorialco/factorial#112674), which will set 1.