Repository navigation
fix(databases): complete the dead restartNumber reindex-recovery trigger - #1374
Merged
Merged
Conversation
table() re-triggers a secondary-index backfill when the persisted attribute descriptor's restartNumber is older than the current restart generation — the PID-reuse-robust detector for an index left incomplete by a crashed worker. But restartNumber was never persisted onto the descriptor, so the comparison was always `undefined < N` (false) and the trigger never fired. Persist restartNumber symmetrically with indexingPID: set it to the owning worker's startup generation when a backfill is triggered, carry it across metadata-only changes while a backfill is in progress, and clear it on clean completion (retained on park/crash). The trigger now compares against `workerData.restartNumber ?? manageThreads.restartNumber` — the owning worker's stable startup generation, not the mutable global counter, which has already advanced to the replacement generation during a worker's shutdown/drain and would otherwise defeat recovery. Existing descriptors have no persisted restartNumber, so `undefined < N` stays false and there is no upgrade reindex storm. Resolves #1359 (part of #1354). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Contributor
|
Reviewed; no blockers found. |
heskew
marked this pull request as ready for review
June 18, 2026 04:51
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
This was referenced Jun 18, 2026
kriszyp
approved these changes
Jun 18, 2026
kriszyp
left a comment
Member
There was a problem hiding this comment.
Dead restartNumber path correctly wired up, good crash-recovery test coverage. LGTM! 🟢
Reviewed by Claude Sonnet 4.6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
table()re-triggers a secondary-index backfill when the persisted attribute descriptor'srestartNumberis older than the current restart generation. This is the PID-reuse-robust complement to theindexingPIDcheck: it recovers an index left incomplete by a worker that crashed mid-backfill, even when the replacement worker reuses the old PID. The bug:restartNumberwas never persisted onto the descriptor, so the comparison was alwaysundefined < N(false) and the trigger was dead.This persists
restartNumbersymmetrically withindexingPID— set on backfill start, carried across metadata-only changes during an in-progress build, cleared on clean completion (retained on park/crash).Resolves #1359 (part of #1354).
Where to look
resources/databases.ts—table()(set/carry + the two trigger sites) andrunIndexing()(clear on clean completion).workerData?.restartNumber ?? manageThreads.restartNumber— the owning worker's stable startup generation, falling back to the maintainedmanageThreadscounter on the main thread. Cross-model review (Codex) flagged that stamping the mutablemanageThreads.restartNumberis wrong during a worker's shutdown/drain window: the global counter has already advanced to the replacement generation, so an index built then would be stamped with the replacement's generation and the replacement worker would see an equal generation (+ a possibly-reused PID) and never recover it. Using the spawn generation avoids that.Back-compat
Existing descriptors have no persisted
restartNumber→undefined < Nisfalse, so upgrading does not trigger a reindex storm. On the main thread the generation is1, so1 < 1isfalse(inert until an actual restart bumps the counter).Tests
unitTests/resources/indexRestartNumber.test.js(RocksDB-only): (1)restartNumberis persisted on a parked/incomplete index and cleared after a clean completion; (2) a backfill re-triggers when the persisted generation is older than the current one, and not when equal. Re-trigger is asserted viaindexingOperationpromise identity (the promise carries acrossresetDatabases, so truthiness is unreliable — mirrors the existing F1 test).🤖 Generated with Claude Code (Opus 4.8)