Skip to content

fix(databases): complete the dead restartNumber reindex-recovery trigger - #1374

Merged
kriszyp merged 1 commit into
mainfrom
1359-complete-restartnumber
Jun 18, 2026
Merged

kriszyp merged 1 commit into
mainfrom
1359-complete-restartnumber

Conversation

@heskew

@heskew heskew commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary

table() re-triggers a secondary-index backfill when the persisted attribute descriptor's restartNumber is older than the current restart generation. This is the PID-reuse-robust complement to the indexingPID check: it recovers an index left incomplete by a worker that crashed mid-backfill, even when the replacement worker reuses the old PID. The bug: restartNumber was never persisted onto the descriptor, so the comparison was always undefined < N (false) and the trigger was dead.

This persists restartNumber symmetrically with indexingPID — set on backfill start, carried across metadata-only changes during an in-progress build, cleared on clean completion (retained on park/crash).

Resolves #1359 (part of #1354).

Where to look

  • resources/databases.ts — table() (set/carry + the two trigger sites) and runIndexing() (clear on clean completion).
  • Operand decision (worth a look): the trigger compares against workerData?.restartNumber ?? manageThreads.restartNumber — the owning worker's stable startup generation, falling back to the maintained manageThreads counter on the main thread. Cross-model review (Codex) flagged that stamping the mutable manageThreads.restartNumber is wrong during a worker's shutdown/drain window: the global counter has already advanced to the replacement generation, so an index built then would be stamped with the replacement's generation and the replacement worker would see an equal generation (+ a possibly-reused PID) and never recover it. Using the spawn generation avoids that.

Back-compat

Existing descriptors have no persisted restartNumber → undefined < N is false, so upgrading does not trigger a reindex storm. On the main thread the generation is 1, so 1 < 1 is false (inert until an actual restart bumps the counter).

Tests

unitTests/resources/indexRestartNumber.test.js (RocksDB-only): (1) restartNumber is persisted on a parked/incomplete index and cleared after a clean completion; (2) a backfill re-triggers when the persisted generation is older than the current one, and not when equal. Re-trigger is asserted via indexingOperation promise identity (the promise carries across resetDatabases, so truthiness is unreliable — mirrors the existing F1 test).


🤖 Generated with Claude Code (Opus 4.8)

table() re-triggers a secondary-index backfill when the persisted attribute
descriptor's restartNumber is older than the current restart generation — the
PID-reuse-robust detector for an index left incomplete by a crashed worker. But
restartNumber was never persisted onto the descriptor, so the comparison was
always `undefined < N` (false) and the trigger never fired.

Persist restartNumber symmetrically with indexingPID: set it to the owning
worker's startup generation when a backfill is triggered, carry it across
metadata-only changes while a backfill is in progress, and clear it on clean
completion (retained on park/crash). The trigger now compares against
`workerData.restartNumber ?? manageThreads.restartNumber` — the owning worker's
stable startup generation, not the mutable global counter, which has already
advanced to the replacement generation during a worker's shutdown/drain and
would otherwise defeat recovery. Existing descriptors have no persisted
restartNumber, so `undefined < N` stays false and there is no upgrade reindex storm.

Resolves #1359 (part of #1354).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@heskew
heskew requested review from cb1kenobi and kriszyp June 18, 2026 04:34
@claude

claude Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@heskew
heskew marked this pull request as ready for review June 18, 2026 04:51
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead restartNumber path correctly wired up, good crash-recovery test coverage. LGTM! 🟢

Reviewed by Claude Sonnet 4.6

@kriszyp
kriszyp merged commit 0f0afca into main Jun 18, 2026
48 checks passed
@kriszyp
kriszyp deleted the 1359-complete-restartnumber branch June 18, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dead restartNumber reindex trigger in table loading (never fires)

2 participants