Skip to content

Publish the restoring marker by rename so a torn write cannot unblock a half-purged database - #2644

Merged
cb1kenobi merged 5 commits into
mainfrom
fix/restore-marker-atomic-write
Sep 21, 2026
Merged

cb1kenobi merged 5 commits into
mainfrom
fix/restore-marker-atomic-write

Conversation

@cb1kenobi

@cb1kenobi cb1kenobi commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

beginRestore() opened the restoring marker with openSync(markerPath, 'w'), which truncates before the database name is written, and scanBlockedRestores() skips any marker whose first line is empty. On a recovery attempt — over a directory an earlier restore may have half-purged — that marker is the only thing keeping the directory from loading, so a crash in the truncate-to-write window left a half-purged database that loads as healthy on the next boot.

The marker is now published temp → fsync → rename → parent fsync, so the path only ever holds the previous content or the complete new content, and an intact marker is not rewritten at all: the content a recovery attempt would write is the content already there, so the write was pure risk.

For the human reviewer

  1. An intact marker is reused rather than rewritten, so its timestamp stops tracking the latest attempt. Nothing parses the second line, and the original interrupted restore's time is arguably the more useful one, but it is a visible change for anyone reading the file by hand. The alternative — rewriting it — is the defect.
  2. "Intact" means the first line equals the database directory name, not merely non-empty. A write torn mid-name leaves a prefix that resolves to a different metadata key and therefore blocks nothing, so a prefix is repaired rather than preserved. This also repairs debris left by the buggy version on upgrade.
  3. The temp name is fixed rather than unique. The caller holds the exclusive per-database restore lock, so a temp left by an earlier crash is this database's own debris and is simply overwritten. It deliberately does not end in .restoring, which is how scanBlockedRestores selects markers.

Changes

  • dataLayer/restoreMarker.ts — publishRestoringMarker writes a temp sibling, fsyncs it, renames it over the marker path and fsyncs the directory; markerIsIntact compares the marker's first line against the database directory name; and beginRestore publishes only when there is no intact marker, flushing the directory either way.

Verification

Route (a) — extended unitTests/dataLayer/restoreMarker.test.js, the suite that already covers this module, and ran it through npm run test:unit:backup (the repo's focused backup suite: 128 passing). Fails-on-base check performed: with the source reverted and the tests kept, leaves an intact marker byte-identical on a recovery attempt and overwrites a temp file a crashed write left behind both fail, and pass on the branch.

The test file is unitTests/dataLayer/restoreMarker.test.js, which gains cases for the byte-identical recovery marker, the torn-name repair, the empty-marker repair, and the absence of temp debris.

Complexity: medium


Cross-model review. Reviewed against main by codex gpt-6-astra (xhigh), Gemini 3.1 Pro and cursor-composer — three outside families. The round found a real defect: preexisting was sampled before the restore lock was acquired, so an attempt that raced a crashing predecessor could believe an inherited marker was its own and clear it, unblocking a half-purged directory — the exact failure this PR exists to prevent. Fixed in ac0708b7. One reported blocker was not adopted: the claim that the temporary file is never flushed before rename is incorrect, fsyncSync(fd) sits between writeSync and closeSync. The Harper-domain adjudication leg was killed (SIGKILL), so findings were triaged by the author rather than adjudicated.

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex; blocked=domain(exit--1); declined=cursor-grok; rounds=1; full=1 @ ac0708b

Human-Review-Need: 4 @ ac0708b

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces atomic publishing of the database restoring marker to prevent torn writes from leaving invalid or empty markers. It implements a temporary file write followed by a rename and directory fsync, and adds a check to ensure existing markers are intact before reusing them. Additionally, comprehensive unit tests have been added to verify these behaviors under various failure scenarios. I have no feedback to provide as there are no review comments to address.

@cb1kenobi cb1kenobi added this to the v5.3 milestone Sep 16, 2026
@cb1kenobi
cb1kenobi force-pushed the fix/restore-marker-atomic-write branch 7 times, most recently from 421d01c to fc65c9a Compare September 18, 2026 19:22
@cb1kenobi
cb1kenobi force-pushed the design/archive-restore branch from 0377938 to bcabc17 Compare September 18, 2026 21:31
@cb1kenobi
cb1kenobi force-pushed the fix/restore-marker-atomic-write branch 2 times, most recently from 165e5f1 to fb41b43 Compare September 21, 2026 03:13
@cb1kenobi
cb1kenobi force-pushed the design/archive-restore branch from bcabc17 to 66fcfff Compare September 21, 2026 03:13
@cb1kenobi
cb1kenobi force-pushed the fix/restore-marker-atomic-write branch from fb41b43 to 6b47034 Compare September 21, 2026 14:09
@cb1kenobi
cb1kenobi force-pushed the design/archive-restore branch from 66fcfff to 36cedab Compare September 21, 2026 14:09
@cb1kenobi
cb1kenobi marked this pull request as ready for review September 21, 2026 14:15
Comment thread dataLayer/restoreMarker.ts
@claude

claude Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@cb1kenobi
cb1kenobi force-pushed the fix/restore-marker-atomic-write branch from 6b47034 to a47f899 Compare September 21, 2026 14:24
cb1kenobi added a commit that referenced this pull request Sep 21, 2026
@cb1kenobi
cb1kenobi force-pushed the fix/restore-marker-atomic-write branch from b29e30d to 83460b8 Compare September 21, 2026 15:06

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

🤖 Reviewed with Codex

cb1kenobi and others added 3 commits September 21, 2026 14:15
beginRestore() opened the marker with openSync(markerPath, 'w'), which
truncates before the database name is written, and scanBlockedRestores()
skips any marker whose first line is empty. On a first attempt that is
benign. On a recovery attempt — where the directory may already be
half-purged by a failed restore — the marker being truncated is the only
thing keeping that directory from loading, so a crash in that window
left a half-purged database that loads as healthy on the next boot.

Two changes close it. The marker is now published temp -> fsync ->
rename -> parent fsync, so the marker path only ever holds the previous
content or the complete new content. And an intact marker is not
rewritten at all: the content a recovery attempt would write is the
content already there, so the write is pure risk. A marker whose first
line is empty — what a torn write under the old code left behind — is
still repaired, so an instance upgrading with that debris present
recovers the block rather than inheriting it.

The temp name is fixed rather than unique because the caller holds the
exclusive per-database restore lock: a temp left by an earlier crash is
this database's own debris and is overwritten. It deliberately does not
end in `.restoring`, which is how scanBlockedRestores selects markers.

Fixes #2634

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two gaps in keeping an intact marker, from cross-model review.

'Intact' meant a non-empty first line, but the startup scan maps a
marker to a database by that line. A write torn mid-name leaves a prefix
("som" for "somedb"), which is non-empty and so was preserved — and
resolves to a different metadata key, so it blocks nothing at all while
the real database loads. The check now requires the line to be the whole
database name.

Preserving a marker also skipped the directory fsync. The publisher that
wrote it may have been interrupted between the rename and its own flush,
leaving the directory entry — the thing the scan reads — only in the
page cache, so a recovery attempt that reuses it inherits that exposure.
The flush now runs either way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cb1kenobi
cb1kenobi changed the base branch from design/archive-restore to main September 21, 2026 19:16
@cb1kenobi
cb1kenobi force-pushed the fix/restore-marker-atomic-write branch from 83460b8 to 4d9a0f7 Compare September 21, 2026 19:16
@cb1kenobi
cb1kenobi enabled auto-merge September 21, 2026 19:54
@cb1kenobi
cb1kenobi merged commit 56511db into main Sep 21, 2026
48 checks passed
@cb1kenobi
cb1kenobi deleted the fix/restore-marker-atomic-write branch September 21, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants