Repository navigation
Publish the restoring marker by rename so a torn write cannot unblock a half-purged database - #2644
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces atomic publishing of the database restoring marker to prevent torn writes from leaving invalid or empty markers. It implements a temporary file write followed by a rename and directory fsync, and adds a check to ensure existing markers are intact before reusing them. Additionally, comprehensive unit tests have been added to verify these behaviors under various failure scenarios. I have no feedback to provide as there are no review comments to address.
cb1kenobi
force-pushed
the
fix/restore-marker-atomic-write
branch
7 times, most recently
from
September 18, 2026 19:22
421d01c to
fc65c9a
Compare
cb1kenobi
force-pushed
the
design/archive-restore
branch
from
September 18, 2026 21:31
0377938 to
bcabc17
Compare
cb1kenobi
force-pushed
the
fix/restore-marker-atomic-write
branch
2 times, most recently
from
September 21, 2026 03:13
165e5f1 to
fb41b43
Compare
cb1kenobi
force-pushed
the
design/archive-restore
branch
from
September 21, 2026 03:13
bcabc17 to
66fcfff
Compare
cb1kenobi
force-pushed
the
fix/restore-marker-atomic-write
branch
from
September 21, 2026 14:09
fb41b43 to
6b47034
Compare
cb1kenobi
force-pushed
the
design/archive-restore
branch
from
September 21, 2026 14:09
66fcfff to
36cedab
Compare
cb1kenobi
marked this pull request as ready for review
September 21, 2026 14:15
github-actions
Bot
requested review from
Ethan-Arrowood,
heskew and
kriszyp
September 21, 2026 14:15
Contributor
|
Reviewed; no blockers found. |
cb1kenobi
force-pushed
the
fix/restore-marker-atomic-write
branch
from
September 21, 2026 14:24
6b47034 to
a47f899
Compare
cb1kenobi
added a commit
that referenced
this pull request
Sep 21, 2026
cb1kenobi
force-pushed
the
fix/restore-marker-atomic-write
branch
from
September 21, 2026 15:06
b29e30d to
83460b8
Compare
kriszyp
approved these changes
Sep 21, 2026
kriszyp
left a comment
Member
There was a problem hiding this comment.
Looks great!
🤖 Reviewed with Codex
beginRestore() opened the marker with openSync(markerPath, 'w'), which truncates before the database name is written, and scanBlockedRestores() skips any marker whose first line is empty. On a first attempt that is benign. On a recovery attempt — where the directory may already be half-purged by a failed restore — the marker being truncated is the only thing keeping that directory from loading, so a crash in that window left a half-purged database that loads as healthy on the next boot. Two changes close it. The marker is now published temp -> fsync -> rename -> parent fsync, so the marker path only ever holds the previous content or the complete new content. And an intact marker is not rewritten at all: the content a recovery attempt would write is the content already there, so the write is pure risk. A marker whose first line is empty — what a torn write under the old code left behind — is still repaired, so an instance upgrading with that debris present recovers the block rather than inheriting it. The temp name is fixed rather than unique because the caller holds the exclusive per-database restore lock: a temp left by an earlier crash is this database's own debris and is overwritten. It deliberately does not end in `.restoring`, which is how scanBlockedRestores selects markers. Fixes #2634 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two gaps in keeping an intact marker, from cross-model review.
'Intact' meant a non-empty first line, but the startup scan maps a
marker to a database by that line. A write torn mid-name leaves a prefix
("som" for "somedb"), which is non-empty and so was preserved — and
resolves to a different metadata key, so it blocks nothing at all while
the real database loads. The check now requires the line to be the whole
database name.
Preserving a marker also skipped the directory fsync. The publisher that
wrote it may have been interrupted between the rename and its own flush,
leaving the directory entry — the thing the scan reads — only in the
page cache, so a recovery attempt that reuses it inherits that exposure.
The flush now runs either way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
cb1kenobi
force-pushed
the
fix/restore-marker-atomic-write
branch
from
September 21, 2026 19:16
83460b8 to
4d9a0f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beginRestore()opened the restoring marker withopenSync(markerPath, 'w'), which truncates before the database name is written, andscanBlockedRestores()skips any marker whose first line is empty. On a recovery attempt — over a directory an earlier restore may have half-purged — that marker is the only thing keeping the directory from loading, so a crash in the truncate-to-write window left a half-purged database that loads as healthy on the next boot.The marker is now published temp → fsync → rename → parent fsync, so the path only ever holds the previous content or the complete new content, and an intact marker is not rewritten at all: the content a recovery attempt would write is the content already there, so the write was pure risk.
For the human reviewer
.restoring, which is howscanBlockedRestoresselects markers.Changes
dataLayer/restoreMarker.ts—publishRestoringMarkerwrites a temp sibling, fsyncs it, renames it over the marker path and fsyncs the directory;markerIsIntactcompares the marker's first line against the database directory name; andbeginRestorepublishes only when there is no intact marker, flushing the directory either way.Verification
Route (a) — extended
unitTests/dataLayer/restoreMarker.test.js, the suite that already covers this module, and ran it throughnpm run test:unit:backup(the repo's focused backup suite: 128 passing). Fails-on-base check performed: with the source reverted and the tests kept,leaves an intact marker byte-identical on a recovery attemptandoverwrites a temp file a crashed write left behindboth fail, and pass on the branch.The test file is
unitTests/dataLayer/restoreMarker.test.js, which gains cases for the byte-identical recovery marker, the torn-name repair, the empty-marker repair, and the absence of temp debris.Complexity: medium
Cross-model review. Reviewed against
mainby codexgpt-6-astra(xhigh), Gemini 3.1 Pro and cursor-composer — three outside families. The round found a real defect:preexistingwas sampled before the restore lock was acquired, so an attempt that raced a crashing predecessor could believe an inherited marker was its own and clear it, unblocking a half-purged directory — the exact failure this PR exists to prevent. Fixed inac0708b7. One reported blocker was not adopted: the claim that the temporary file is never flushed before rename is incorrect,fsyncSync(fd)sits betweenwriteSyncandcloseSync. The Harper-domain adjudication leg was killed (SIGKILL), so findings were triaged by the author rather than adjudicated.Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex; blocked=domain(exit--1); declined=cursor-grok; rounds=1; full=1 @ ac0708b
Human-Review-Need: 4 @ ac0708b