fix(replication): classify gone/corrupt source blobs as permanent via forwarded statusCode (#429)

[kriszyp]

Fixes #429. Pairs with harper#1425 — must ship together (see Coordination below).

Problem (#429)

A confidently incomplete/truncated source blob (Blob is incomplete) was misclassified as a transient failure on the receiver, permanently wedging the replication resume cursor:

[error] [replication]: Blob save failed for 24b9 from plr-… Blob error: Blob is incomplete …
[warn]  [replication]: Error sending blob Error: Blob is incomplete

Core threw a plain Error with no .code, so sendBlobs forwarded errorCode: undefined, and isPermanentSourceBlobErrorCode(undefined) → false → hasBlobGap held the cursor, expecting a reconnect to re-stream successfully. But a truncated stub at the source never becomes complete, so every reconnect reproduces the identical error and the cursor for that DB is pinned forever. Same failure class as #403, for incomplete blobs instead of missing ones.

Fix

harper#1425 reworks core's blob read paths to throw a BlobReadError carrying an HTTP-style statusCode (404 gone / 500 confidently corrupt-incomplete / 503 transient) instead of a bare Error. This PR is the harper-pro half:

• sendBlobs now forwards errorStatus (the BlobReadError.statusCode) alongside errorCode.
• isPermanentSourceBlobErrorCode(errorCode, errorStatus) classifies 404 and 500 as permanent → advance the resume cursor past + markSourceBlobUnavailable (→ feat(replication): proactive blob repair sweep for already-committed missing blobs #418/Proactive blob backfill: repair already-committed records whose blobs are missing/corrupt (no recovery once the resume cursor advances past them) #388 backfill + Sustained blob-replication timeouts cause silent, unrecoverable blob loss with no operator signal or self-repair ([customer-cluster]: 344/345 attachments, ~663 MB) #386 divergence metric), while 503 stays transient → hold the gap, retry on reconnect.

Everything downstream already existed (markSourceBlobUnavailable → isUnrecoverableSourceBlobError → advance + record divergence); this just routes the new error shape into it, exactly as #429 proposed — except keyed on statusCode rather than the ad-hoc ERR_BLOB_INCOMPLETE .code the issue anticipated, since #1425 supplies the richer taxonomy.

Why it must pair with harper#1425 (regression guard, not just #429)

Once #1425 lands, core wraps even ENOENT into a code-less BlobReadError(404). Without this PR forwarding/honoring errorStatus, a missing source blob would lose its errorCode: 'ENOENT' signal and the receiver would start holding on it — regressing the already-shipped #405 advance-past (observed live on JJill: "unrecoverable at source … advancing" 1,549×). The errorStatus === 404 arm preserves it. harper#1425 also sets code: 'ENOENT' on the 404 so an older receiver (pre-this-PR) stays correct too — mixed-version safe in both directions.

Compatibility

• Pre-#1425 sender → still forwards errorCode: 'ENOENT' (handled); errorStatus is undefined (ignored).
• Sender that forwards neither code nor status → stays on the safe hold default.

Testing

Extended unitTests/replication/blobReplicationFailure.test.mjs for the new errorStatus arm (404/500 permanent, 503 transient, either-signal-suffices, unrelated statuses ignored) — 24 passing.

Integration note: the existing cluster fixture only models the ENOENT/404 path. End-to-end coverage of the 500 (#429) path needs core #1425 merged into the core submodule first; left as a noted follow-up (fixture-blob-fail-source-truncate) rather than a half-test.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates the replication logic to classify permanent source blob failures using HTTP-style status codes (such as 404 and 500) in addition to file system error codes like 'ENOENT'. This ensures compatibility with newer senders that wrap read errors. Feedback on this change highlights a risk in treating generic 500 status codes as permanent failures, as transient internal errors could be misclassified, potentially leading to skipped blobs and data loss. It is recommended to use a more specific status code, such as 422, to represent confidently corrupt or incomplete blobs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using 500 (Internal Server Error) to classify a permanent failure is risky because 500 is typically the generic fallback status code for any unexpected internal error or unhandled exception. If a transient error (such as a temporary disk I/O glitch, database lock, or out-of-memory condition) occurs on the sender and is mapped to a generic 500 status code, the receiver will misclassify it as a permanent failure, skip the blob, and advance the resume cursor. This could lead to unintended data divergence or silent data loss.

Consider using a more specific status code (e.g., 422 Unprocessable Entity or a custom status/error code) to represent 'confidently corrupt/incomplete' to avoid conflating it with generic internal server errors. This would require coordination with the core PR harper#1425.

Suggested change

	export function isPermanentSourceBlobErrorCode(errorCode: unknown, errorStatus?: unknown): boolean {
	return errorCode === 'ENOENT' || errorStatus === 404 || errorStatus === 500;
	}
	export function isPermanentSourceBlobErrorCode(errorCode: unknown, errorStatus?: unknown): boolean {
	return errorCode === 'ENOENT' || errorStatus === 404 || errorStatus === 422;
	}

References

1. When handling transient stream or save failures in a replication system, avoid immediately terminating the connection to force a reconnect if the reconnect will simply re-stream the same failing data and re-trigger the failure. Instead, handle the gap with a bounded tradeoff (such as clamping the resume cursor) until a natural reconnect or restart.

[claude]

Reviewed; no blockers found.

[kriszyp]

Added the end-to-end integration coverage (replicationBlobIncompleteSource.test.mjs + fixture-blob-truncate-source) — sibling to the existing replicationBlobSourceUnavailable.test.mjs, but driving the incomplete/500 path instead of ENOENT/404. The fixture truncates a deterministic subset of source blob files (header intact, body short) so a replication read hits core's BlobReadError('Blob is incomplete', 500); the test asserts the receiver logs "advancing the resume cursor past it" and that the driving error was the incomplete/500 arm (not ENOENT).

Validation boundary (please read): I built this branch against the harper#1425 core (pointed the worktree's core submodule at the #1425 tip locally — not committed; the PR diff carries no core-pointer change) and confirmed the build is coherent and the fixture loads (node A logs [blob-truncate-source] installed …). I could not run the 2-node test to green in-sandbox: it fails in before() at ECONNREFUSED …:9933 bringing up the replication server — and the existing sibling test fails identically here, so it's the known in-sandbox replication-bind limitation, not this change. It's stress-gated (HARPER_RUN_STRESS_TESTS=1) and a no-op skip until harper#1425 syncs into harper-pro's pinned core; it should be exercised by stress CI after that sync.

Unit layers are green locally on both sides (core blob suite 37 passing incl. the 500/404 cases; this PR's blobReplicationFailure 24 passing). 🤖 (Opus 4.8)