Repository navigation
Base-copy replication wedges in Receiving when sender has records referencing blob files missing on disk #1337
Description
Activity
- changed the title
[-]Rolling v4→v5 upgrade silently drops blob-bearing records on receivers with stale shared structures[/-][+]v4→v5 rolling upgrade: msgpackr sharedStructures not persisted for data tables[/+]on Jun 17, 2026 - changed the title
[-]v4→v5 rolling upgrade: msgpackr sharedStructures not persisted for data tables[/-][+]Base-copy replication resync for blob-bearing databases never completes — receivers stuck in `Receiving` state, records silently dropped[/+]on Jun 17, 2026 Diagnosis: the trigger is application code calling
blob.save(), which isn't a Harper API — not a core defect.Pulled the logs from the affected cluster (
cluster:v4-to-v5 customer:akamai). The whole thing is one base-copy resync window (2026-06-17 00:47–02:37 UTC), and the order of events is:TypeError: blob.save is not a function— 1,266×, onse-sto, thrown while resolving PageCache records (the cache source-resolution path inTable.ts), immediately before the timeout storm.- Then
Timeout waiting for blob stream …— 242,257×, including anuncaughtExceptionvariant.
blob.save()is not a method on Harper's blob type — it doesn't exist anywhere in core / harper-pro / legacy harperdb (only test files reference it). So this is application/component code (the PageCache resolver) calling a non-existent method on a blob it gets back from a v4→v5-migrated record. That throw aborts the record's apply before the blob's save stream is wired up, which is what leaves the receiver unable to finish those blobs and the base-copy unable to converge.So the record loss here is downstream of an application bug. The fix is for that component to stop calling
blob.save()— blobs are persisted as part of the normal record write, there's no explicit save to call. Nothing for Harper to change to make the trigger go away.What we are hardening (#409): the orphaned blob stream that the failed apply leaves behind was being torn down by the blob-timeout sweep with no
'error'listener, which Node promoted to a process-leveluncaughtException— that's the 242k storm, and it can destabilize the node well beyond the one bad record. #409 makes the receiver absorb that teardown so a blob that errors / never saves (for any reason, app bug included) degrades gracefully instead of crashing the process. It deliberately does not make the copy converge while the app keeps callingblob.save().— Claude (Opus 4.8)
- changed the title
[-]Base-copy replication resync for blob-bearing databases never completes — receivers stuck in `Receiving` state, records silently dropped[/-][+]Base-copy replication wedges in `Receiving` when sender has records referencing blob files missing on disk[/+]on Jun 17, 2026 - added a parent issue
on Jul 7, 2026
Summary
When a v5 node attempts a base-copy resync to a peer, and the sender's primary store contains any record whose
Blobfield references a file that doesn't exist on the sender's disk, the receiver's connection wedges permanently inlastReceivedStatus: "Receiving". Base-copy never reachesCOPY_COMPLETE, the watermark never advances, the blob-bearing database's replication is fully stuck. No errors at the API level; symptom appears only as silent record propagation loss between specific peer-pairs.Minimal reproduction
Blobfield, insert N records with content aboveFILE_STORAGE_THRESHOLD(8192 bytes) so blobs go to disk files<dataRoot>/blobs/<db>/cluster_status.connections[0].database_sockets[].lastReceivedStatusstays at"Receiving"indefinitely; only records before the gap applyMigration is NOT required for reproduction — the bug fires on any v5 source whose primary store references a missing blob file. We initially suspected the v4→v5 migration path but ruled it out: migration logs are clean, and the reproduction does not use a v4 binary.
Test results
Receiving(stuck)Test asserts post-fix behavior. Same setup, same parameters, deterministic.
Cross-references