Blobs as a content layer: location-independent blob identity #2146

The architectural gap

Harper's record layer already has a real distributed identity model: a record is named by its primary key, versioned, replicated, and convergent — any node can hold it, serve it, and reconcile it. Blob content has none of that. A blob's identity is a node-private counter, which means blob content doesn't exist as a first-class thing in the cluster — it exists only as a side effect of one node's record pipeline. It can't be named across nodes, asked for again, reconciled, moved, or tiered. Every capability we want from blobs — shared storage (#202), branching (#644), resilient replication (#1645), self-healing serving (#2134) — stalls on that same missing property.

The proposal: give blob content its own identity plane, and make placement a separate concern.

• Identity: (node_id, fileId) — minted exactly once by the node that creates the blob, meaningful on every node, forever. Stored as node-namespaced subtrees ({db}/{node_id}[.{epoch}]/…) with today's dense counters and 4096-way fan-out beneath each namespace.
• Placement: which volumes, nodes, mounts, or tiers hold a copy — a policy decision, resolved locally at read time, never encoded in the reference (today's storageIndex leaves the reference for exactly this reason).

This is the separation storage systems keep converging on — stable names above, movable bytes below — applied with two deliberate deviations from full content-addressing: identity is minted before content finishes streaming (so records can reference in-flight blobs, as they do today), and lifecycle stays slaved to record versions (GC unchanged, no refcounting).

What falls out of stable identity

Transfer becomes idempotent — push demotes from correctness mechanism to optimization. Today the inline push inside the record stream is load-bearing: it happens exactly once, from exactly one sender, and when it fails mid-flight we must either wedge (hold the gap: #1337, #1472, harper-pro#683) or lose (advance past it: the harper-pro#403/#405/#429 lineage). The PENDING stubs, receive-in-flight registry, hasBlobGap, idle watchdogs, and sourceBlobUnavailable taxonomy are all compensations for non-retryability. With stable names, the minting node still pushes proactively (no freshness regression), but a failed push costs nothing: any node pulls (node_id, fileId) from any holder, any number of times, materializing via temp-file + atomic rename. The failure machinery isn't fixed — it's obviated.

Repair becomes convergence. "The set of blobs node X has minted" is an enumerable, single-writer namespace. Reconciling it — after a partition, a botched copy, a disk swap — is anti-entropy by name, per subtree. Because references stop being rewritten per node, record bytes become identical cluster-wide, making content-hash divergence detection meaningful (harper-pro#432) and giving metadata/blob desync (harper-pro#645) a stable name to reconcile against. The repair sweep (harper-pro#385/#388) becomes an automatic fetch-by-id loop instead of a manual record-mapping exercise.

Topology changes become metadata operations. Node dirs are provenance, not ownership: a clone copies them verbatim and mints under its own node_id — zero rewrites. Snapshot-restore bumps an epoch so a restored writer can't collide with its own past or a zombie original. A branch (#644) is just another namespace, dissolving the branched-allocator question. Base copy becomes record store + per-node subtree sync.

Placement becomes a spectrum. Once content is named independently of location, "where do the bytes live" admits answers the current architecture can't express:

• Shared storage (Add support for storing blobs on shared storage #202): a shared mount is the degenerate case — fetch-from-holder is a filesystem read. Single-writer subtrees are also the friendliest workload a distributed filesystem can be given: no cross-node directory contention, trivial split-brain repair.
• Sparse replication: today, replicating a record forces replicating its blob bytes, everywhere. Stable identity permits records-everywhere, content-on-demand — an edge node holds hot blob content only and pulls on miss. For cache-shaped workloads (page bodies, assets) this changes the storage economics of a fleet.
• Tiering: (node_id, fileId) maps one-to-one onto an object-store keyspace. Cold blob content in S3-compatible storage behind the same fetch-on-miss path is an increment, not a redesign.

Serving becomes self-healing. A read miss (#2134) stops being terminal-per-node: the serve path fetches by name from any holder and serves. Blob loss degrades from permanent divergence to cache miss.

Content flow

The mechanism, end to end:

1. Write on A — record commit gated on the blob's durable local write (existing pre-commit gate); the record carries (A, f1).
2. Record replication A→B — ref only. B commits immediately and the cursor advances; record replication never stalls on blob bytes again.
3. Blob arrival at B — A pushes on commit (keeps today's blob-freshness timing); B pulls if the push hasn't landed by record apply. Both funnel into one single-flight fetch per (node_id, fileId).
4. Read at B — local hit serves directly; a miss triggers the same fetch, and the reader tails the file as it materializes (the existing read-while-writing machinery).
5. Reconciliation sweep — periodically re-enqueues any ref without a local file. This makes the system convergent without a durable fetch queue: the store itself is the recoverable queue.

sequenceDiagram
    participant C as Client
    participant A as Node A (origin)
    participant B as Node B (replica)

    C->>A: PUT record {blob}
    A->>A: write blob {db}/A/f1 (durable)
    A->>A: commit record with ref (A, f1)
    par record stream (existing replication)
        A->>B: record + ref (A, f1)
        B->>B: commit immediately, cursor advances
    and blob push on commit (idempotent prefetch)
        A->>B: PUSH blob (A, f1) — pooled mTLS HTTP
        B->>B: temp write, atomic rename
    end
    Note over B: blob absent at record apply?<br/>enqueue PULL (A, f1) — same single-flight fetch,<br/>Range-resumable, retried against any holder

The par block is the decoupling: the two lanes share no ordering, no backpressure, and no failure coupling. A failed push costs nothing — the pull lane and the sweep converge on the same fetch.

The one deliberate weak point: origin loss before the first copy lands (the window between record commit on A and any peer holding the blob). Push-on-commit keeps that window as small as today's inline push; the pull design does not widen it.

The per-blob resolution lifecycle — states, retry and resume semantics, peer selection — is specified in the companion issue #2148.

Why now

Three roadmap directions — shared storage (#202), branched databases (#644), and the blob-wedge/loss family (#1645) — are independently converging on the same missing primitive, and the compensation machinery around the third is still growing (harper-pro#336, #659, #683). Meanwhile fleet operations lean ever harder on clone/rehydrate flows that node-local identity makes fragile. Building the primitive once is cheaper than compensating for its absence in three more places.

Adoption path — each phase independently shippable

1. Read-both. Nodes understand the new reference encoding (a new msgpackr ext type — unknown-type reads fail loudly rather than resolving a fileId in the wrong namespace). Inert.
2. Node-local storage flip. New blobs mint (self_node_id, fileId); the existing tree grandfathers in place as the node's own namespace — ids unchanged, files unmoved. The wire protocol is unchanged — receivers keep re-minting local blobs from the streamed bytes as today — but record sends ship stored value bytes verbatim (GET_RECORD responds with the raw value buffer; the typed-structure sync exists to support raw sends), so the new ref encoding does reach peer decoders. That is why this phase is gated on phase 1 being fleet-complete: read-both is the compatibility mechanism. Shared storage (Add support for storing blobs on shared storage #202) is fully enabled at this phase. Downgrade past this point fails loudly, which is the correct failure mode.
3. Wire v2, per-link. Send refs; push-as-prefetch; pull-on-miss. Negotiated per link (today's handshake is an exact-match harperdb-replication-v1 subprotocol, so a small capability mechanism is a prerequisite); v1 links keep today's inline behavior via sender down-conversion. The wedge/lose dilemma disappears link by link.
4. Retire the compensations; light up placement. Stubs, gap tracking, and the receive registry become dead code; base-copy-by-rsync, fetch-on-miss serving, sparse replication, and tiering become available as policy.

The absence rule needs no new infrastructure at any phase: blob lifetime is already driven exclusively by record supersession, so fetch while the current record version references the blob; stop when a newer version drops the ref. Missing-while-referenced is by definition the loss case, now answered by N holders instead of one.

Alternatives

• Snowflake IDs (Add support for storing blobs on shared storage #202 thread): the same shape — (machine, unique-within-machine) — with identity hidden in bits. Machine-id assignment remains; snapshot-restore can re-mint from the pre-snapshot clock window (the epoch hazard, implicit instead of explicit); 63 bits force BigInt through hot paths; sparse timestamp ids break low-bits directory sharding (the per-ms sequence bits are ~always zero); existing integer ids can't be absorbed. Node namespaces keep dense counters, short dirs, no clocks, and grandfather the tree in place.
• Content addressing: dedupe and integrity for free, but breaks mint-before-write (records reference in-flight blobs), adds GB-scale hashing to the write path, and forces refcounting GC immediately.
• Record-derived ids: reverse lookup for sweeps, but coexisting versions and cross-node version ties force a version+tiebreak component — a counter with extra steps. The reverse-lookup win comes cheaper by recording the owning record key in the blob header.
• Keep patching the push path: the compensation list is the cost curve; each new failure mode buys another stub/registry/watchdog because the transfer stays non-retryable.

Scope boundaries

Blob lifetime bugs (#1606, #595, #1832, #708, #2062, the retention half of #2134) are GC-policy issues this proposal doesn't change — but their worst case downgrades from permanent divergence to recoverable re-fetch. GC stays record-driven and 1:1; cross-record sharing/dedup becomes possible but is out of scope. Prerequisites: the wire-capability mechanism (phase 3) and an epoch hook in the restore path (a namespace lock file fences zombies on shared mounts).

Companion issue: #2148 — the resolution/serving layer (blob store as an internal Resource, owned-relationship model, pluggable sources).

Related: #1645, #202, #644, #1337, #1472, #2134, #141; harper-pro#385, #388, #336, #683, #659, #645, #208, #432.

Priority: low — design/direction proposal, no immediate action required. Drafted with AI assistance (Claude Code).