Blob × replication regression tests (Tier 1: deterministic cluster integration) #411

Motivation

A cluster of production incidents on the JJill preprod Fabric cluster (and others) all share one shape that our test suite does not cover: blobs × TTL/eviction × replication/base-copy × failure-and-recovery. Each was found in the field, not CI:

• Cache expiration scan orphans blob files (unlink on fixed deletionDelay timer, decoupled from RocksDB commit) harper#1364 — orphaned blob reference created two ways: (1) failed inbound blob transfer + TTL eviction on the origin, (2) expiration-scan force-commit unlinking a blob before the record delete commits.
• fix(blob): tolerate source-unavailable blobs in pre-commit so a missing blob can't wedge replication harper#1353 — a source-missing blob wedges replication in pre-commit (ENOENT).
• harper-pro#409 / Base-copy replication wedges in Receiving when sender has records referencing blob files missing on disk harper#1337 — an orphaned receive-side blob stream crashing the process.
• harper-pro#403 — repeated blob ENOENT permanently pinning the replication resume cursor.
• The JJill incident itself: orphaned blobs → base-copy stream stalls permanently on ENOENT → 100% backpressure → replication wedge → downstream deploy failures.

Today integrationTests/cluster has blob fault-injection fixtures (fixture-blob-fail-injector, fixture-blob-fail-transient, fixture-large-blob-authoritative) driving blobSaveRejectionContainment.test.mjs. That proves containment (a failed inbound save doesn't crash the receiver). It never asserts convergence or no-orphan. This issue closes that gap with deterministic, per-PR regression tests; harper-pro#412 covers the randomized chaos/stress tier.

The invariant oracle (shared by both tiers)

After an op sequence + fault injection, quiesce, then on every node assert:

1. No orphaned references — for every record carrying a blob, the blob is readable and its content-hash equals the source's.
2. Convergence / liveness — all nodes reach an identical (primaryKey → blobHash) set within a bound; cluster_status shows blobReplicationFailures stabilized and the resume cursor advancing (never permanently pinned).
3. Bounded orphan files — leaked blob files (the safe direction of #1364: a skipped unlink) are reclaimed by cleanupOrphans; blob-store file count does not grow unbounded across cycles.
4. Isolation — system/deploy replication stays live while a data-DB blob copy is backpressured/wedged.

A small shared assertion helper implementing (1)–(4) should be the first deliverable; every scenario below ends by calling it.

Scope — deterministic cluster integration tests (one per failure mode)

Small, seeded, fixed fault points (in the style of the existing .test.mjs). Extend the injector as needed beyond receive-side write-fail to also: skip the blob transfer entirely (commit metadata, never send the blob), delete a blob file out from under a live record, and drop blob chunks mid-stream.

• Shared invariant-oracle helper (items 1–4 above), usable from cluster tests.
• Base-copy past a missing blob (the JJill wedge, highest value): create records whose blobs are missing everywhere, trigger a full base-copy (node restart/join), assert the copy advances past them and the cluster converges — does not stall.
• TTL-orphan race (#1364 Path 1): write short-TTL blob records on an edge node, fail the transfer to the leader, let TTL evict on the edge, trigger a copy → assert convergence + no surviving orphaned reference.
• Expiration force-commit (#1364 Path 2): short TTL + volume sufficient to trip the long-transaction force-commit (storage.debugLongTransactions: true to observe) → assert blob files are not unlinked under records that still exist.
• Resume cursor not pinned (Blob replication wedges permanently when source blob is gone (ENOENT) on an expiration cache table — held resume cursor never recovers #403): inject repeated source-missing-blob ENOENT → assert the cursor advances past (loud) and transient failures hold-then-recover.
• Deploy-under-backpressure (the JJill coupling): wedge/backpressure a data-DB blob copy, then deploy_component → assert it still replicates to peers (system-table replication isolated from data-DB blob backpressure).

Where it lives / reuse

• harper-pro/integrationTests/cluster/ — multi-node harness + the existing blob fixtures.
• Extend fixture-blob-fail-injector with the new injection modes (env-toggled, same pattern).
• Engines: parameterize across RocksDB and LMDB where feasible (the orphan paths differ subtly by engine).

Acceptance criteria

• Each scenario above is a deterministic test that fails on the pre-fix code and passes on current main (verify the regression value — e.g. base-copy-past-missing-blob should hang/fail without fix(blob): tolerate source-unavailable blobs in pre-commit so a missing blob can't wedge replication harper#1353).
• Runs in the normal per-PR integration job (no nightly-only dependency).
• The oracle helper is reused by all scenarios and by the Tier-2 chaos harness.

Related: HarperFast/harper#1364, HarperFast/harper#1353, HarperFast/harper#1369, harper-pro#409, harper-pro#403. Chaos/stress tier: harper-pro#412.