Give the #1854 RocksDB oracle a checkpoint it cannot race

[kriszyp]

Before #2429, integrationTests/database/delete-index-atomicity-rocksdb.test.ts intermittently failed across the integration matrix because its independent readOnly: true oracle opened the live RocksDB directory while compaction could unlink an SST named by the MANIFEST. #2429 stopped the bleeding with a bounded error-signature retry; this rebased PR replaces that retry with an immutable checkpoint, removing the race by construction while preserving the raw-storage oracle and every Arm A / Arm B assertion.

/Snapshot/ publishes the checkpoint through createCheckpoint(), and openDbi() refuses any live-directory fallback. Nothing writes to a checkpoint, so there is no missing-SST open race, error matcher, retry budget, or follow-up needed when HarperFast/rocksdb-js#812's secondary-mode open lands. Taking the checkpoint also flushes the memtable required by this disableWAL fixture and captures all column families through one operation instead of separate read-only opens at unrelated instants; the suite is quiescent whenever it refreshes the oracle.

refreshOracle() closes handles and removes the previous checkpoint before publishing the next, bounding pinned hardlinks to one. It retains a returned path for recursive teardown only when it exactly matches the expected per-sequence checkpoint, so a fixture regression fails before any handle opens it without risking removal of an invalid location. The target is derived fixture-side from the store and sits beside database/, outside the directories scanned as databases.

The snapshot proof pins the raw handle's .path to the checkpoint, proves that the checkpoint contains the preceding unflushed write, stays frozen after the live database advances, and is replaced by a refresh that sees the new write. The oracle remains a second @harperfast/rocksdb-js handle decoding composite [indexedValue, primaryKey] keys directly; no Harper query surface substitutes for the index-side observation.

For the human reviewer

1. Checkpoint instead of the now-merged live-open retry. This rebase deliberately removes Deflake the #1854 RocksDB oracle: reopen when a live compaction removes an SST mid-open #2429's bounded classify-and-reopen loop rather than stacking both mechanisms. Keeping Deflake the #1854 RocksDB oracle: reopen when a live compaction removes an SST mid-open #2429 is smaller, and waiting for readOnly:true open races a live writer's compaction and reports a healthy database as corrupt; no OpenAsSecondary alternative exists rocksdb-js#812 would avoid fixture checkpoint scaffolding; the checkpoint is chosen because it eliminates the race and error-shape coupling now, with no measurable suite-time cost. Reversal is localized to the fixture route and oracle helpers. Design framing cleared before implementation: Framing-Verdict: chosen-approach-sound.
2. Checkpointed index entries compared with live HTTP primary liveness. Decoding the primary column family from the same checkpoint would avoid split-time observation, but audited deletes store null tombstones under the same key at a variable payload offset. hasLiveRecord() therefore uses a direct primary-key HTTP lookup while the index side stays raw; this is safe because the arms issue no client traffic between refreshOracle() and findPhantoms(). A future background workload in this fixture would require revisiting that assumption.
3. Remove the old checkpoint before creating the new one. Publishing first would preserve the last snapshot if refresh fails and improve failure forensics; removing first keeps hardlinked SSTs bounded to one checkpoint. The harness uses a fresh temporary data root per run, and the choice is easy to reverse if debugging value outweighs the disk bound.

Verification

Route: the changed integration test against live Harper/RocksDB instances, plus its client-surface companion anchor.

• The original stress campaign reproduced the base failure twice in 36 runs at 12-way parallelism, then completed 144/144 at the reproducing load, 36/36 at 6-way, and 12/12 at 3-way with the checkpoint implementation: 192 runs, zero race signatures, with 36-run wall time effectively unchanged (114 seconds on base versus 115–121 seconds with checkpoints).
• After rebasing onto main with Deflake the #1854 RocksDB oracle: reopen when a live compaction removes an SST mid-open #2429, npm run build passed and npm run test:integration -- "integrationTests/database/delete-index-atomicity-rocksdb.test.ts" passed three times at the final behavior: 9/9 each, 27/27 total. These runs also settle the reviewed question that the pinned rocksdb-js binding creates the checkpoint parent.
• npm run test:integration -- "integrationTests/resources/audit-false-delete-rollback.test.ts": 2/2 passed.
• npx prettier --check integrationTests/database/delete-index-atomicity-rocksdb.test.ts integrationTests/database/delete-index-atomicity-rocksdb/resources.js, npm run lint:required, and git diff --check: passed.
• The earlier fix: thread {transaction} into removeEntry for audit:false table deletes #1869 sensitivity run remains applicable: reverting the transaction option made Arm A and Arm B on ItemF fail with the expected orphan while the audit:true controls and oracle proofs stayed green; the rebase preserves those assertions unchanged.
• test:integration:all, test:unit:main, and test:unit:resources were not rerun for this test-only rebase; CI remains the full-suite signal.

Refs #1854, #2400, HarperFast/rocksdb-js#812

Complexity: medium

_{Review-Coverage: authored=codex; ran=cursor-composer,claude; adjudicated=domain; blocked=gemini(auth); declined=cursor-grok; rounds=2 @ c4d173b}

_{Human-Review-Need: 3 (decisions: checkpoint-over-live-retry, snapshot-index-live-primary, replace-before-create) @ c4d173b}

[gemini-code-assist]

Code Review

This pull request updates the RocksDB delete-index atomicity integration tests to use RocksDB checkpoints instead of opening the live database directory directly, preventing compaction races that lead to MANIFEST corruption. It introduces a /Snapshot/ endpoint to create checkpoints and adds an 'oracle proof' test to verify that the oracle reads from an immutable snapshot. Feedback on these changes suggests parsing the snapshot response body once instead of twice in the test file, and strictly validating the sequence parameter type in the snapshot resource to avoid overly permissive coercion by Number().

[claude]

Reviewed; no blockers found.