RocksDB read snapshot leaks permanently: registry holds a strong ref to TransactionHandle, GC finalizer never calls close(), and DatabaseTransaction.abort() cannot release a save()-created transaction #2107

Activity

1. harper-joseph commented on Aug 5, 2026

   

   harper-joseph

   on Aug 5, 2026

   ContributorAuthor

   More actions

   Root cause found — three interacting defects in core/resources/DatabaseTransaction.ts (v5.1.26)

   Correcting the original report first: the snapshot is not taken at process start and it is not
   engine-wide. Per-database oldestSnapshotTime:

   node A (uptime 115.4h)
     render_schedule   held 114.31 h   keysWritten 12,804,581   reseeks 43,647
     page_cache        held  60.46 h   keysWritten 12,355,032   reseeks    942
     coordination / crawl_stats / render_service / sitemaps  ->  0 (none held)
   
   node B (30 min uptime)
     all six databases ->  0 (none held)
   

   So a snapshot is leaked at an arbitrary later time, only on the two highest-write databases, and
   never released. page_cache leaked 60h ago but shows few reseeks because its access is point reads;
   render_schedule is range-scanned ~4×/sec, hence 43,647 reseeks and the 21× scan degradation.

   Defect 1 — getReadTxn() resets the watchdog timer on every call (line 139)

   getReadTxn(): ReadTransaction {
       this.readTxnRefCount = (this.readTxnRefCount || 0) + 1;
       this.timeout = txnExpiration;    // <-- reset the timeout, on EVERY call
       if (this.transaction) { ... return this.transaction; }

   The monitor decrements txn.timeout -= txnExpiration once per interval and only acts when
   txn.timeout <= 0. So any transaction touched more often than STORAGE_MAXTRANSACTIONOPENTIME
   (default 30 s) is immortal from the watchdog's perspective. On a table read ~4×/sec that is
   guaranteed, permanently.

   This is the load-bearing defect: it explains why the leak lands only on hot databases, why it starts
   at an arbitrary time, and why it never recovers. A read snapshot's cost is a function of its
   absolute age, not of idleness — but the watchdog only bounds idleness.

   Defect 2 — the read-only release path is commented out (lines 292–308)

   this.open = TRANSACTION_STATE.CLOSED;
   if (--this.readTxnsUsed > 0) {
       // we still have outstanding iterators using the transaction, we can't just commit/abort it
       if (this.writes.length > 0) { this.open = TRANSACTION_STATE.LINGERING; }
       /*
       commitResolution = this.writes.length > 0
           ? transaction?.commit({ renewAfterCommit: true })   // CommitAndTryCreateSnapshot
           : null;   // don't abort, we still have outstanding reads to complete
       */
   } else {
       trackedTxns.delete(this);
       this.transaction = null;    // <-- the ONLY path that frees the snapshot

   For a read-only transaction with readTxnsUsed > 1 this branch does nothing at all —
   writes.length is 0 so not even LINGERING is set — while line 290 has already marked it CLOSED.
   commit() has no state guard (line 273 only throws on timedOut), so it would self-heal on the
   next tick by decrementing to 0 — but Defect 1 means there is no next tick.

   Contributing asymmetry: getReadTxn() sets readTxnsUsed = 1 by assignment (line 152) while
   useReadTxn() then increments it (line 164), so the resting value after all users finish is 1,
   not 0. Final release therefore depends on disregardReadTxn()'s compound condition
   (line 183: --readTxnRefCount === 0 && readTxnsUsed === 1) — and only 2 of ~10 getReadTxn()
   call sites ever call disregardReadTxn() (Table.ts:744 and 1240; the sites at 715, 1206, 1671,
   1867, 2538, 4038, 4046, 4075, 4091, 4305 do not). For request-scoped transactions commit()/abort()
   force-drains and this is harmless; for a reused long-lived transaction it is not.

   Defect 3 — the read-only watchdog branch is silent (lines 611–651)

   if (txn.hasPendingWrites() && !txn.sourceApply && !txn.isReplay) {
       harperLogger.error(`Transaction was open too long and has been aborted ...`);   // logs
       txn.abortDueToTimeout();
   } else {
       // Read-only long transaction ... commit to close out the snapshot
       txn.commit();          // no logging whatsoever
       txn.timeout = txnExpiration;
   }

   Only the write-bearing path logs. A read-only snapshot held for 114 hours produces zero operator
   signal — grep 'open too long' over 24h of logs on the affected node returns 0. This is why the
   condition went unnoticed until range-scan latency became user-visible.

   Suggested fixes

   1. Bound absolute age, not idleness. Stop resetting timeout in getReadTxn() when
      this.transaction already exists, or track createdAt separately and have the monitor act on
      now - createdAt for read snapshots. Without this, nothing else matters.
   2. Implement the read-only release. When writes.length === 0 and the watchdog fires, the
      snapshot can be dropped outright — in-flight iterators should hold their own reference rather than
      relying on the shared readTxnsUsed base of 1. Either finish the commented-out
      renewAfterCommit path or abort() + null the transaction for the read-only case.
   3. Log the read-only branch (warn, with db.name and age). A silent 114-hour snapshot should be
      impossible to have unnoticed.
   4. Surface it as health. oldestSnapshotTime is already in system_information.metrics; nothing
      alerts on it. Age of oldest snapshot per database, plus trackedTxns.size (already exported at
      line 663), would have made this a five-minute diagnosis instead of a multi-hour one.
   5. Unify acquire/release. Make every getReadTxn() site pair with disregardReadTxn(), or
      replace the two counters with a single scoped helper. Two counters with an intentional off-by-one
      base and ~8 unpaired call sites is very hard to reason about.

   Note STORAGE_DEBUGLONGTRANSACTIONS would not have caught this either: it captures start stack
   traces but they're only printed in the abort path, which read-only transactions never reach.

   What we have not identified

   Which caller holds the leaked transaction. Given txnForContext(context) (Table.ts:4038/4305), the
   likely shape is a long-lived (non-request-scoped) context whose DatabaseTransaction is reused, but
   we have not proven that from outside the process. getTrackedTxns() is exported but not reachable via
   the ops API — if you can suggest a way to dump it on a live node we can identify the holder on the
   one node still exhibiting this (uptime 4d19h, 525 ms scans, snapshot held 114 h).
2. harper-joseph

   changed the title [-]RocksDB snapshot pinned at process start is never released — obsolete versions are never reclaimed, degrading secondary-index range scans 21x over ~5 days[/-] [+]Read snapshot leaks on high-write databases and is never released: getReadTxn() resets the long-txn watchdog on every call, and the read-only release path is commented out[/+] on Aug 5, 2026
3. harper-joseph commented on Aug 5, 2026

   

   harper-joseph

   on Aug 5, 2026

   ContributorAuthor

   More actions

   Traced to the leak site: the storage-reclamation handler leaks the snapshot that then blocks reclamation

   Correcting my previous comment: it is not directCommitSync/replay. Replay ran for six
   databases in ~25 seconds at boot (render_service, signals, sitemaps, system included), and
   four of those hold no snapshot — so replay is not the discriminator. (directCommitSync() at
   DatabaseTransaction.ts:554 is still a real bug — it does trackedTxns.delete(this) and
   this.transaction?.commitSync() but never this.transaction = null, unlike the correct release
   path at 309-312, so the object keeps serving an already-committed transaction from getReadTxn()'s
   early return and is invisible to the watchdog. Just not this leak.)

   The correlation that holds

   Node uptime 115h. Boot 2026-08-01T03:44:55Z. RECLAMATION_INTERVAL = 1h default.

   event	time
   boot	03:44:55
   first hourly reclamation would fire	~04:44:55
   render_schedule snapshot created (still held)	04:45:25
   purgeLogs starts throwing hourly (Error: Database not open)	2026-08-02T19:34
   reclamation failure	2026-08-03T10:34:13
   JavaScript execution has taken too long ...	2026-08-03T10:36:10
   page_cache snapshot created (still held)	2026-08-03T10:36:28

   render_schedule leaked 30 seconds into the first reclamation cycle after boot. page_cache
   leaked ~2 minutes after a reclamation failure. Both are per-database snapshots; the four
   databases that reclamation does not touch hold none.

   Why reclamation is running at all here

   It only runs under disk pressure:

   const RECLAMATION_THRESHOLD = envMgr.get(...) ?? 0.4;   // 40% free required
   const priority = RECLAMATION_THRESHOLD / availableRatio;
   if (priority > 1 || previousPriority > 1) { ... handler(...) }

   This node's data volume is at 75% (427G/576G), so availableRatio = 0.26 and
   priority = 0.4 / 0.26 = 1.54. Handlers therefore fire every hour, indefinitely.

   Three defects in that path

   A. purgeLogs() is neither awaited nor guarded — auditStore.ts:164-172:

   function scheduleAuditCleanup(newCleanupDelay?: number): Promise<void> {
       if (isReadOnlyMode()) return;
       if (auditStore instanceof RocksTransactionLogStore) {
           auditStore.rootStore.purgeLogs({ before: ... });   // not awaited, not try/caught
           return;
       }
       ...

   Note the non-RocksTransactionLogStore branch below it correctly passes snapshot: false to
   getRange — the hazard is understood there, but the purgeLogs branch has no equivalent care.
   purgeLogs throws Error: Database not open from rocksdb-js database.ts:749, and whatever
   snapshot/iterator it established before throwing is never released.

   B. One throwing handler skips every remaining handler for that path —
   storageReclamation.ts:88-108: the try/catch wraps the entire for (const entry of handlers)
   loop, not each handler. So a single failure aborts the rest of that path's reclamation for the
   cycle, every cycle. Logged once per cycle as Error running storage reclamation handlers.

   C. The leak is self-perpetuating. The leaked read snapshot is precisely what forbids RocksDB
   from discarding obsolete versions — so the mechanism whose job is reclamation has permanently
   disabled reclamation for that database. Every subsequent hourly attempt can only fail.

   Combined with the three defects from my previous comment (watchdog timer reset on every
   getReadTxn(), the commented-out read-only release, and the silent read-only watchdog branch), the
   snapshot is immortal and invisible.

   Downstream impact, already measured

   3.9 keys per live row after 5 days, numberReseeksIteration 43,647 vs 51 on a fresh node, 5.3B vs
   27.8M SST reads, dbSeekMicros median 256ms vs 35ms, and bounded range scans 21x slower — which
   saturates a serialized job-claim mutex and produced multi-second latency for end users.

   Suggested fixes, in priority order

   1. Guard and await purgeLogs() in scheduleAuditCleanup, and ensure it releases its
      snapshot/iterator on the throw path. Error: Database not open is a foreseeable state (closing
      during shutdown/reload) and should not leak.
   2. Move the try/catch inside the handler loop in runReclamationHandlers so one bad handler
      doesn't cancel the rest.
   3. Then the four items from the previous comment (absolute-age watchdog, implement the read-only
      release, log the read-only branch, surface oldestSnapshotTime as health).

   Operational note for anyone hitting this

   Because the trigger is availableRatio < STORAGE_RECLAMATION_THRESHOLD, a node under disk pressure
   leaks on the first reclamation cycle and then degrades on a ~5-day clock, while a node with plenty
   of free space never runs the handler and never leaks. That likely explains why this hasn't been
   widely reported. Reclamation on the affected node is throwing anyway, so it is reclaiming nothing —
   lowering the threshold below the actual free-space ratio stops the hourly leak-inducing runs without
   losing any reclamation that is currently working. (Untested; we have not changed it.)

   Environment: harperfast/harper-pro:5.1.26, 4-node Fabric cluster, Node v24.18.1.
4. harper-joseph commented on Aug 5, 2026

   

   harper-joseph

   on Aug 5, 2026

   ContributorAuthor

   More actions

   Retraction of the reclamation theory, and the verified trigger

   Please disregard my previous comment's reclamation analysis. Two measurement errors:

   1. I read the wrong filesystem. Reclamation checks the ratio for the Harper root path; on all four
      nodes that is 80% free, so priority = 0.4 / 0.80 = 0.50, which is < 1 — the reclamation
      handlers never run. My "75% full → priority 1.54" came from the container's data mount, not the
      path reclamation actually stats.
   2. The timing didn't hold up. I generalised from one node. Across four:

   node	boot	snapshot created	offset
   yc0	19:00:13	19:01:01	boot +48s
   v3t	19:42:10	19:42:56	boot +46s
   cd5	08-01 03:44:55	04:45:25	boot +1.01h
   e9v	22:53:08	none	—

   So it is not the hourly reclamation cycle. (The purgeLogs unguarded-call and the
   runReclamationHandlers try/catch-scope observations still stand as independent code smells, and
   directCommitSync's missing this.transaction = null is still a real bug — none of them is this.)

   The actual trigger: the watchdog fires while a long-running READ is still in flight

   Every node emits JavaScript execution has taken too long and is not allowing proper event queue cycling at boot+51–59s (some startup scan). The warning is logged after the blocking window
   ends, so:

   node	block warning	snapshot created	inside the block?
   yc0	19:01:03 (boot+51s)	19:01:01	yes — 2s before the warning
   v3t	19:43:08 (boot+59s)	19:42:56	yes — 12s before the warning
   e9v	22:54:07 (boot+59s)	none	watchdog tick missed the window
   cd5	10:36:10 ([http/81])	10:36:28	yes — 18s after that warning

   All three leaks were created inside an event-loop-blocking window. e9v hit the same startup block and
   did not leak — which is what makes this a race rather than a deterministic path, and matches the
   ~30s STORAGE_MAXTRANSACTIONOPENTIME tick against a block of a few seconds.

   The sequence, against the code

   1. A long-running read-only operation is in flight, blocking the event loop.
   2. The long-transaction monitor fires (txn.timeout <= 0).
   3. hasPendingWrites() is false (read-only, and no writes anywhere in the .next chain), so it takes
      the silent else branch: txn.commit().
   4. In commit(), this.open = TRANSACTION_STATE.CLOSED is set, then --this.readTxnsUsed > 0
      because the iterator is still in flight. That branch does nothing for a read-only transaction —
      writes.length === 0 so not even LINGERING — and the intended release is the commented-out
      block at lines 302-308.
   5. this.transaction still holds the RocksDB snapshot, and the txn is still in trackedTxns.
   6. Every subsequent getReadTxn() sets this.timeout = txnExpiration (line 139), so the watchdog
      never gets another chance. On a table read ~4x/sec that is permanent.
   7. Nothing is logged, because step 3 took the branch that has no logging.

   That is why a restart is the only recovery, and why it is invisible: an immortal read snapshot with
   no operator signal.

   Why only the hot databases

   The leak needs a long read to coincide with a watchdog tick, so the exposure scales with how much
   long-scan activity a database sees. On this deployment render_schedule (range-scanned ~4x/sec by a
   job-claim query) and page_cache (1.6M rows, blob-backed) leaked; render_service, sitemaps,
   crawl_stats and coordination never did in 5 days.

   Fixes, revised priority

   1. Implement the read-only release in commit() (lines 292-308). When writes.length === 0
      there is nothing to preserve for correctness — either finish the renewAfterCommit path or
      abort() and null this.transaction, and let in-flight iterators hold their own reference
      instead of relying on the shared readTxnsUsed base of 1.
   2. Don't reset the watchdog timer on an existing transaction (line 139). Bound absolute age for
      read snapshots, not idleness — otherwise any recovery path is unreachable, as here.
   3. Log the read-only watchdog branch. A 114-hour read snapshot must not be silent.
   4. Surface oldestSnapshotTime age per database as health. It is already in
      system_information.metrics; nothing alerts on it. This would have been a five-minute diagnosis.

   Reproduction hint

   Hold a long read-only range scan open across a STORAGE_MAXTRANSACTIONOPENTIME boundary on a table
   receiving concurrent writes, then keep reading that database. oldestSnapshotTime for it should stop
   advancing, estimateNumKeys should climb well past the live row count, and
   numberReseeksIteration should grow without bound (we measured 43,647 vs 51 on a fresh node, with
   bounded limit 20 range scans going 33ms → 703ms over five days).

   Environment: harperfast/harper-pro:5.1.26, Node v24.18.1, 4-node Fabric cluster.
5. harper-joseph commented on Aug 6, 2026

   

   harper-joseph

   on Aug 6, 2026

   ContributorAuthor

   More actions

   Decisive narrowing: the snapshot holder is NOT a tracked DatabaseTransaction

   We got worker-thread debugger access on two nodes that were actively holding a leaked snapshot, with
   no restart and no config change, and enumerated trackedTxns on every thread.

   yc0   uptime 5h21m   17 live inspectors   render_schedule snapshot HELD since 19:01:01
   v3t   uptime 4h39m   17 live inspectors   render_schedule snapshot HELD since 19:42:56
   

   Result on yc0 — all 17 threads, tid 0 through 16, zero evaluation errors:

   port 9229  tid=0   trackedTxns=0
   port 9230  tid=1   trackedTxns=0
   ...
   port 9245  tid=16  trackedTxns=0
   total tracked transactions across all threads: 0
   

   render_schedule has held a snapshot for 5h20m, and no tracked transaction on any thread holds a
   transaction object. This is a true negative, not a silent failure — every thread evaluated cleanly and
   reported its real threadId.

   What this eliminates

   1. Any tracked DatabaseTransaction. The holder is outside trackedTxns entirely.
   2. The long-transaction watchdog as a factor. startMonitoringTxns only iterates trackedTxns, so it
      can never see this holder. That alone explains the complete absence of log signal — my earlier
      "the read-only branch doesn't log" theory isn't needed and, more importantly, my proposed
      mechanism (watchdog fires mid-read → commit() → --readTxnsUsed > 0 → do-nothing branch) is
      wrong, because that path leaves the transaction in trackedTxns.
   3. ImmediateTransaction. It bypasses tracking, but its getReadTxn() returns undefined by
      design, so it holds no snapshot.
   4. directCommitSync / crash-recovery replay. Retracting this too. All four nodes replayed at boot,
      and the leaking nodes leaked a database that was not replayed on that boot:

   node	databases replayed at boot	leaked
   yc0	sitemaps (65)	render_schedule
   v3t	system (77)	render_schedule
   e9v	sitemaps, system	none
   cd5	coordination, crawl_stats, page_cache (6078), render_service, sitemaps	none

   cd5 replayed 6,078 page_cache records and holds no page_cache snapshot. Replay is not the
   discriminator. (directCommitSync() missing this.transaction = null at DatabaseTransaction.ts:554
   is still a real bug — it removes from trackedTxns while retaining the transaction — just not this one.)

   5. Storage reclamation (retracted in the previous comment: the Harper root path is 80% free, so
      priority = 0.50 < 1 and the handlers never run).

   What is solidly established

   1. Bounded range scans on the affected table degrade 21x over ~5 days; point reads and
      primary-key range scans are unaffected (2.5ms on an aged store vs 2.6ms fresh), and a low-churn
      secondary index is unaffected too. So it is specific to a high-churn secondary index.
   2. The cause is retained obsolete versions: estimateNumKeys 1,901,091 for ~490k live rows (3.9/row),
      numberReseeksIteration 43,647 vs 51 on a fresh node, sstReadMicros count 5.3B vs 27.8M,
      dbSeekMicros median 256ms vs 35ms, liveSstFilesSize 362MB vs 98MB.
   3. Cost is independent of limit (20 rows and 200 rows both ~712ms) and localised to a key range
      containing zero live rows — the scan is skipping dead versions, not retrieving data.
   4. The retention is a held RocksDB snapshot (oldestSnapshotTime), per-database, appearing at
      arbitrary times, and held by something outside DatabaseTransaction tracking.
   5. A full process restart clears it (703ms → 88ms, store 675MB → 377MB, reseeks 43,647 → 18).
   6. Render throughput is unaffected throughout — renders/hour held at 11,000–12,700 across claim
      latencies spanning 9ms to 661ms on four nodes over 17 hours. The damage is tail latency, not capacity.

   What's left

   A store-level snapshot with no DatabaseTransaction behind it — most plausibly an abandoned lazy
   getRange iterator, which is exactly the hazard the snapshot: false comments throughout Table.ts,
   blob.ts and auditStore.ts guard against ("don't hold a read transaction this whole time", "we don't
   want to keep read transaction snapshots open"). Something acquires one without that guard and never
   drains or closes it.

   Next step from our side: Runtime.queryObjects against the rocksdb-js transaction/iterator prototypes
   to enumerate live snapshot holders directly on a node that is currently leaking. We have two such nodes
   with full worker-thread debugger access, so this is reachable — we'll report back.

   If anyone on your side can point at which store-level read paths can acquire a snapshot outside
   DatabaseTransaction, that would shortcut it considerably.

   Unrelated bug found while getting worker access (worth its own issue)

   core/server/threads/threadServer.js registers the inspector-close handler only for the main thread:

   if (isMainThread) {
       const closeInspector = () => { try { require('inspector').close(); } catch ... };
       for (const signal of ['SIGINT','SIGTERM','SIGQUIT','exit']) process.on(signal, closeInspector);
   } else {
       port = startingPort + getWorkerIndex();
   }
   try { require('inspector').open(port, host, waitForDebugger); }
   catch (error) { harperLogger.trace(`Could not start debugging on port ${port} ...`); }   // TRACE

   Worker threads never close their inspector, so when a worker generation is replaced the ports stay
   bound and the new workers fail with address already in use — logged at trace, therefore invisible.
   Observed 96 times on one node. Consequence: worker-thread debuggability is permanently lost after any
   worker replacement until a full process restart, silently. That cost us hours tonight.

   Environment: harperfast/harper-pro:5.1.26, Node v24.18.1, 4-node Fabric cluster.
6. harper-joseph commented on Aug 6, 2026

   

   harper-joseph

   on Aug 6, 2026

   ContributorAuthor

   More actions

   ROOT CAUSE FOUND — orphaned native TransactionHandle pinned by the rocksdb-js registry

   Two independent defects compose. Either one alone would be survivable; together they turn a JS-level
   slip into a permanent, silent, unrecoverable-without-restart resource leak.

   Verified against the exact deployed code: harper-pro v5.1.26, core submodule
   43a375fd889f2395f4db207b0a4c9685e6423c68, @harperfast/rocksdb-js 2.4.1.

   Bug 1 — rocksdb-js: the registry holds a strong ref, and the GC finalizer never closes

   DBDescriptor::transactionAdd stores a strong shared_ptr (db_descriptor.cpp:828-833):

   this->transactions.emplace(id, txnHandle);                              // std::map<uint32_t, shared_ptr<TransactionHandle>>
   this->closables[txnHandle.get()] = std::weak_ptr<Closable>(txnHandle);  // weak, but the map above is strong

   TransactionHandle::close() is the only path that releases the snapshot and deregisters
   (transaction_handle.cpp:254-320):

   void TransactionHandle::close() {
       ...
       this->dbHandle->descriptor->transactionRemove(shared_from_this());
       ...
       this->txn->ClearSnapshot();      // <-- the only ClearSnapshot on this path
       delete this->txn;

   But the napi finalizer for a GC'd NativeTransaction never calls it (transaction.cpp:105-117):

   NAPI_STATUS_THROWS(::napi_wrap(env, jsThis, reinterpret_cast<void*>(txnHandle),
       [](napi_env env, void* data, void* hint) {
           auto* txnHandle = static_cast<std::shared_ptr<TransactionHandle>*>(data);
           if (*txnHandle) { (*txnHandle).reset(); }   // drops only the JS-side ref
           delete txnHandle;                           // close() is NEVER called
       }, nullptr, nullptr));

   Consequence: a native transaction dropped without commit() or abort() is immortal. The
   registry's strong ref keeps it alive, so the destructor can never run; ClearSnapshot() is never
   reached; and snapshotSet stays true, pinning oldestSnapshotTime until process exit. GC cannot
   rescue it, and nothing is logged at any level.

   Bug 2 — harper core: abort() cannot release a transaction created by save()

   readTxnsUsed is declared with no initializer (DatabaseTransaction.ts:107), so it is
   undefined until getReadTxn() sets it to 1 (line 152).

   save() creates a native transaction on the write-first path and assigns this.transaction
   without setting readTxnsUsed and without trackedTxns.add(this)
   (DatabaseTransaction.ts:215-232):

   transaction ??= this.transaction;
   if (!transaction) {
       transaction = new RocksTransaction(operation.store.store as RocksStore);
       if (this.open === TRANSACTION_STATE.OPEN) {
           this.transaction = transaction;   // readTxnsUsed still undefined; not tracked
       } else { immediateCommit = true; }

   abort() never touches this.transaction — it releases only through the refcount loop
   (DatabaseTransaction.ts:504-505):

   abort(): void {
       while (this.readTxnsUsed > 0) this.doneReadTxn();   // undefined > 0 === false -> body never runs
       this.open = TRANSACTION_STATE.CLOSED;

   So the arithmetic decides the outcome:

   release path	expression	result
   commit() (line 292)	--undefined > 0 → NaN > 0 → false	falls to the else branch → commits/aborts correctly ✅
   abort() (line 505)	undefined > 0 → false	loop never runs, this.transaction untouched → leaked ❌
   doneReadTxn() (line 170)	--undefined === 0 → NaN === 0 → false	no release → leaked ❌

   A write-first transaction that aborts instead of committing leaks its snapshot permanently.
   commit()'s own rejection handler calls this.abort() (line 498-501), and abortDueToTimeout()
   calls abort(), so any error during commit of a write-first transaction hits this.

   The snapshot itself is established lazily by the first read through the transaction —
   save() → operation.store.getEntry(operation.key, { transaction }) → ensureSnapshot()
   (transaction_handle.cpp:514, 544, 557-560). Note these databases are all optimistic mode, so
   putSync/removeSync alone would not set one (lines 581, 618 gate on Pessimistic) — it is the
   read in the write path that pins it.

   Empirical confirmation — registryStatus() on all four nodes

   rocksdb-js exposes registryStatus(), which reports live TransactionHandle counts per database
   straight from the process-global registry. Read-only, no GC, no mutation. Sampled on all 17 threads
   per node, 3 instants 20s apart, zero evaluation errors (51 thread-samples per node). Reporting
   the per-database floor across samples, so transient in-flight transactions drop out:

   node	state	render_schedule txns	every other database
   yc0	snapshot HELD since 19:01:01	2, 2, 2	0
   v3t	snapshot HELD since 19:42:56	2, 2, 2	0
   e9v	clean	0, 0, 0	0
   cd5	clean	0, 0, 0	0

   Perfect separation. render_schedule is the only database on the cluster with live native
   transactions, it is live on exactly the two nodes holding a leaked snapshot, and Harper's
   trackedTxns is 0 on all 17 threads of both (previous comment). Those two handles are
   unreachable from any DatabaseTransaction — i.e. orphaned, not merely long-lived.

   Why save() is the only possible origin

   There are exactly four new RocksTransaction(...) sites in the deployed core. Three are eliminated:

   1. DatabaseTransaction.ts:146 (getReadTxn) — calls trackedTxns.add(this), so it would have
      shown up in the all-thread trackedTxns enumeration. It did not.
   2. DatabaseTransaction.ts:424 (commit retry) — explicitly aborts on every exit, with a comment
      already naming this hazard ("so the throw does not leak its native handle").
   3. Table.ts:5178 (eviction commitItems) — aborts on every branch, and applies to eviction-
      configured tables (page_cache), not render_schedule.

   That leaves save() at line 220 as the only construction site that can produce an untracked
   orphan, which is exactly what we measured.

   Why this produced the reported symptom

   A pinned oldestSnapshotTime forbids dropping obsolete row versions, so on a high-churn secondary
   index the range scans walk dead versions: estimateNumKeys 1,901,091 for ~490k live rows
   (3.9/row), numberReseeksIteration 43,647 vs 51 on a fresh node, dbSeekMicros median 256ms vs
   35ms. Bounded scans degrade ~21x over ~5 days while point reads stay flat. That saturates the
   serialized claim mutex (ρ > 1), which is what surfaced as multi-second cache-hit tail latency.
   Compaction still runs — it just cannot discard anything. Only a full process restart clears it,
   because the registry dies with the process.

   Suggested fixes

   rocksdb-js (the important one — it converts any slip into a permanent leak):

   • Hold the registry ref as weak_ptr, or have the napi finalizer call close() when the JS wrapper
     is collected and the transaction was never committed/aborted.
   • Expose transaction age/snapshotSet/id via registryStatus() so this is observable. Right now the
     only signal that a database has an orphan is comparing a bare count against zero.
   • Consider warning when a TransactionHandle has held a snapshot beyond some threshold.

   harper core:

   • Initialize readTxnsUsed = 0 and have save() set readTxnsUsed = 1 + trackedTxns.add(this)
     when it constructs the transaction, so the object is tracked and refcounted like any other.
   • Make abort() release unconditionally rather than relying solely on the refcount loop:
     after the while, if this.transaction is still set, abort it and null it. That is a one-line
     belt-and-braces fix that would have prevented this regardless of the counter state.
   • The } else {\n} empty branch at DatabaseTransaction.ts:232-233 is dead and can go.

   One correction to the previous comment

   I suggested the likely holder was an abandoned lazy getRange iterator. That was wrong and I am
   retracting it: src/binding/iterator never references snapshots at all, and DBIteratorHandle
   takes TransactionHandle* as a raw pointer (db_iterator_handle.h:34), so an iterator neither
   creates a snapshot nor keeps a transaction alive. Only TransactionHandle calls SetSnapshot().

   I also checked, and discarded, a correlation between the snapshot-pin timestamps and the
   JavaScript execution has taken too long throttle warning (which fires within seconds of both
   pins). That warning occurs 494-613 times on every node including both clean ones, so it carries no
   signal here.

   Environment: harperfast/harper-pro:5.1.26, Node v24.18.1, 4-node Fabric cluster, all databases
   optimistic mode.
7. harper-joseph

   changed the title [-]Read snapshot leaks on high-write databases and is never released: getReadTxn() resets the long-txn watchdog on every call, and the read-only release path is commented out[/-] [+]RocksDB read snapshot leaks permanently: registry holds a strong ref to TransactionHandle, GC finalizer never calls close(), and DatabaseTransaction.abort() cannot release a save()-created transaction[/+] on Aug 6, 2026
8. harper-joseph commented on Aug 6, 2026

   

   harper-joseph

   on Aug 6, 2026

   ContributorAuthor

   More actions

   Filed the worker-inspector bug separately as #2108, so this issue stays focused on the snapshot leak.
9. harper-joseph commented on Aug 6, 2026

   

   harper-joseph

   on Aug 6, 2026

   ContributorAuthor

   More actions

   Correction + the actual leak path (supersedes my previous comment's origin claim)

   Two things to fix in what I wrote above, then the mechanism, which I now have both in source and
   measured.

   Retracting the "save() is the only possible origin" argument

   That elimination was invalid. I argued that a getReadTxn()-created transaction was excluded
   because it calls trackedTxns.add(this) and trackedTxns was empty. But trackedTxns.delete(this)
   runs before the native release in both release paths:

   doneReadTxn() {                          // line 168
       if (--this.readTxnsUsed === 0) {
           trackedTxns.delete(this);        // removed FIRST
           ... this.transaction?.abort();   // release SECOND

   } else {                                 // commit(), line 306
       trackedTxns.delete(this);
       this.transaction = null;             // both cleared BEFORE the native commit is awaited
       if (transaction) { ... transaction.commit() ... }

   So "untracked" does not exclude any construction site. An empty trackedTxns is consistent with a
   transaction created anywhere and released through a path that failed to close the native handle.

   The actual path — a failed commit is deliberately left open, and nobody closes it

   rocksdb-js does not close a transaction whose commit fails. In the async commit complete callback
   (transaction.cpp:333-437):

   if (state->status.ok()) {
       state->handle->close();                                   // success -> ClearSnapshot + deregister
       state->callResolve();
   } else {
       // Normal error path: reset to Pending so JS can retry.
       if (state->handle && state->handle->state == TransactionState::Committing) {
           state->handle->state = TransactionState::Pending;      // deliberately LEFT OPEN
       }
       ...
       state->callReject(error);                                  // no close()
   }

   That is a reasonable contract — the caller may want to retry. It makes the caller responsible for
   aborting when it gives up.

   Harper drops the reference before the commit settles. commit() clears both the tracking set and
   this.transaction before awaiting (line 306-307), so by the time the rejection arrives the only
   surviving reference to the still-open native transaction is the local transaction variable:

   trackedTxns.delete(this);
   this.transaction = null;                         // <-- cleared before the await
   commitResolution = transaction.commit();

   And the non-retryable branch neither retries nor aborts (line 466-471):

   if (error.code === 'ERR_BUSY' || error.code === 'ERR_TRY_AGAIN') {
       ...
       return this.commit({ transaction });          // retry: correct, transaction is carried forward
   } else throw error;                               // <-- neither commits, retries, nor aborts

   The rethrow reaches the outer handler, which does this.abort() — and abort() can only reach
   this.transaction, which is already null:

   abort(): void {
       while (this.readTxnsUsed > 0) this.doneReadTxn();   // doneReadTxn() early-returns on !this.transaction

   So the transaction is now unreachable from JS, still open, with its snapshot set. Note the codebase
   already guards this exact hazard one branch earlier — the MAX_RETRIES path aborts explicitly,
   with the comment "so the throw does not leak its native handle" (line 452-457). The non-retryable
   branch is missing the same guard.

   Then bug 1 from my previous comment makes it permanent: the registry's strong shared_ptr keeps the
   handle alive forever, and the GC finalizer only does txnHandle->reset().

   Measured confirmation

   Runtime.queryObjects against the rocksdb-js Transaction.prototype, all 17 threads of a leaking
   node: ZERO live JS Transaction objects — while 3 native handles hold snapshots on
   render_schedule. Every thread returned clean structured data (0 errors), so this is a true negative.
   The wrappers have been garbage collected; the handles survive only on the registry's strong
   reference. That is exactly what the finalizer-without-close() predicts, and it rules out the
   alternative shape ("some live JS object is still holding it").

   Two independent APIs agree on the count, and agree across every thread:

   	yc0	v3t	e9v	cd5
   uptime	6.2h	5.5h	2.4h	1.0h
   registryStatus() txns on render_schedule	3	2	0	0
   getDBIntProperty('rocksdb.num-snapshots')	3	2	0	0
   every other database	0	0	0	0
   first snapshot pinned at	boot +0.8m	boot +0.7m	—	—

   num-snapshots is identical on all 17 threads (verified), so it is process-wide, and it tracks the
   registry transaction count exactly. render_schedule is the only database cluster-wide with any live
   native transaction.

   Timing model (revised — my "boot-only" reading was wrong)

   The dominant event is at ~45 seconds after boot: both leaking nodes pinned their first snapshot
   there, and both clean nodes dodged it and are still clean hours later. But accrual also continues
   slowly — yc0 went 2 → 3 during steady state at ~6h uptime. I initially misread this as boot-only
   because oldest-snapshot-time only reports the oldest, so it never moves once the first one lands.

   Operationally the distinction barely matters: only the oldest snapshot gates reclamation, so a
   single orphan is sufficient to stop RocksDB dropping obsolete versions for the rest of the process's
   life. That is why the symptom looks like "degrades with uptime" — the clock starts ~45s in.

   What I have NOT established

   The triggering error is not logged. Filtering to the current process generation only, there is no
   commit or transaction error on the leaking node — the sole promise rejection is an unrelated TLS
   failure during a peer restart. So the mechanism above is proven possible and is consistent with
   every measurement, but I have not observed the specific failing commit. Identifying it needs
   instrumentation at transaction-construction time; the evidence in current state is gone, because GC
   already collected the wrappers.

   Also unproven: why render_schedule and not page_cache, which has 4x the keys (2.2M vs 528k)
   and far more write volume, yet has never held an orphan on any node. The plausible reason is that
   render_schedule is the only residency-pinned table here (setResidencyById), so its writes
   race ownership/replication checks and can fail in ways that are not ERR_BUSY/ERR_TRY_AGAIN. That
   is a hypothesis, not a finding.

   Fixes (none depend on identifying the caller)

   rocksdb-js — the important one, because it turns any caller slip into a permanent silent leak:

   • Hold the registry reference as weak_ptr, or have the napi finalizer call close() when the
     wrapper is collected and the transaction was never committed/aborted. Then a dropped transaction
     self-heals at GC instead of pinning a snapshot forever.
   • Expose transaction id / age / snapshotSet from registryStatus(), and warn when a handle has held
     a snapshot beyond a threshold. Today the only signal is a bare count compared against zero.

   harper core:

   • Mirror the existing MAX_RETRIES guard in the non-retryable branch: abort transaction before
     rethrowing.
   • Better: don't clear this.transaction until the commit settles, so abort() can always reach it.

   Environment: harper-pro 5.1.26, core submodule 43a375fd, @harperfast/rocksdb-js 2.4.1,
   Node v24.18.1, all databases optimistic mode.
10. harper-joseph commented on Aug 6, 2026

    

    harper-joseph

    on Aug 6, 2026

    ContributorAuthor

    More actions

    Correction: page_cache leaks too — the blast radius is not limited to one table

    I wrote above that "render_schedule is the only database cluster-wide with any live native
    transaction" and treated "why not page_cache?" as an open puzzle needing a residency explanation.
    Both were wrong, and the error was a sampling artifact: every node I measured for that claim had
    1–6 h of uptime. On an aged node, page_cache leaks as well:

    database           snapshot held   keysWritten   numberReseeksIteration
    render_schedule        114.31 h      12,804,581                  43,647
    page_cache              60.46 h      12,355,032                     942
    render_service, sitemaps, coordination, crawl_stats: no snapshot held
    

    So this is a general defect on high-write databases, exactly as the mechanism predicts — not
    something specific to one table. Withdrawing the residency hypothesis entirely; it is unnecessary.

    But the harm from a held snapshot is not proportional to retention

    Same bug, essentially the same write volume (12.4M vs 12.8M keys), and a 46x difference in
    reseeks. The cost of retained versions is paid only by whoever range-scans that keyspace:

    • render_schedule is range-scanned continuously — RenderQueue.claim walks the nextRenderTime
      index from every worker several times a second. Retained versions turn into reseeks (43,647),
      dbSeekMicros median 256 ms, and the scan degrades ~21x. Because claim is mutex.withLock
      serialized, that saturates and becomes the visible latency event.
    • page_cache is point-read on the serve path (cache key → primary key get). A point lookup
      finds the newest version first and never walks the obsolete ones, hence 942 reseeks over 60 h.

    Two further reasons page_cache retention is cheaper than its size suggests:

    1. Blobs are not pinned by a snapshot. cleanupUnusedBlobs() (blob.ts:1539) deletes a
       superseded blob file unless its fileId is still referenced by the committed record
       (collectRetainedFileIds). It is not snapshot-aware. So the retained row versions are just small
       blob-pointer records — the multi-hundred-KB rendered HTML is in the blob file, which gets deleted
       on schedule regardless of the snapshot.
    2. Consequently the retained bytes are small per version, so page_cache's store bloat is modest
       relative to its 2.2 M keys.

    (Side effect worth noting separately: because blob cleanup ignores snapshots, a read through a
    stale snapshot can reference an already-deleted blob file. That is a plausible contributor to the
    BlobReadError: Blob file not found / "Error sending blob" noise, which is currently treated as
    benign superseded-version churn.)

    Which channel actually caused the elevated cache-hit response times

    The measurement says the claim path, not page_cache retention. Over the window in which the aged
    node was holding both snapshots, cache-hit latency was fine in the overwhelming majority of
    buckets and spiked only where render_queue spiked: 357 calm buckets with a max of 8.4 ms, versus
    4 spiking buckets with a median of 8,368 ms. If page_cache retention were degrading the serve path
    directly, the 357 calm buckets would not have been calm — page_cache held its snapshot throughout
    all of them.

    So the causal chain to slow cache hits runs through render_schedule: retained versions → reseeks →
    21x slower bounded scans → the serialized claim mutex saturates (ρ > 1) → cache-hit requests queue
    behind it on the shared worker event loop.

    Caveat, stated plainly: I have not measured cache-hit latency on the same node with and without
    a page_cache snapshot, so I cannot put a number on a possible smaller direct contribution. All four
    nodes currently hold zero page_cache snapshots, so that contrast isn't available right now; I have a
    watcher running that will catch the next page_cache pin, which makes it a clean before/after on one
    node.

    Net effect on the recommendations

    Unchanged, and if anything reinforced — the fix is upstream and table-agnostic:

    • rocksdb-js: registry should hold a weak_ptr, or the finalizer must call close(). This is the
      fix that matters, because it makes any high-write database self-heal rather than just the one we
      happened to notice.
    • harper core: abort the local transaction in the non-retryable commit branch (mirroring the
      MAX_RETRIES guard), and/or don't clear this.transaction until the commit settles.
    • Monitoring: num-snapshots per database, not just on the table whose symptom is loudest. A
      nonzero count on any database means that database can never reclaim again for the process
      lifetime; whether it hurts depends on whether anything range-scans it.
11. harper-joseph commented on Aug 6, 2026

    

    harper-joseph

    on Aug 6, 2026

    ContributorAuthor

    More actions

    Filed #2109 for a composing defect found while chasing the serve-path impact of this leak: getRecordCount computes its extrapolation base via getKeysCount, which in rocksdb-js is one synchronous native full-key iteration on the JS thread — measured 2.47s unbroken on a healthy 2.2M-key table (CPU profile), turning a 6ms cache-hit into 1,009ms (live probe). The leak here multiplies that: the same iteration also walks every retained obsolete version, so the block grows with process age. The two issues together produce the multi-second serve-path latencies that started this investigation.
12. harper-joseph commented on Aug 6, 2026

    

    harper-joseph

    on Aug 6, 2026

    ContributorAuthor

    More actions

    Live accrual event captured, correlated with a write burst — supports the failed-commit trigger.

    At 02:31Z we deployed a config change that reschedules ~76k catalog entries (render cadence 12h → 6h), producing a burst of render_schedule writes across the cluster. Within ~15 minutes:

    node	orphaned native txns before	after (stable across 3 samples, 20s apart)
    v3t	3	8
    yc0	3	4
    e9v	0	0
    cd5	0	0

    registryStatus() and rocksdb.num-snapshots agree exactly on every sample. No process restarted (container + worker uptimes continuous), so these are steady-state accruals under write contention — consistent with the proposed path (commit rejection on a non-ERR_BUSY/ERR_TRY_AGAIN error → bare rethrow → abort() can't reach the already-nulled transaction). Still nothing in the logs at any level when each orphan appears.

    Also matches the earlier observation that the two big accrual moments are boot catch-up and write bursts — and that exposure is probabilistic per node (two nodes took the same burst with zero orphans).

    No operational change from this alone: only the oldest snapshot gates reclamation and v3t's oldest is unchanged (19:42:56Z), scans still ~72ms. It just makes the reproducer shape concrete: mass rewrite of an indexed table under concurrent load.
13. kriszyp

    transferred this issue fromHarperFast/harper-proon Aug 6, 2026
14. kriszyp

    set

    Priority

    to

    P1

    on Aug 6, 2026
15. kriszyp

    added this to the v5.2 milestone on Aug 6, 2026
16. kriszyp

    added the

    Bug

    issue type on Aug 6, 2026
17. kriszyp

    added a parent issue on Aug 6, 2026

    • [Epic] Transaction/write atomicity & optimistic-concurrency correctness #1652
18. kriszyp

    added 2 commits that reference this issue on Aug 9, 2026

    feat(stats): expose rocksdb.num-snapshots

    ...

    1b5427c

    fix(txn): guard the read-refcount release and prove the snapshot retu…

    ...

    ca002ae
19. kriszyp commented on Aug 10, 2026

    

    kriszyp

    on Aug 10, 2026

    Member

    More actions

    Your final analysis is correct, and I've reproduced the core defect against the current main of rocksdb-js (2.7.1) — it is not fixed there either. Thank you for the depth here; the registryStatus() / rocksdb.num-snapshots cross-check and the Runtime.queryObjects negative result are what made this diagnosable, and the retractions along the way were the right calls.

    Confirmed: the registry keeps a dropped transaction alive forever

    Verified in source on current main and reproduced live:

    baseline               num-snapshots= 0  oldest-snapshot-time= 0           registry.transactions= 0
    after read in txn      num-snapshots= 1  oldest-snapshot-time= 1786236991  registry.transactions= 1
    after GC x8            num-snapshots= 1  oldest-snapshot-time= 1786236991  registry.transactions= 1
    

    Both halves of your mechanism hold:

    • DBDescriptor::transactionAdd stores a strong shared_ptr in transactions, while the closables entry written on the very next line is a weak_ptr.
    • The napi finalizer only does (*txnHandle).reset(); delete txnHandle;.
    • ~TransactionHandle() does call close(), which is the only ClearSnapshot() path — but the destructor can never run, because the registry reference outlives the wrapper.

    So the destructor is correct and unreachable. That's the invariant that was violated: the registry owns what the JS wrapper should own, which turns any caller slip anywhere into a permanent, silent, restart-only leak.

    Your reading of the failed-commit contract is also right: a rejected commit deliberately resets to Pending without closing, so the caller can retry. Reasonable on its own; lethal only in combination with the above.

    What's fixed, and where

    The specific trigger you hit is already fixed on main, but not on your version. The terminal (non-ERR_BUSY/ERR_TRY_AGAIN) commit branch now aborts the transaction before rethrowing — landed 2026-07-22, shipped in v5.2.0. Your 5.1.26 predates it. That is the single change most likely to stop the accrual you're measuring.

    The abort() gap you identified was still live on main. You were right to retract the "save() is the only possible origin" argument, but the underlying defect you found there is real: abort() releases the native handle only through while (this.readTxnsUsed > 0) this.doneReadTxn(), and only getReadTxn() ever sets readTxnsUsed. A write-first link leaves it undefined, so undefined > 0 is false and the handle is stranded. It's reachable: a write-only link in a multi-store chain never calls getReadTxn(), and abortDueToTimeout() walks the chain calling abort() on exactly those links. Fixed now, along with directCommitSync()'s missing this.transaction = null — which, as you noted, is a real bug on its own.

    Fixes in flight:

    • rocksdb-js (#768) — the napi finalizer now releases the handle when V8 collects the wrapper, so a dropped transaction self-heals at GC instead of pinning a snapshot for the process lifetime. One note on your suggestion: we did not make the registry reference a weak_ptr. An async get holds a raw TransactionHandle* and relies on close() cancelling and waiting for in-flight work; letting the last shared_ptr drop destroy the handle would race that. Going through close() from the finalizer gets the same self-healing with that machinery intact. A commit in flight is the one case that defers — the commit state still owns the handle — and the commit-completion paths release it when they settle.
    • harper core (#2128) — the two release paths above.
    • Observability, which you asked for twice and were right to: rocksdb.num-snapshots is exposed as a stat and Harper surfaces it per database in system_information.metrics as numSnapshots, and registryStatus() now reports per-handle id and ageMs instead of a bare count. oldestSnapshotTime alone can't show accrual — as you found, it stops moving once the first snapshot is pinned; a count that stays nonzero, plus a handle age past any plausible request lifetime, can.

    We deliberately stopped short of reporting per-handle snapshotSet and state, which is what you'd actually want. Those are written by the owning thread and by the commit-completion callback without any lock, so reporting them from another thread is a data race; making them atomic put a barrier on every read and write path for the sake of a diagnostic. The per-database snapshot count answers the same operational question without that cost.

    For your cluster

    The path forward is 5.2.x. The terminal-abort fix is already there, the abort() and directCommitSync() fixes are landing on main now, and the rocksdb-js change lands with the next release of that dependency. We are not planning to backport any of this to 5.1 — the release is on maintenance, and this is a multi-part fix across two repos, which is exactly the shape that goes wrong when it's cherry-picked.

    While you're still on 5.1.26: rocksdb.num-snapshots is readable per database today via getDBIntProperty, no release needed. Keying your existing watcher on a count that stays nonzero, rather than on oldestSnapshotTime, will tell you when a node has entered the state — and a rolling restart remains the way out of it until you're on 5.2.

    Two smaller things from your write-up that we're keeping regardless of this issue: the unguarded purgeLogs() call in scheduleAuditCleanup, and the try/catch wrapping the whole handler loop in runReclamationHandlers rather than each handler. You correctly retracted them as the cause here; they're still worth fixing.

    Filing #2108 and #2109 separately was the right instinct — thanks for keeping this one focused.

    One thing on the write-up itself

    The investigation was genuinely excellent and I don't want to discourage any of the rigor. But the issue as it stands is ten comments, most of them long, four of which retract or supersede an earlier one, and the body at the top still describes the original theory. Working out what we currently believe took me longer than verifying the actual defect did — and I already knew the code. Anyone triaging this cold, or a future reader who hits the same symptom, is going to bounce off it.

    What would help, roughly in order of value:

    • Keep the issue body current. Edit it as the conclusion moves, so the top of the page always states the live theory and the current status. The comment thread is then the audit trail, not the place the answer hides.
    • Lead each comment with one line of status — "supersedes my previous comment's origin claim", "confirms X, retracts Y" — before the evidence. Several of yours do this, and they're much easier to follow.
    • Separate the finding from the working. The claim and the one measurement that decides it up top, the rest below for anyone who wants to check it. The registryStatus() / num-snapshots table was the whole ballgame and it's most of the way down a long comment.

    None of this is a complaint about the depth — the depth is why this got diagnosed at all. It's about making that depth reviewable by someone who wasn't in it with you.
20. kriszyp

    added 2 commits that reference this issue on Aug 10, 2026

    fix(transaction): release a transaction dropped without commit or abort

    ...

    98f18f5

    feat(stats): expose rocksdb.num-snapshots

    ...

    01cdddb
21. kriszyp

    added 7 commits that reference this issue on Aug 19, 2026

    fix(transaction): release a transaction dropped without commit or abort

    ...

    1052b04

    feat(stats): expose rocksdb.num-snapshots

    ...

    8200050

    fix(txn): release the native handle unconditionally on abort and fail…

    ...

    c9c3c7d

    fix(txn): guard the read-refcount release and prove the snapshot retu…

    ...

    641f9ec

    fix(transaction): release a transaction dropped without commit or abort

    ...

    c130300

    feat(stats): expose rocksdb.num-snapshots

    ...

    ed6694b

    Release the RocksDB transaction handle on abort and on a failed direc…

    ...

    1693743