Secondary-index reads return partial/empty results for the second table accessed in a single request (5.1.19 and 5.1.22)

Summary

In Harper 5.1.19 and 5.1.22, within a single resource request, the first
access to a table returns an incomplete result set if a different table
was read earlier in that same request. Reading that table first — or as the only
read in the request — returns the full set.

Only the secondary-index read path is affected. An unconditioned full scan of
the same table, in the same position in the same request, returns every row. The
records are intact; the index scan comes back short.

The magnitude ranges from a partial count to zero. It silently
empties/undercounts any endpoint that aggregates across multiple tables in one
request.

Environment

Harper 5.1.19, and reproduced identically on 5.1.22.
@harperfast/rocksdb-js 2.4.0.
Reproduced on a single node (macOS) and on a 2-node Fabric cluster.
Component plugins: rest + graphqlSchema + jsResource; TypeScript resources.

Reproduction — self-contained, no real data required

Two tables, one indexed attribute, seeded synthetically. 100% deterministic.

The essential ingredient is the write pattern: every key gets rows on every
wave, so a single key's index entries are scattered across many sorted runs
rather than sitting contiguously in one. Seeding the same rows in a single pass
does not reproduce.

`schemas/repro.graphql`

type GenA @table(database: "metrics-repro") @export {
  id: ID @primaryKey
  repositoryId: String @indexed
  n: Int @indexed
  body: String
}

type GenB @table(database: "metrics-repro") @export {
  id: ID @primaryKey
  repositoryId: String @indexed
  n: Int @indexed
  body: String
}

`resources/Gen.ts`

import { Resource, databases } from "harper";

const db: any = (databases as any)["metrics-repro"];
const PAD = "x".repeat(1024);

// Every key gets rows on EVERY wave, so one key's index entries end up
// scattered across many sorted runs rather than contiguous in one.
async function generate(table, keys, perKeyPerWave, waves, bodyKb) {
  const body = PAD.repeat(bodyKb);
  for (let w = 0; w < waves; w++)
    for (let r = 0; r < keys; r++)
      for (let i = 0; i < perKeyPerWave; i++) {
        const n = w * perKeyPerWave + i;
        await db[table].put({
          id: `${table}-r${r}-n${n}`,
          repositoryId: `repo-${r}`,
          n,
          body,
        });
      }
}

async function drain(table, key, scan) {
  const query =
    scan === "full"
      ? { conditions: [] }
      : {
          operator: "and",
          conditions: [
            { attribute: "repositoryId", comparator: "equals", value: key },
          ],
        };
  let n = 0;
  for await (const r of db[table].search(query)) {
    if (scan === "full" && r.repositoryId !== key) continue;
    n++;
  }
  return n;
}

export class Gen extends Resource {
  static loadAsInstance = false;

  async get(target: any) {
    const p = (k, d) => String(target?.get?.(k) ?? d);
    if (p("action", "probe") === "gen") {
      const table = p("table", "GenA");
      await generate(table, 50, 5, 20, 4); // 50 keys x 5 rows x 20 waves, 4KB body
      return { table, written: 50 * 5 * 20 };
    }
    const key = p("key", "repo-0");
    const scan = p("scan", "index");
    const order = p("tables", "GenB,GenA").split(",").filter(Boolean);
    const out = {};
    for (let i = 0; i < order.length; i++)
      out[`${i + 1}:${order[i]}`] = await drain(order[i], key, scan);
    return out;
  }
}

Steps

# 1. Seed (once)
curl -k "$HOST/Gen?action=gen&table=GenA"
curl -k "$HOST/Gen?action=gen&table=GenB"

# 2. Probe — each request counts repo-0 in the given table order
curl -k "$HOST/Gen?tables=GenA"
curl -k "$HOST/Gen?tables=GenA,GenB"
curl -k "$HOST/Gen?tables=GenB,GenA"
curl -k "$HOST/Gen?tables=GenB,GenA&scan=full"

Results — every count should be 100

Request	Result
`?tables=GenA`	`GenA = 100` ✅
`?tables=GenB`	`GenB = 100` ✅
`?tables=GenA,GenB`	`100`, `100` ✅
`?tables=GenB,GenA`	`100`, `GenA = 0` ❌
`?tables=GenB,GenA&scan=full`	`100`, `100` ✅ (full scan is correct)

Stable across 5/5 consecutive runs and across every key (repo-0, repo-1,
repo-7, repo-23, repo-49 — all return GenA = 0).

What this narrows it to

It is the secondary-index path, not the data. Same table, same request
position: via the indexed attribute → 0; via an unconditioned full scan
filtered in memory → 100. Every record is present and readable.

The damage belongs to the table read second, not the one read first.
Crossing a long-lived production table with a freshly-created copy of it:

Read order	Result
old `PullRequest` → new `ReproIssue`	962, 895 ✅
new `ReproPullRequest` → old `Issue`	962, 547 ❌
old `Commit` → new `ReproIssue`	2820, 895 ✅
new `ReproCommit` → old `Issue`	2820, 547 ❌

What precedes it is irrelevant; it only has to be a different table.

It is latched per request. GenB → GenA → GenA → GenA returns the same
wrong count every time. Once a table's read view is established within a
request, re-reading does not recover the missing entries.
Row content is not involved. Copying the rows of an affected production
table into a brand-new table — identical fields, values and indexes — does
not reproduce. Only the write history differs.
The losses are contiguous key runs. On the production Issue table
(895 → 547), walking the result in primary-key order gives 74 kept-runs and
74 missing-runs (kept runs 1–155 entries, missing runs 1–24). Whole
contiguous stretches drop out — consistent with a sorted run missing from the
merged iterator view rather than per-record filtering.
It is not a stale-by-time snapshot. Rows written in the same bulk write
split 543 kept / 327 lost. Neither syncedAt nor updatedAt separates the
kept set from the missing set.
@harperfast/rocksdb-js 2.4.0 does not reproduce it standalone. Driving
the binding directly — two column families on one database, the same
interleaved wave seeding, index-shaped keys, then getRange on B followed by
A — returns the full count in every order. That test does not replicate
Harper's per-request transaction/snapshot handling or its record cache, so it
is suggestive rather than conclusive, but it points at the Harper layer.

What does NOT fix it

Running the reads sequentially (awaited one at a time) instead of Promise.all.
Explicitly closing each async iterator (iterator.return()) after draining.
Wrapping each read in its own transaction().
Re-reading the affected table later in the same request.
Upgrading 5.1.19 → 5.1.22.

Impact

Any endpoint aggregating across multiple tables in a single request silently
returns partial or zero data for every table except the first one read. In our
application this zeroed out issues, commits and releases on a dashboard that
reads PullRequest + Issue + Commit + Release in one Promise.all, while
each of those tables returns correct data when queried on its own.

The only reliable workaround we have found is one table per request, which is
a significant restructuring for aggregation endpoints.

Questions

Is this a known issue, and is a fix planned?
Where is the per-request read view for a table established, and is there a
supported way to force a fresh/isolated index read per table within one
request?
Does a full compaction of the affected column family clear it? If so we would
take that as an interim mitigation — can you point us at the right operation?

Happy to provide access to a reproducing instance, or the full probe component.

Comment generated by kAIle (Claude Opus 4.8)

Thank you for this write-up — the interleaved-wave seeding detail, the read-order table, and the contiguous-run analysis made this dramatically faster to pin down. Reproduced, and I have it isolated to a specific layer.

Status

Confirmed on main. I have it reproducing deterministically in a Harper unit test (two tables in one database, seeded in waves, read in one request), with the same signature you reported: reading GenB → GenA returns 0 for the second table, GenA → GenB returns 81/100. Single-table reads and full scans return the complete set.

Answering your questions

Where is the per-request read view established?

DatabaseTransaction.getReadTxn() (resources/DatabaseTransaction.ts) creates the RocksDB read transaction lazily, on the first read of the request, and every later read in that request reuses the cached one.

Which reads share it is decided by txnForContext() (resources/Table.ts). It walks a chain of per-database transactions and matches on the database path — and every table in a database shares that path. So all tables in one database share a single read transaction, bound to whichever table was touched first. That is why the failure follows read order, and why it is latched for the life of the request.

Is there a supported way to force a fresh read view per table in one request?

Not currently, and this is the crux of the bug rather than a missing knob. Note that wrapping each read in its own transaction() cannot work as a workaround: an already-open transaction is reused rather than nested, which matches your finding that it doesn't help.

Would compaction clear it?

No — and I'd skip that as a mitigation. Your own evidence already rules the data out (full scans return every row, and the records are intact), and my reproduction confirms the index scan itself returns the complete key set. Nothing is wrong on disk, so there is nothing for compaction to repair.

What it is — and what it is not

Some things that looked like plausible causes and are now ruled out, each tested rather than reasoned about:

Not the storage engine. Your standalone rocksdb-js test was sound, and I reached the same conclusion from another angle: point reads issued under a transaction bound to a different column family return complete results in 2.4.0.
Not the index scan. It yields all 100 keys, with and without the transaction.
Not filtering or record loading. Nothing is skipped or dropped during hydration, and no index entry comes back empty.
Not an early transaction close. The read transaction is released only after the short iteration finishes, not during it.

The loss requires four conditions together — remove any one and results are complete:

Both tables in the same database
A request transaction (i.e. a normal resource request)
Asynchronous iteration of the results
Large record values (your 4KB body; dropping it returns the full set)

The result-transform stage emits all 100 records, but the async iterator delivers only 81. So the truncation is in the asynchronous iteration path, and it only affects tables that reuse an existing read transaction rather than creating one — which is exactly why the first table read is always complete and every subsequent one can be short.

Root-causing the final step is in progress; I'll follow up when the fix is drafted. A regression test covering the read-order matrix is already written.

Interim mitigation worth trying

Rather than restructuring to one table per request, try splitting the aggregated tables across separate databases (@table(database: "...")). Cross-database reads in a single request returned complete results in every test I ran, because each database gets its own read transaction. That should let your dashboard keep its single Promise.all shape.

Please treat that as a mitigation, not a guarantee — it sidesteps the trigger rather than fixing the defect, so it's worth verifying against your own data before you rely on it. And thank you for the offer of a reproducing instance; the synthetic repro was sufficient, so no access needed.

Comment generated by kAIle (Claude Opus 4.8)

@kriszyp

Root cause isolated

Following up on my previous comment with a much sharper result — and one correction to it. I said the truncation was in "the asynchronous iteration path". That was the right direction but not precise: it is specifically the asynchronous record load, and it has a clean minimal reproduction.

The failing primitive

primaryStore.getEntry(id, { transaction, async: true }) returns undefined for records that are demonstrably present, when transaction was created by a different table in the same database:

call	result
sync, foreign txn	100/100 ✅
async, foreign txn	90/100 ❌
async, no txn	100/100 ✅
async, own txn (this table created it)	100/100 ✅
concurrent async gets / interleaved key encoding, no txn	100/100 ✅

Why the loss is partial, and why the count moves between runs

The magnitude wandered across runs (0, 70, 81, 85, 87). It tracks cache residency exactly:

keys	foreign txn, async
warm (read moments earlier)	50/50 ✅
cold (never read)	35/50 ❌

Store.get in rocksdb-js first attempts a synchronous in-memory-cache read (ONLY_IF_IN_MEMORY_CACHE_FLAG). Those hits resolve inline and are correct. Only reads that fall through to the deferred native read — context.get(key, resolve, reject, txnId) — go wrong, and only when the transaction belongs to another store.

That explains the whole reported range: a completely cold table read second loses everything (the reporter's GenA = 0), a partly warm one loses a fraction, and the same query gives different counts on repeat runs.

How it reaches the user

txnForContext (resources/Table.ts) matches on database path, and every table in a database shares that path — so all tables in a database reuse one read transaction, bound to whichever table was touched first. Rows arrive at transformEntryForSelect as bare primary keys and are hydrated there by loadLocalRecord, which sets async: true. The deferred read misses, record == null, and the row is silently dropped as a SKIP. No error, no log — which is what makes this so quiet.

@kriszyp — this lands on your May fix

Your diagnosis in 7b6d909 ("a RocksTransaction scoped to the outer table's column family cannot read from a different table's column family, so getEntry returns null") is confirmed — with two refinements:

It only holds on the async path. The synchronous read handles a foreign-CF transaction correctly today (measured above), which is why this was so hard to pin down.
targetTable._readTxnForContext(context) is a no-op for tables in the same database — it walks the same next chain and matches on the same database path, handing back the very transaction it was meant to avoid. It does the right thing across databases, which is consistent with the fix working for the case you had in front of you.

So the joined-select path is still exposed to this whenever the target table is in the same database.

I have a deterministic regression test ready (two tables, one database, wave-seeded, read in both orders, plus full-scan and cross-database controls). What I can't see from JS is the native side: why the deferred read with a foreign txnId fails while the sync read succeeds. I ruled out two plausible JS-level explanations by experiment — shared key-buffer aliasing and cross-CF synchronous reads — so I don't want to guess past that point.

Which direction do you want this to go?

Fix it natively in rocksdb-js — make the deferred async read resolve the column family the way the sync path already does. This looks like the real repair, but it's your call whether the native read should support a cross-CF transaction at all, or whether Harper shouldn't be handing it one.
Per-store read transactions in Harper — key txnForContext by store rather than database path, so each table gets a transaction bound to its own column family. This is what the May fix assumed already happened. I'd want your read first: one transaction per database looks deliberate for write atomicity, so this likely means separating read-transaction scope from write scope, and that's a bigger change than I'd make unilaterally.
Targeted mitigation now — skip the async: true path when the read transaction wasn't created by this store, falling back to the synchronous read that measures correct. Contained and patchable, at the cost of event-loop yielding on those reads.

My instinct is 3 as something shippable soon, with 1 as the actual fix, and 2 only if you think the per-database transaction scope is wrong independent of this bug. But you know the transaction and storage layers far better than I do, so I'd rather take your direction than pick for you.

Happy to implement whichever you prefer, or to hand over the reproduction if you'd rather drive it.

Comment generated by kAIle (Claude Opus 4.8)

#1. Definitely #1. Async should work for any CF in a database using a transaction for that database, and if it doesn't, it should be fixed.

chore(deps): pin @harperfast/rocksdb-js to ~2.4.1 on v5.1

test(database): pin the #1881 cross-table secondary-index read contract

Cut the batch to the two anchors that can actually fail, and close th…

Make the ownership check able to fire: collide the keys, own the row

test(database): pin the #1881 cross-table secondary-index read contract

Cut the batch to the two anchors that can actually fail, and close th…

Make the ownership check able to fire: collide the keys, own the row

Navigation Menu

Repository navigation

Secondary-index reads return partial/empty results for the second table accessed in a single request (5.1.19 and 5.1.22) #1881

Summary

Environment

Reproduction — self-contained, no real data required

`schemas/repro.graphql`

`resources/Gen.ts`

Steps

Results — every count should be 100

What this narrows it to

What does NOT fix it

Impact

Questions

Status

Answering your questions

What it is — and what it is not

Interim mitigation worth trying

Root cause isolated

The failing primitive

Why the loss is partial, and why the count moves between runs

How it reaches the user

@kriszyp — this lands on your May fix

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Secondary-index reads return partial/empty results for the second table accessed in a single request (5.1.19 and 5.1.22) #1881

Description

Summary

Environment

Reproduction — self-contained, no real data required

schemas/repro.graphql

resources/Gen.ts

Steps

Results — every count should be 100

What this narrows it to

What does NOT fix it

Impact

Questions

Activity

kylebernhardy commented on Jul 21, 2026

Status

Answering your questions

What it is — and what it is not

Interim mitigation worth trying

kylebernhardy commented on Jul 21, 2026

Root cause isolated

The failing primitive

Why the loss is partial, and why the count moves between runs

How it reaches the user

@kriszyp — this lands on your May fix

kriszyp commented on Jul 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

`schemas/repro.graphql`

`resources/Gen.ts`