Repository navigation
test: regression guard for heterogeneous schema-less ingest (#1282) - #1284
Conversation
Adds an integration test that inserts 2000 distinct-shape records into an open (non-@Sealed) table and asserts every record reads back intact (and that a deep record round-trips exactly), guarding the high-shape-cardinality record-encoder path. #1282 reported most such records decoding as null (#1163 "shared structure missing") on a single node. That was traced to a stale structon (< 1.0.7) in node_modules, not a defect in harper main: with the pinned structon ^1.0.7 the real RecordEncoder reads back 50,000/50,000. This test pins the invariant so a structon regression or accidental downgrade can't silently reintroduce the loss. No production code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a regression test suite to guard against schema-less heterogeneous ingest data loss (#1282). It adds a test that ingests 2,000 highly varied record shapes into an open table and verifies they can all be read back intact. The review feedback suggests two improvements: mapping over the filtered 'intact' array instead of 'rows' to safely build the 'byId' map without throwing a TypeError on null values, and adding an assertion to verify that the table becomes ready within the polling timeout to improve test debuggability.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
Reviewed; no blockers found. |
- Map over `intact` (not raw `rows`) when building the byId lookup to avoid TypeError on null/malformed decode entries - Add `tableReady` flag + `ok()` assertion after the polling loop so a 30 s timeout produces a clear failure instead of a confusing generic error on the subsequent insert Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2 participants
Summary
Adds an integration test (
integrationTests/database/heterogeneous-ingest.test.ts+ fixture) that inserts 2,000 distinct-shape records into an open (non-@sealed) table and asserts all 2,000 read back intact, plus an exact deep-equal round-trip of one deep record. No production code change.Why
#1282 reported the vast majority of heterogeneous-ingest records decoding as
null(#1163 "shared structure missing") on a single node. Investigation traced this to a stalestructon(< 1.0.7) innode_modules— not a defect in harper main. harper pinsstructon ^1.0.7, and with that pinned version the realRecordEncoderreads back 50,000/50,000 (the catastrophic 6/50,000 only reproduces against structon 1.0.4). structon ≥ 1.0.7 both bounds the typed-structure dictionary (the CDI-OOM cap) and correctly handles the two-byte-record path thatRecordEncoder'smaxOwnStructures = 256engages.This test pins the data-integrity invariant so a future
structonregression — or an accidental downgrade — can't silently reintroduce the loss.Where to look
search_by_idreturning all ids with intact objects; the high shape cardinality (2,000) is deliberately well above msgpackr's ~32-structure one-byte-record boundary, to exercise the two-byte-record + structure-recycling path where Data loss: schema-less heterogeneous ingest triggers single-node #1163 'shared structure missing', records decode as null #1282's loss occurred.maxOwnStructures = 256against a stale structon 1.0.4 makes it fail withgot 6 (rows returned=6); on pinned 1.0.7 it passes 2,000/2,000.Notes
agy) leg hung and was skipped (caveat). Harper-domain review surfaced only that this guards forward-looking integrity and does not recover any data already written under a stale-structon install.