Repository navigation
Conversation
…, tighten tests - Strip the always-stamped trio's bits before the EVM materialize call, since setBlockHeader immediately overwrites them from the item (avoids decoding/ hex-encoding hash+timestamp for every block only to discard the result). - Move build_columns' mask-union fold inside the function instead of duplicating it at all 4 call sites (both EVM/SVM decoders in block_store.rs and transaction_store.rs). - Avoid a full raw-Block clone per SVM block: take the response's blocks once, build the lean per-slot header from a borrow, then move the owned blocks into the store (mirrors the EVM source's existing borrow-then-move pattern). - Name the SVM raw-block-store gate's required query fields as a constant instead of an inline string match, mirroring EVM's REQUIRED_BLOCK_FIELDS. - Add BlockStore tests that exercise the public materialize() method with a populated store (EVM + SVM), and a merge test asserting a re-fetched block overwrites the stale entry for the same number - closing the gap where only the empty-store path and private decode_* functions were tested.
… params Store an always-known header (timestamp/hash) alongside every inserted EVM block, keyed the same way as the raw form, rather than threading them through materialize() as extra caller-supplied arrays. This keeps materialize()'s contract simple (block_numbers + masks, decode purely from store state) and extends the same pattern Number already used (resolve from the map key) to the rest of the trio. - BlockStore::insert_evm now takes (number, timestamp, hash, raw: Option<Arc<Block>>) instead of insert_evm_raw(number, raw) - the header is always inserted, the full raw block only when an event selected a field beyond the trio. - decode_evm_block_columns overlays number/timestamp/hash from the stored header unconditionally, so a caller no longer needs to pass them in. - evm_hypersync_source::process_response inserts a header for every response block, reusing the already-computed out_blocks headers instead of gating the store insert on has_extra_block_fields. - BlockStore.res drops setBlockHeader (post-hoc JS mutation) in favor of makeBlockHeader, used only for the no-store-lookup fast path; the Rust- backed path returns fully-formed objects directly. - Rust tests updated: the trio-specific decode tests now target header resolution instead of Number's old mask-gated key lookup, and decode_applies_each_rows_own_mask exercises a genuine extra field (gasUsed) since the trio is no longer mask-gated.
…kHash from items The always-available number/timestamp/hash special-casing was spread across the code (JS hasExtraFields/& ~7 gates, trio-stripped masks, an unconditional header overlay in Rust decode). Now it lives in one place: the config-side field selection, which always extends the EVM selection with the trio, so every EVM mask carries its bits. Everything downstream is generic mask logic: - decode_evm_block_columns is a plain build_columns pass; each arm names its data source (number from the key, timestamp/hash from the stored header, the rest from the raw block) and all are gated by each row's own mask. - BlockStore.res loses hasExtraFields/stripTrioBits/setBlockHeader entirely. materializeEvmItems always materialises through the store; the SVM path's only gate is the generic "any non-zero mask". With event.block now built solely from the store/payload, the item-level timestamp and blockHash duplicates have no remaining consumers, so they are removed from Internal.item/eventItem: - raw_events' block_hash/block_timestamp columns are extracted from the payload block by the ecosystem toRawEvent (names differ per ecosystem). - The progress-latency metric reads the payload block via the ecosystem's blockTimestampName, threaded into applyBatchProgress. - Sources and test fixtures stop constructing the fields.
EvmHeader{timestamp,hash} + an optional raw block was a memory optimization:
a header-only entry is ~98 bytes vs 384 bytes for Arc<simple_types::Block>
(measured via size_of - the full struct's ~27 boxed-Option fields cost a
pointer slot each regardless of which are populated). But that split adds a
second stored type, a raw-extraction step in decode, and a header/raw
insert split - real code weight for a per-chain-type memory saving.
Traded the memory optimization for simplicity: StoredBlock::Evm goes back to
holding Arc<simple_types::Block> alone, decode_evm_block_field reads
timestamp/hash straight off it via fill_masked exactly like every other
field, and the store keeps every response block (not just ones with extra
fields selected) since the trio now depends on the record being present.
One fetch response = one immutable chunk: sorted keys + one column per selected field (flat fixed/var byte columns with validity bitmaps, boxed lists for collection cells). Store lifecycle is append/drop chunks: prune drops whole chunks below progress (watermark on a bisected one), rollback truncates tails. Reads resolve per field newest-chunk-first, which unions rows that arrived with different field subsets and deduplicates identical rows re-fetched by overlapping partitions. Adjacent runs of small chunks coalesce (k-way merge preserving newest-wins) once the stack exceeds 64 chunks.
Each fetch response's blocks become one chunk (sorted number/slot keys + only the populated fields' columns) instead of per-row Arc<struct> entries in a BTreeMap. materialize gathers the requested cells under the lock into scratch columns and decodes off-thread as before; napi surface and decode output are unchanged. Sources hand the store their owned response blocks wholesale (insert_evm_blocks/insert_svm_blocks). Per stored EVM block this drops the fixed 368-byte struct shell (plus per-field boxes) to roughly the selected fields' payload bytes.
Replace the per-row simple_types::Transaction map with the shared ChunkStore keyed by (block, txIndex). SVM token balances move to a duplicate-key companion table in the same store, gathered by key range at materialisation; the old HashMap union in build_svm_store goes away since transactionIndex is key-derived. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbaYTDU4XoGRSDGsQyDhsn
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
DZakh
added a commit
that referenced
this pull request
Jul 8, 2026
* Add EVM block store (Rust core + hypersync source) Introduce a per-chain BlockStore mirroring TransactionStore: raw blocks kept in Rust keyed by block number, deduplicated, with selected fields materialised in columnar form on demand. Extract the shared columnar machinery into field_columns so both stores reuse it. The EVM hypersync source now retains raw blocks in a BlockStore page and returns lean block headers (number/timestamp/hash) instead of full blocks; get_event_items returns the page alongside the response. Blocks are only stored when an event selects a field beyond the always-required trio. ReScript wiring follows in a subsequent commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Wire EVM block store through the ReScript runtime Blocks for EVM HyperSync are no longer carried inline on the event payload. The source returns a BlockStore page (raw blocks, deduplicated by number) plus lean block headers; the page merges onto ChainState and selected block fields are materialised onto event.block at batch prep — mirroring the transaction store. number/timestamp/hash are always stamped from the item, so a block whose events select only those needs no store lookup. - BlockStore.res binding + materializeItems (dedup by block, trio from item, short-circuit when no further fields selected) - FieldMask: shared selection-bitmask helpers extracted from TransactionStore - payload.block made optional; RPC/Fuel/Simulate/SVM keep it inline (no page) - eventConfig gains blockFieldMask; ChainState merges/prunes/rolls back the block store alongside the transaction store - BlockStore_test plus the cross-language field-order contract test SVM block store deferred to a follow-up; SVM blocks stay inline for now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Add SVM block store and expand the public SVM block Extend BlockStore to SVM: ecosystem-tagged store with an SvmBlockField decoder, raw solana blocks kept keyed by slot, materialised in columnar form on demand. Block materialisation is now ecosystem-aware — EVM writes the block onto payloads that omit it; SVM enriches the minimal inline block the source attaches with the real hash plus blockHeight/parentSlot/parentBlockhash from the store. The public svmInstructionBlock gains blockHeight/parentSlot/parentBlockhash and a real (no longer always-empty) hash; the SVM hypersync query requests the extra columns and the source returns a block-store page. Tests: SVM block_store unit tests, SVM field-order contract + enrich/dedup ReScript tests; updated SVM source/client tests for the new columns and tuple. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Add block_fields field selection for SVM; rename to height/parentHash SVM instructions can now select block fields via field_selection.block_fields, mirroring EVM. slot/time/hash stay always-included; height/parentSlot/parentHash are opt-in. The selection drives both the HyperSync query columns (fetch only what's used) and the block-store materialisation mask. - human_config: SvmBlockField enum + block_fields on SvmFieldSelection (+ regen svm.schema.json) - system_config/public_config: resolve and serialise selected_block_fields - EventConfigBuilder/Config/Internal: thread blockFields to the SVM event config; blockFieldMask now built from always-included trio + selection - SvmHyperSyncSource: request only the selected block columns - Rename the SVM block fields blockHeight->height and parentBlockhash->parentHash across the store, public svmInstructionBlock, and field arrays Tests: SVM block-field selection drives query columns; renamed field-order contract; updated SVM source/router test literals. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Require SVM time/hash in field selection; generate the block type per selection SVM block fields are now fully opt-in: only `slot` is always present (stamped on the inline block from the item); `time`/`hash`/`height`/`parentSlot`/`parentHash` must be listed under field_selection.block_fields. The source builds a `{slot}` inline block and only requests the extra columns it needs; materialisation short-circuits when nothing is selected. The per-instruction `block` type is now generated from the selection, mirroring `transaction`: selected fields get their real type (nullable ones as `T | undefined`), unselected fields render as `FieldNotSelected<...>`, and `slot` is always present. SvmInstruction gains a `Block` type param bound from the program-table descriptor. - human_config/Internal: SvmBlockField gains Time/Hash (+ regen svm.schema.json) - codegen: svm_block_field_specs + svm_block_ts_type; block added to the SVM program-table descriptor; index.d.ts SvmInstruction<…, Block> - Envio.svmInstructionBlock time/hash made optional - tests: block-type render + spec/runtime-match tests; RpcSource field arrays disambiguated for the new Hash variant Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Address review: gate SVM store, slim SVM block header, dedup materialize/codegen - SVM source now populates the block store only when an instruction selected a block field beyond the always-fetched slot/blockhash/blockTime, mirroring the EVM has_extra_block_fields gate (no more per-page Arc-clones when unused). - Slim the SVM response Block to {slot, blockhash, blockTime} — the selectable fields (height/parentSlot/parentHash) live only in the store, so they're no longer double-shipped across napi. - Extract groupByBlock so materializeEvmItems/materializeSvmItems share the run-length grouping; extract svm_field_line shared by svm_transaction_ts_type and svm_block_ts_type. - Fix the stale svmInstructionBlock time/hash docs (absent, not 0/empty) and pin the hasExtraFields '& ~7' trio-bit assumption with a BlockStore_test contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Fix clippy too_many_arguments on svm_field_line Derive the token-balance boolean toggle from the knob inside the function instead of passing it as an eighth argument. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Fix SVM block-field selection: config-parse drop, time/hash materialization, hash optionality Three defects in the SVM block-field path found during review: - Config.res `svmEventDescriptorSchema` never declared `blockFields`, so rescript-schema stripped it on parse and every SVM block-field selection was silently dropped before reaching the event config (whole feature dead). - The SVM source stamped only `slot` inline while `materializeSvmItems` expected time/hash from the store, but the Rust gate never populates the store for the always-fetched blockhash/block_time — so selecting only `time`/`hash` left them undefined. Stamp the slot/time/hash trio inline from the response (mirroring EVM's item-stamped trio) and gate materialize on `hasExtraFields` so only height/parentSlot/parentHash consult the store. - Codegen marked `SvmBlockField::Hash` required though `svmInstructionBlock.hash` is optional (a slot can lack a block row); make it optional and assert name+optionality parity against Envio.res in the runtime-match test. Also correct stale "slot/time/hash always excluded" comments (only `slot` is) and drop redundant field docs. Tests: schema-preserves-blockFields regression, SVM source stamps the inline trio, materialize keeps the trio when only time/hash selected, and an SVM trio-field-code (0/1/2) pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Refactor stores to chunk-columnar architecture (#1378) * Address block store review: avoid redundant work, dedupe mask folding, tighten tests - Strip the always-stamped trio's bits before the EVM materialize call, since setBlockHeader immediately overwrites them from the item (avoids decoding/ hex-encoding hash+timestamp for every block only to discard the result). - Move build_columns' mask-union fold inside the function instead of duplicating it at all 4 call sites (both EVM/SVM decoders in block_store.rs and transaction_store.rs). - Avoid a full raw-Block clone per SVM block: take the response's blocks once, build the lean per-slot header from a borrow, then move the owned blocks into the store (mirrors the EVM source's existing borrow-then-move pattern). - Name the SVM raw-block-store gate's required query fields as a constant instead of an inline string match, mirroring EVM's REQUIRED_BLOCK_FIELDS. - Add BlockStore tests that exercise the public materialize() method with a populated store (EVM + SVM), and a merge test asserting a re-fetched block overwrites the stale entry for the same number - closing the gap where only the empty-store path and private decode_* functions were tested. * Move EVM block trio resolution into BlockStore instead of materialize params Store an always-known header (timestamp/hash) alongside every inserted EVM block, keyed the same way as the raw form, rather than threading them through materialize() as extra caller-supplied arrays. This keeps materialize()'s contract simple (block_numbers + masks, decode purely from store state) and extends the same pattern Number already used (resolve from the map key) to the rest of the trio. - BlockStore::insert_evm now takes (number, timestamp, hash, raw: Option<Arc<Block>>) instead of insert_evm_raw(number, raw) - the header is always inserted, the full raw block only when an event selected a field beyond the trio. - decode_evm_block_columns overlays number/timestamp/hash from the stored header unconditionally, so a caller no longer needs to pass them in. - evm_hypersync_source::process_response inserts a header for every response block, reusing the already-computed out_blocks headers instead of gating the store insert on has_extra_block_fields. - BlockStore.res drops setBlockHeader (post-hoc JS mutation) in favor of makeBlockHeader, used only for the no-store-lookup fast path; the Rust- backed path returns fully-formed objects directly. - Rust tests updated: the trio-specific decode tests now target header resolution instead of Number's old mask-gated key lookup, and decode_applies_each_rows_own_mask exercises a genuine extra field (gasUsed) since the trio is no longer mask-gated. * Make block-store trio handling fully mask-driven; drop timestamp/blockHash from items The always-available number/timestamp/hash special-casing was spread across the code (JS hasExtraFields/& ~7 gates, trio-stripped masks, an unconditional header overlay in Rust decode). Now it lives in one place: the config-side field selection, which always extends the EVM selection with the trio, so every EVM mask carries its bits. Everything downstream is generic mask logic: - decode_evm_block_columns is a plain build_columns pass; each arm names its data source (number from the key, timestamp/hash from the stored header, the rest from the raw block) and all are gated by each row's own mask. - BlockStore.res loses hasExtraFields/stripTrioBits/setBlockHeader entirely. materializeEvmItems always materialises through the store; the SVM path's only gate is the generic "any non-zero mask". With event.block now built solely from the store/payload, the item-level timestamp and blockHash duplicates have no remaining consumers, so they are removed from Internal.item/eventItem: - raw_events' block_hash/block_timestamp columns are extracted from the payload block by the ecosystem toRawEvent (names differ per ecosystem). - The progress-latency metric reads the payload block via the ecosystem's blockTimestampName, threaded into applyBatchProgress. - Sources and test fixtures stop constructing the fields. * Collapse EvmHeader back into simple_types::Block EvmHeader{timestamp,hash} + an optional raw block was a memory optimization: a header-only entry is ~98 bytes vs 384 bytes for Arc<simple_types::Block> (measured via size_of - the full struct's ~27 boxed-Option fields cost a pointer slot each regardless of which are populated). But that split adds a second stored type, a raw-extraction step in decode, and a header/raw insert split - real code weight for a per-chain-type memory saving. Traded the memory optimization for simplicity: StoredBlock::Evm goes back to holding Arc<simple_types::Block> alone, decode_evm_block_field reads timestamp/hash straight off it via fill_masked exactly like every other field, and the store keeps every response block (not just ones with extra fields selected) since the trio now depends on the record being present. * Add chunked columnar store core One fetch response = one immutable chunk: sorted keys + one column per selected field (flat fixed/var byte columns with validity bitmaps, boxed lists for collection cells). Store lifecycle is append/drop chunks: prune drops whole chunks below progress (watermark on a bisected one), rollback truncates tails. Reads resolve per field newest-chunk-first, which unions rows that arrived with different field subsets and deduplicates identical rows re-fetched by overlapping partitions. Adjacent runs of small chunks coalesce (k-way merge preserving newest-wins) once the stack exceeds 64 chunks. * Rewrite block store on chunked columns Each fetch response's blocks become one chunk (sorted number/slot keys + only the populated fields' columns) instead of per-row Arc<struct> entries in a BTreeMap. materialize gathers the requested cells under the lock into scratch columns and decodes off-thread as before; napi surface and decode output are unchanged. Sources hand the store their owned response blocks wholesale (insert_evm_blocks/insert_svm_blocks). Per stored EVM block this drops the fixed 368-byte struct shell (plus per-field boxes) to roughly the selected fields' payload bytes. * Rewrite transaction store on chunked columns + token-balance sub-table Replace the per-row simple_types::Transaction map with the shared ChunkStore keyed by (block, txIndex). SVM token balances move to a duplicate-key companion table in the same store, gathered by key range at materialisation; the old HashMap union in build_svm_store goes away since transactionIndex is key-derived. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbaYTDU4XoGRSDGsQyDhsn --------- Co-authored-by: Claude <noreply@anthropic.com> * Make SVM block materialization mirror EVM's always-included trio (#1381) * Make SVM block materialization mirror EVM: always-included slot/time/hash trio SVM's slot/time/hash are now an always-included trio (like EVM's number/timestamp/hash), force-unioned into every event's field selection rather than being user-selectable. The Rust block store is now always populated for SVM (removing the has_extra_block_fields gate), so materialize() always returns a self-sufficient block. This lets SvmHyperSyncSource stop stamping an inline block at item construction (mirroring EVM, which omits `block` until batch prep), and removes enrichBlock/Object.assign entirely: materializeEvmItems and materializeSvmItems collapse into one BlockStore.materializeItems, since both ecosystems now follow the identical group/materialize/assign path. time/hash come out of the selectable SvmBlockField config enum (this feature is unreleased, so no existing configs break); generated instruction block types now render them as always-present nullable fields instead of FieldNotSelected sentinels, matching slot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rkB51uUrb79UrD78sq4Xc * Trim comments to non-obvious invariants only Drop comments that just narrated the refactor or restated what the surrounding code already shows (e.g. "mirroring EVM's X"), keeping only the ones explaining a fact a reader couldn't get from the code itself (why time/hash stay nullable, why grouping relies on item ordering). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rkB51uUrb79UrD78sq4Xc * Regenerate svm.schema.json after SvmBlockField doc comment trim schemars reads the SvmBlockField doc comment into the schema description, so the comment-cleanup commit left the committed schema stale. * Make svmInstructionBlock.hash non-optional, matching EVM's trio Solana's blockhash is a non-optional String at the HyperSync source (unlike block_time/parent_slot/parent_blockhash/block_height, which are genuinely Option<T>), and EVM's equivalent always-included fields (number/timestamp/hash) render as non-optional despite the same theoretical missing-row concern. hash inherited its `| undefined` from when it was still a user-selectable field, before it joined the always-included trio; that reasoning no longer applies now that hash is guaranteed alongside slot. time keeps `| undefined`, since Solana can genuinely omit a block's time even when the block row is present. * Stop marking SVM fields optional-when-selected unless genuinely nullable svm_block_field_specs/svm_transaction_field_specs derived optionality from the materializer's Option<T> columns, but that Option-ness mostly just means "the wire query didn't request this column" — SvmHyperSyncSource unions every event's selection into the query it issues chain-wide, so once a field is selected, the returned column is real. That's the same pattern EVM's own optional block fields already follow (query-gated, rendered non-optional once selected). height/parentSlot/parentHash (block) and feePayer/success/fee/ computeUnitsConsumed/recentBlockhash (transaction) now render as their plain type once selected. err/version keep `| undefined`: those are genuinely null by business logic regardless of selection (err on a successful tx, version on a legacy one), not a selection artifact. svm_block_field_specs_match_runtime now compares names only, matching the already-established svm_transaction_field_specs_match_runtime precedent: Envio.res's svmInstructionBlock is a pre-narrowing fallback that stays optional for any individually-selectable field regardless of this per-selected-field nullability question, so parity with it was never the right check for non-trio fields. --------- Co-authored-by: Claude <noreply@anthropic.com> * Extract grouping logic and make blockStore optional for Fuel (#1382) * Address block-store review findings: dedup stores, drop Fuel's dead BlockStore, fix SVM optionality - Flip computeUnitsConsumed back to optional in the generated SVM transaction type: Solana's own RPC docs describe it as reported only "when available" (compute-unit metering post-dates mainnet), so it can be absent even when selected. - Hoist the Ecosystem enum, bytes() helper, and field-name-listing boilerplate out of block_store.rs/transaction_store.rs and into the shared field_columns.rs module, removing four near-duplicate definitions. - Extract the "group adjacent items by key, OR-ing masks" algorithm shared by BlockStore.res's groupByBlock and TransactionStore.res's materializeItems into FieldMask.groupAdjacent, so the grouping invariant lives in one place. - Make ChainState's blockStore option<BlockStore.t> (None for Fuel), matching the existing page-level pattern, instead of always allocating a live but permanently-empty Rust store/Mutex for Fuel chains. This also removes the now-redundant ~ecosystem switch threaded through materializeBlockItems/ materializeBatchItems/materializePageItems/materializeBatchTransactions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PdHqppW444ZCCNxm1U8Lpj * Revert BlockStore Option<> change for Fuel, keep it always constructed Fuel chains keep a live BlockStore again instead of None, matching the original PR's design (every chain holds one, mirroring TransactionStore). The ~ecosystem switch in materializeBlockItems/materializeBatchItems/ materializePageItems/materializeBatchTransactions comes back with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PdHqppW444ZCCNxm1U8Lpj * Inline TransactionStore's grouping instead of the generic FieldMask helper Going through FieldMask.groupAdjacent here meant building a (blockNumber, transactionIndex) tuple per item (an allocation the original code avoided by comparing the two ints separately) plus an extra pass to unzip the grouped keys back into blockNumbers/transactionIndices for materialize. Not worth it on this per-batch hot path, so back to the specialized loop. BlockStore.res keeps using FieldMask.groupAdjacent — its key is a bare int, so it doesn't pay the tuple cost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PdHqppW444ZCCNxm1U8Lpj --------- Co-authored-by: Claude <noreply@anthropic.com> * Store SVM block hash/parentHash as StrCol instead of bytes (#1387) blockhash/parentBlockhash are base58 strings; storing them via var_from + utf8() round-tripped them through raw bytes and re-validated UTF-8 on every materialize read with String::from_utf8_lossy. Add a dedicated StrCol (like VarCol, but the shared buffer is a String) so the column is text end-to-end: no byte-slice cast on insert, and no UTF-8 re-validation on read since the buffer's validity is an invariant of String. Co-authored-by: Claude <noreply@anthropic.com> * Replace chunk-based storage with merge-on-insert table (#1388) * Replace chunk-columnar field stores with merge-on-insert Table Blocks, transactions, and SVM token balances now dedup on insert instead of accumulating immutable chunks that only get reconciled by coalescing or pruning. Each key owns one slot in a flat columnar Table; a re-fetched key (overlapping partitions, reorgs) overwrites its slot's cells in place, so a large field like EVM `input` is stored once rather than once per response. Var/list cells are individually boxed so an overwrite frees the old blob immediately. Token balances move from a duplicate-key chunk table to a Table keyed by (slot, transactionIndex, account); the SVM query layer now force-adds `account` to the field selection whenever token balances are requested, so the key component is always populated. chunk_store.rs is replaced by field_table.rs; the AnyCol interchange layer (fills/decoders/hex helpers) is unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPirgAMkqHXCoXAPkNDkAs * Address review feedback: trim comments, avoid a clone, add coverage - svm_hypersync_source/query.rs: scope the account force-add comment to our own caller convention instead of asserting general field-selection semantics that conflict with the vendored crate's own doc. - field_table.rs/transaction_store.rs: trim several doc comments that restated what the code already shows, keeping only the non-obvious invariants/preconditions. - transaction_store.rs: avoid cloning `account` per row in insert_svm_token_balances by building the var columns first (borrowing rows) and then consuming rows to build the keys. - Add missing test coverage: a TransactionStore-level merge-dedup test (mirrors the existing BlockStore one), a Table::clear() unit test, and a ReScript-level SvmHyperSyncSource test pinning that token-balance queries always carry `account`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPirgAMkqHXCoXAPkNDkAs --------- Co-authored-by: Claude <noreply@anthropic.com> * Add Robinhood (4663) to the Network enum HyperSync added chain 4663 (Robinhood) to its API, which the hypersync-health-check asserts is present in the Network enum. Add it as a HypersyncChain; endpoint URL auto-derives from the chain id and reorg depth defaults to unknown (None). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Address review feedback: Robinhood explorer, materialize function naming - Add NetworkWithExplorer to Robinhood (chain 4663 has a public Blockscout explorer at robinhoodchain.blockscout.com), so contract-import can use it. - Rename EventProcessing.materializeBatchTransactions to materializeBatchEvents: it already materialises both transaction and block fields via ChainState.materializeBatchItems, so the old name was misleading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Regenerate CommandLineHelp.md for the Robinhood chain addition `make update-help`: adding NetworkWithExplorer to Robinhood added `robinhood` to the --blockchain option's possible values, which the generated docs snapshot test checks against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Materialize transaction and block masks in a single pass over batch items Per review feedback on EventProcessing.res: ChainState.materializeBatchItems and materializePageItems each called TransactionStore.materializeItems and BlockStore.materializeItems, which independently re-walked the same items array to build their own grouping/masks. Replace with groupBatchItems, a single Array.forEach that builds both the (blockNumber, transactionIndex) transaction groups and the blockNumber-only block groups together, tracking each dimension's adjacent-run independently since they can diverge mid-batch. The two materialize calls against the Rust stores still run concurrently. Behavior is unchanged (verified against the full test suite); added a regression test pinning that a transaction-group break doesn't wrongly interrupt a same-block run in the block dimension, and vice versa. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Remove dead code: unused grouping abstraction and vestigial conversions - FieldMask.groupAdjacent had exactly one caller (BlockStore.groupByBlock); TransactionStore inlines its own grouping and ChainState.groupBatchItems now does too, so the generic version served no other consumer. Inline its logic directly into BlockStore.groupByBlock and remove the abstraction. - RpcSource.res: getBlockTimestamp/getBlockHash externals were only used to stamp timestamp/blockHash onto Internal.item, which no longer carries those fields (they're read off the payload's own block now). getBlockNumber is still used for the item's blockNumber. - svm_hypersync_source/types.rs: TryFrom<simple::Block> for Block was only reachable via QueryResponse::try_from's generic try_map(r.blocks), but that call's only call site always empties r.blocks first (the raw blocks go into the BlockStore, and the header is filled in afterwards from Block::from_raw) so the conversion only ever ran on an empty Vec. Build blocks: Vec::new() directly instead and drop the now-unreachable TryFrom impl. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh * Remove BlockStore.materializeItems/groupByBlock and TransactionStore.materializeItems Neither had a real caller left: ChainState.materializeBatchItems/ materializePageItems now go through groupBatchItems + applyTransactionGroups/ applyBlockGroups, calling the raw materialize bindings directly. These two per-store functions were kept alive only by their own unit tests. Migrate the coverage they had (dedup by adjacency, mask OR-ing across a shared group, empty-mask still stamps a value, inline items skipped) into ChainState_materialize_test.res against materializePageItems, the path that actually runs in production. BlockStore_test.res/TransactionStore_test.res keep only their field-code contract tests (fieldCodes/orMask/blockFields ordering), which are independent of the removed functions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh --------- Co-authored-by: Claude <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors block, transaction, and token balance stores from row-oriented storage to a chunk-columnar architecture. Each fetch response becomes an immutable chunk with sorted keys and optional columns per field, enabling efficient deduplication, partial processing, and selective field materialization without materializing large fields until needed.
Key Changes
New
chunk_store.rsmodule: Implements the core chunk-columnar storage with:BitVec: Validity bitmap for sparse columnsNumCol<T>,FixedCol,VarCol,ListCol<T>: Typed column storage for different data shapesAnyCol: Enum over all column types with uniform interfaceChunk<K>: One response's rows with sorted keys, prune watermark, and optional columnsChunkStore<K>: Append-only chunk collection with automatic coalescing at 64 chunks, supporting prune (drop processed rows) and rollback (truncate tail)block_store.rsrefactored:BTreeMap<i64, Arc<Block>>withChunkStore<i64>evm_block_col()andsvm_block_col()fill_maskeddependency; masking now happens during gathertransaction_store.rsrefactored:BTreeMap<(i64, u32), Arc<Transaction>>withChunkStore<(i64, u32)>for transactionsChunkStore<(i64, u64)>for SVM token balances (duplicate-key table)evm_tx_col()andsvm_tx_col()Nonefrom column buildersField materialization pipeline:
evm_tx_col,evm_block_col, etc.) extract raw bytes/values from responsesinput,logs_bloom) only touched on rows that selected themReScript runtime updates:
timestampandblockHashfromInternal.item(now derived from block store lookup)BlockStore.hasExtraFieldslogic (removed trio-bit clearing)Notable Implementation Details
unique_keysflag distinguishes one-row-per-key stores (blocks, transactions) from duplicate-key token-balance tablehttps://claude.ai/code/session_01UbaYTDU4XoGRSDGsQyDhsn