Repository navigation
Add block store for per-chain block field materialization - #1369
Merged
Merged
Conversation
Introduce a per-chain BlockStore mirroring TransactionStore: raw blocks kept in Rust keyed by block number, deduplicated, with selected fields materialised in columnar form on demand. Extract the shared columnar machinery into field_columns so both stores reuse it. The EVM hypersync source now retains raw blocks in a BlockStore page and returns lean block headers (number/timestamp/hash) instead of full blocks; get_event_items returns the page alongside the response. Blocks are only stored when an event selects a field beyond the always-required trio. ReScript wiring follows in a subsequent commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Blocks for EVM HyperSync are no longer carried inline on the event payload. The source returns a BlockStore page (raw blocks, deduplicated by number) plus lean block headers; the page merges onto ChainState and selected block fields are materialised onto event.block at batch prep — mirroring the transaction store. number/timestamp/hash are always stamped from the item, so a block whose events select only those needs no store lookup. - BlockStore.res binding + materializeItems (dedup by block, trio from item, short-circuit when no further fields selected) - FieldMask: shared selection-bitmask helpers extracted from TransactionStore - payload.block made optional; RPC/Fuel/Simulate/SVM keep it inline (no page) - eventConfig gains blockFieldMask; ChainState merges/prunes/rolls back the block store alongside the transaction store - BlockStore_test plus the cross-language field-order contract test SVM block store deferred to a follow-up; SVM blocks stay inline for now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Extend BlockStore to SVM: ecosystem-tagged store with an SvmBlockField decoder, raw solana blocks kept keyed by slot, materialised in columnar form on demand. Block materialisation is now ecosystem-aware — EVM writes the block onto payloads that omit it; SVM enriches the minimal inline block the source attaches with the real hash plus blockHeight/parentSlot/parentBlockhash from the store. The public svmInstructionBlock gains blockHeight/parentSlot/parentBlockhash and a real (no longer always-empty) hash; the SVM hypersync query requests the extra columns and the source returns a block-store page. Tests: SVM block_store unit tests, SVM field-order contract + enrich/dedup ReScript tests; updated SVM source/client tests for the new columns and tuple. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
SVM instructions can now select block fields via field_selection.block_fields, mirroring EVM. slot/time/hash stay always-included; height/parentSlot/parentHash are opt-in. The selection drives both the HyperSync query columns (fetch only what's used) and the block-store materialisation mask. - human_config: SvmBlockField enum + block_fields on SvmFieldSelection (+ regen svm.schema.json) - system_config/public_config: resolve and serialise selected_block_fields - EventConfigBuilder/Config/Internal: thread blockFields to the SVM event config; blockFieldMask now built from always-included trio + selection - SvmHyperSyncSource: request only the selected block columns - Rename the SVM block fields blockHeight->height and parentBlockhash->parentHash across the store, public svmInstructionBlock, and field arrays Tests: SVM block-field selection drives query columns; renamed field-order contract; updated SVM source/router test literals. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Bring in main (incl. #1361 typed transaction-store ecosystem). Apply the same to BlockStore: replace the ECO_* u8 constants + atomics with a construction-time Ecosystem { Evm { should_checksum }, Svm, Fuel } enum and new_evm/new_svm/new_fuel factories; merge no longer learns the ecosystem (debug-asserts page==store). ReScript BlockStore.make takes ~ecosystem/~shouldChecksum and ChainState derives both from config, mirroring TransactionStore.
… selection
SVM block fields are now fully opt-in: only `slot` is always present (stamped on
the inline block from the item); `time`/`hash`/`height`/`parentSlot`/`parentHash`
must be listed under field_selection.block_fields. The source builds a `{slot}`
inline block and only requests the extra columns it needs; materialisation
short-circuits when nothing is selected.
The per-instruction `block` type is now generated from the selection, mirroring
`transaction`: selected fields get their real type (nullable ones as
`T | undefined`), unselected fields render as `FieldNotSelected<...>`, and `slot`
is always present. SvmInstruction gains a `Block` type param bound from the
program-table descriptor.
- human_config/Internal: SvmBlockField gains Time/Hash (+ regen svm.schema.json)
- codegen: svm_block_field_specs + svm_block_ts_type; block added to the SVM
program-table descriptor; index.d.ts SvmInstruction<…, Block>
- Envio.svmInstructionBlock time/hash made optional
- tests: block-type render + spec/runtime-match tests; RpcSource field arrays
disambiguated for the new Hash variant
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds a Rust ChangesBlockStore Feature
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
…ize/codegen
- SVM source now populates the block store only when an instruction selected a
block field beyond the always-fetched slot/blockhash/blockTime, mirroring the
EVM has_extra_block_fields gate (no more per-page Arc-clones when unused).
- Slim the SVM response Block to {slot, blockhash, blockTime} — the selectable
fields (height/parentSlot/parentHash) live only in the store, so they're no
longer double-shipped across napi.
- Extract groupByBlock so materializeEvmItems/materializeSvmItems share the
run-length grouping; extract svm_field_line shared by svm_transaction_ts_type
and svm_block_ts_type.
- Fix the stale svmInstructionBlock time/hash docs (absent, not 0/empty) and pin
the hasExtraFields '& ~7' trio-bit assumption with a BlockStore_test contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Derive the token-balance boolean toggle from the knob inside the function instead of passing it as an eighth argument. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/envio/src/Config.res (1)
702-745: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winAdd
blockFieldstosvmEventDescriptorSchemabefore widening it here.This code now reads
svm["blockFields"]and forwards it intobuildSvmInstructionEventConfig, butsvmEventDescriptorSchemaat Lines 196-217 still doesn't declare that key. That means the new SVM block-field selection either bypasses validation entirely or gets lost during parse, so this feature can fail silently.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/envio/src/Config.res` around lines 702 - 745, The SVM event config builder now expects blockFields from the widened svm descriptor, but svmEventDescriptorSchema does not define that key yet. Update svmEventDescriptorSchema so it includes blockFields with the correct optional array type, then ensure the parsing/widening path that feeds Config.res preserves it into widenedEventItem before buildSvmInstructionEventConfig reads svm["blockFields"].
🧹 Nitpick comments (6)
packages/cli/src/field_columns.rs (1)
68-82: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winFail fast if a column length drifts from the row count.
zipsilently truncates here, so a malformed decoder could partially populate output rows without surfacing the brokenColumns::leninvariant.Proposed fix
unsafe fn set_col<T: ToNapiValue>( env: sys::napi_env, objs: &[sys::napi_value], key: sys::napi_value, values: Vec<Option<T>>, ) -> napi::Result<()> { + if values.len() != objs.len() { + return Err(napi::Error::from_reason(format!( + "column length mismatch: rows={}, values={}", + objs.len(), + values.len() + ))); + } for (obj, cell) in objs.iter().zip(values) { if let Some(v) = cell {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/field_columns.rs` around lines 68 - 82, The set_col helper currently uses zip, which silently truncates when values length does not match objs length and can hide a broken Columns::len invariant. Update set_col to explicitly validate that the number of decoded values matches the number of row objects before iterating, and return an error instead of partially populating rows if there is any mismatch; use the set_col symbol and the surrounding column assignment logic to locate the fix.packages/envio/src/Envio.res (1)
78-86: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDrop redundant field comments.
hash,parentSlot, andparentHashalready describe themselves; keeping only the non-obvious selection/missing-row and height-vs-slot notes better matches the ReScript comment policy.Suggested cleanup
- /** Block hash. */ hash?: string, /** Block height (distinct from slot). Absent when HyperSync didn't return a block for this slot, or the upstream omitted it. */ height?: int, - /** Slot of the parent block. */ parentSlot?: int, - /** Hash of the parent block. */ parentHash?: string,As per coding guidelines,
**/*.res: “Default to writing no comments” and “Don't write a comment that restates what the code already says”.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/envio/src/Envio.res` around lines 78 - 86, Remove the redundant field comments in Envio.res for hash, parentSlot, and parentHash, since their names already explain them. Keep only the non-obvious documentation on height and any selection/missing-row behavior near the relevant block record definition, and update the comments in the block type so they follow the ReScript comment policy.Source: Coding guidelines
packages/envio/src/Internal.res (1)
327-331: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUpdate the SVM block-store docs to match the new runtime contract.
These comments still describe SVM as inline-only and imply
time/hashare always present, but the new path only guarantees inlineslotand usesblockFieldMaskplusBlockStore.materializeSvmItemsto enrich selected fields. Please rewrite these together so the shared contract is accurate in one place. As per coding guidelines, "When refactoring, keep comments that still explain non-obvious behavior; drop or rewrite comments that described the old shape."Also applies to: 445-450, 516-519
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/envio/src/Internal.res` around lines 327 - 331, The shared block payload comments in Internal.res still describe the old SVM inline-only shape and should be rewritten to match the new runtime contract. Update the documentation around getPayloadBlock/setPayloadBlock and the related SVM payload comments so they explain that only inline slot is guaranteed, while blockFieldMask and BlockStore.materializeSvmItems are used to materialize selected fields such as time/hash when needed. Keep the explanation centralized and remove any wording that implies those fields are always present.Source: Coding guidelines
packages/envio/src/sources/FieldMask.res (1)
31-39: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winFail fast when an ordered field list exceeds 32 entries.
orMasktruncates to 32 bits, and this helper is already operating at the ceiling. If either field enum grows again, masks will silently alias/drop bits instead of breaking loudly.Suggested guard
let fieldCodes = (fields: array<string>): dict<int> => { + if fields->Array.length > 32 { + JsError.throwWithMessage("Field masks support at most 32 fields") + } let codes = Dict.make() fields->Array.forEachWithIndex((name, i) => codes->Dict.set(name, i)) codes }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/envio/src/sources/FieldMask.res` around lines 31 - 39, Add an explicit guard in makeMaskFn (or immediately after fieldCodes(fields)) to fail fast when the ordered field list grows beyond 32 entries, since orMask only preserves 32 bits. Use the fieldCodes/makeMaskFn path to validate the input size and raise a clear error instead of allowing silent bit aliasing or truncation.scenarios/test_codegen/test/SvmHyperSyncSource_test.res (1)
17-61: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDerive
blockFieldMaskfromselectedBlockFieldshere. Mirror the SVM config builder instead of hardcoding0.;BlockStorereads this mask, so the helper bypasses the same materialization path and can hide block-field regressions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scenarios/test_codegen/test/SvmHyperSyncSource_test.res` around lines 17 - 61, Derive the block field mask from selectedBlockFields in makeEventConfig instead of hardcoding blockFieldMask to 0. Mirror the same Svm event config builder logic used for transactionFieldMask by computing the mask from the selected block fields before constructing the Internal.svmInstructionEventConfig record, so BlockStore sees the intended block-field selection.packages/envio/src/sources/Svm.res (1)
19-29: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftDerive the SVM block-field ordinals from one authoritative list.
Line 21 hard-codes the Rust/ReScript field-code order, while
transactionFieldsabove already avoids drift by reading the typed list fromInternal. BecauseeventBlockFieldMaskuses array position as the store code, a future reorder here will silently materialize the wrong block columns.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/envio/src/sources/Svm.res` around lines 19 - 29, The SVM block-field ordering is hard-coded in `blockFields`, which can drift from the Rust store’s field codes and cause `eventBlockFieldMask` to select the wrong columns. Replace the inline string list with an authoritative source, similar to how `transactionFields` is derived from `Internal`, and ensure `BlockStore.makeMaskFn` continues to use the same canonical ordinal order for `SvmBlockField`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/block_store.rs`:
- Around line 414-425: BlockStore::merge currently enforces the ecosystem match
only with debug_assert_eq!, so release builds can merge incompatible stores and
later surface as empty block records. Replace the debug-only check with a
runtime validation in BlockStore::merge that fails fast on ecosystem mismatches
before draining data, and apply the same invariant check to the other affected
block-loading path referenced by the review so both code paths reject
cross-ecosystem data consistently. Keep the self-merge guard and preserve the
existing lock/drain flow only after the ecosystem has been verified.
In `@packages/cli/src/config_parsing/public_config.rs`:
- Around line 326-329: The `block_fields` comment in `public_config.rs` is stale
because `time` and `hash` are still selectable SVM block fields; update the doc
comment on `block_fields` to reflect the real invariant by saying only `slot` is
always included and not selectable, and keep the wording aligned with how
`block_fields` is serialized.
In `@packages/cli/src/config_parsing/system_config.rs`:
- Around line 1808-1810: The doc comments for resolve_svm_block_fields still
describe an outdated block-field selection rule. Update the comments near
resolve_svm_block_fields and the related block-field comment block to reflect
the current contract: time and hash may be returned, and only slot is always
present and therefore excluded from the returned selection. Keep the wording
aligned with the function’s actual behavior and referenced field names (slot,
time, hash).
In `@packages/cli/src/hbs_templating/codegen_templates.rs`:
- Around line 2747-2752: The selectable field mapping in the SvmBlockField match
currently marks Hash as required even though svmInstructionBlock.hash can be
absent. Update the mapping in codegen_templates.rs so SvmBlockField::Hash is
treated as optional, and make sure the generated TypeScript type reflects that.
Also extend the relevant parity test around the SvmBlockField / selectable field
spec to assert optionality for Hash, not just that the field name is present.
In `@packages/envio/src/sources/Source.res`:
- Around line 25-28: Update the `Source` record comment so it no longer says
`None` is used for RPC/Fuel/Simulate/SVM inline-block sources; revise the
wording to reflect that `blockStore` pages are now threaded through those paths
too. Keep the comment only if it still explains the `blockStore` contract that
isn’t obvious from the `Source` type itself, and avoid documenting an outdated
`None` behavior.
---
Outside diff comments:
In `@packages/envio/src/Config.res`:
- Around line 702-745: The SVM event config builder now expects blockFields from
the widened svm descriptor, but svmEventDescriptorSchema does not define that
key yet. Update svmEventDescriptorSchema so it includes blockFields with the
correct optional array type, then ensure the parsing/widening path that feeds
Config.res preserves it into widenedEventItem before
buildSvmInstructionEventConfig reads svm["blockFields"].
---
Nitpick comments:
In `@packages/cli/src/field_columns.rs`:
- Around line 68-82: The set_col helper currently uses zip, which silently
truncates when values length does not match objs length and can hide a broken
Columns::len invariant. Update set_col to explicitly validate that the number of
decoded values matches the number of row objects before iterating, and return an
error instead of partially populating rows if there is any mismatch; use the
set_col symbol and the surrounding column assignment logic to locate the fix.
In `@packages/envio/src/Envio.res`:
- Around line 78-86: Remove the redundant field comments in Envio.res for hash,
parentSlot, and parentHash, since their names already explain them. Keep only
the non-obvious documentation on height and any selection/missing-row behavior
near the relevant block record definition, and update the comments in the block
type so they follow the ReScript comment policy.
In `@packages/envio/src/Internal.res`:
- Around line 327-331: The shared block payload comments in Internal.res still
describe the old SVM inline-only shape and should be rewritten to match the new
runtime contract. Update the documentation around
getPayloadBlock/setPayloadBlock and the related SVM payload comments so they
explain that only inline slot is guaranteed, while blockFieldMask and
BlockStore.materializeSvmItems are used to materialize selected fields such as
time/hash when needed. Keep the explanation centralized and remove any wording
that implies those fields are always present.
In `@packages/envio/src/sources/FieldMask.res`:
- Around line 31-39: Add an explicit guard in makeMaskFn (or immediately after
fieldCodes(fields)) to fail fast when the ordered field list grows beyond 32
entries, since orMask only preserves 32 bits. Use the fieldCodes/makeMaskFn path
to validate the input size and raise a clear error instead of allowing silent
bit aliasing or truncation.
In `@packages/envio/src/sources/Svm.res`:
- Around line 19-29: The SVM block-field ordering is hard-coded in
`blockFields`, which can drift from the Rust store’s field codes and cause
`eventBlockFieldMask` to select the wrong columns. Replace the inline string
list with an authoritative source, similar to how `transactionFields` is derived
from `Internal`, and ensure `BlockStore.makeMaskFn` continues to use the same
canonical ordinal order for `SvmBlockField`.
In `@scenarios/test_codegen/test/SvmHyperSyncSource_test.res`:
- Around line 17-61: Derive the block field mask from selectedBlockFields in
makeEventConfig instead of hardcoding blockFieldMask to 0. Mirror the same Svm
event config builder logic used for transactionFieldMask by computing the mask
from the selected block fields before constructing the
Internal.svmInstructionEventConfig record, so BlockStore sees the intended
block-field selection.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: fe63ae69-2034-4bcb-b4f0-ea11449f092e
⛔ Files ignored due to path filters (2)
packages/cli/src/hbs_templating/snapshots/envio__hbs_templating__codegen_templates__test__envio_types_dts_generated_for_svm.snapis excluded by!**/*.snappackages/cli/src/hbs_templating/snapshots/envio__hbs_templating__codegen_templates__test__svm_block_ts_type_renders_optionality_and_unselected.snapis excluded by!**/*.snap
📒 Files selected for processing (48)
packages/cli/src/block_store.rspackages/cli/src/config_parsing/human_config.rspackages/cli/src/config_parsing/public_config.rspackages/cli/src/config_parsing/system_config.rspackages/cli/src/evm_hypersync_source/mod.rspackages/cli/src/field_columns.rspackages/cli/src/hbs_templating/codegen_templates.rspackages/cli/src/lib.rspackages/cli/src/svm_hypersync_source/mod.rspackages/cli/src/svm_hypersync_source/types.rspackages/cli/src/transaction_store.rspackages/envio/index.d.tspackages/envio/src/ChainFetching.respackages/envio/src/ChainState.respackages/envio/src/ChainState.resipackages/envio/src/Config.respackages/envio/src/Core.respackages/envio/src/Envio.respackages/envio/src/EventConfigBuilder.respackages/envio/src/EventProcessing.respackages/envio/src/Internal.respackages/envio/src/sources/BlockStore.respackages/envio/src/sources/Evm.respackages/envio/src/sources/FieldMask.respackages/envio/src/sources/HyperFuelSource.respackages/envio/src/sources/HyperSync.respackages/envio/src/sources/HyperSync.resipackages/envio/src/sources/HyperSyncClient.respackages/envio/src/sources/HyperSyncSource.respackages/envio/src/sources/RpcSource.respackages/envio/src/sources/SimulateSource.respackages/envio/src/sources/Source.respackages/envio/src/sources/Svm.respackages/envio/src/sources/SvmHyperSyncClient.respackages/envio/src/sources/SvmHyperSyncSource.respackages/envio/src/sources/TransactionStore.respackages/envio/svm.schema.jsonscenarios/fuel_test/test/HyperFuelSource_test.resscenarios/test_codegen/test/BlockStore_test.resscenarios/test_codegen/test/EventRouter_svm_test.resscenarios/test_codegen/test/HyperSyncClient_test.resscenarios/test_codegen/test/IndexerState_test.resscenarios/test_codegen/test/RpcSource_test.resscenarios/test_codegen/test/SourceBlockHashes_test.resscenarios/test_codegen/test/SvmHyperSyncClient_test.resscenarios/test_codegen/test/SvmHyperSyncSource_test.resscenarios/test_codegen/test/helpers/MockIndexer.resscenarios/test_codegen/test/rollback/ChainMocking.res
…zation, hash optionality Three defects in the SVM block-field path found during review: - Config.res `svmEventDescriptorSchema` never declared `blockFields`, so rescript-schema stripped it on parse and every SVM block-field selection was silently dropped before reaching the event config (whole feature dead). - The SVM source stamped only `slot` inline while `materializeSvmItems` expected time/hash from the store, but the Rust gate never populates the store for the always-fetched blockhash/block_time — so selecting only `time`/`hash` left them undefined. Stamp the slot/time/hash trio inline from the response (mirroring EVM's item-stamped trio) and gate materialize on `hasExtraFields` so only height/parentSlot/parentHash consult the store. - Codegen marked `SvmBlockField::Hash` required though `svmInstructionBlock.hash` is optional (a slot can lack a block row); make it optional and assert name+optionality parity against Envio.res in the runtime-match test. Also correct stale "slot/time/hash always excluded" comments (only `slot` is) and drop redundant field docs. Tests: schema-preserves-blockFields regression, SVM source stamps the inline trio, materialize keeps the trio when only time/hash selected, and an SVM trio-field-code (0/1/2) pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
* Address block store review: avoid redundant work, dedupe mask folding, tighten tests
- Strip the always-stamped trio's bits before the EVM materialize call, since
setBlockHeader immediately overwrites them from the item (avoids decoding/
hex-encoding hash+timestamp for every block only to discard the result).
- Move build_columns' mask-union fold inside the function instead of
duplicating it at all 4 call sites (both EVM/SVM decoders in block_store.rs
and transaction_store.rs).
- Avoid a full raw-Block clone per SVM block: take the response's blocks once,
build the lean per-slot header from a borrow, then move the owned blocks
into the store (mirrors the EVM source's existing borrow-then-move pattern).
- Name the SVM raw-block-store gate's required query fields as a constant
instead of an inline string match, mirroring EVM's REQUIRED_BLOCK_FIELDS.
- Add BlockStore tests that exercise the public materialize() method with a
populated store (EVM + SVM), and a merge test asserting a re-fetched block
overwrites the stale entry for the same number - closing the gap where only
the empty-store path and private decode_* functions were tested.
* Move EVM block trio resolution into BlockStore instead of materialize params
Store an always-known header (timestamp/hash) alongside every inserted EVM
block, keyed the same way as the raw form, rather than threading them through
materialize() as extra caller-supplied arrays. This keeps materialize()'s
contract simple (block_numbers + masks, decode purely from store state) and
extends the same pattern Number already used (resolve from the map key)
to the rest of the trio.
- BlockStore::insert_evm now takes (number, timestamp, hash, raw: Option<Arc<Block>>)
instead of insert_evm_raw(number, raw) - the header is always inserted, the
full raw block only when an event selected a field beyond the trio.
- decode_evm_block_columns overlays number/timestamp/hash from the stored
header unconditionally, so a caller no longer needs to pass them in.
- evm_hypersync_source::process_response inserts a header for every response
block, reusing the already-computed out_blocks headers instead of gating
the store insert on has_extra_block_fields.
- BlockStore.res drops setBlockHeader (post-hoc JS mutation) in favor of
makeBlockHeader, used only for the no-store-lookup fast path; the Rust-
backed path returns fully-formed objects directly.
- Rust tests updated: the trio-specific decode tests now target header
resolution instead of Number's old mask-gated key lookup, and
decode_applies_each_rows_own_mask exercises a genuine extra field (gasUsed)
since the trio is no longer mask-gated.
* Make block-store trio handling fully mask-driven; drop timestamp/blockHash from items
The always-available number/timestamp/hash special-casing was spread across
the code (JS hasExtraFields/& ~7 gates, trio-stripped masks, an unconditional
header overlay in Rust decode). Now it lives in one place: the config-side
field selection, which always extends the EVM selection with the trio, so
every EVM mask carries its bits. Everything downstream is generic mask logic:
- decode_evm_block_columns is a plain build_columns pass; each arm names its
data source (number from the key, timestamp/hash from the stored header,
the rest from the raw block) and all are gated by each row's own mask.
- BlockStore.res loses hasExtraFields/stripTrioBits/setBlockHeader entirely.
materializeEvmItems always materialises through the store; the SVM path's
only gate is the generic "any non-zero mask".
With event.block now built solely from the store/payload, the item-level
timestamp and blockHash duplicates have no remaining consumers, so they are
removed from Internal.item/eventItem:
- raw_events' block_hash/block_timestamp columns are extracted from the
payload block by the ecosystem toRawEvent (names differ per ecosystem).
- The progress-latency metric reads the payload block via the ecosystem's
blockTimestampName, threaded into applyBatchProgress.
- Sources and test fixtures stop constructing the fields.
* Collapse EvmHeader back into simple_types::Block
EvmHeader{timestamp,hash} + an optional raw block was a memory optimization:
a header-only entry is ~98 bytes vs 384 bytes for Arc<simple_types::Block>
(measured via size_of - the full struct's ~27 boxed-Option fields cost a
pointer slot each regardless of which are populated). But that split adds a
second stored type, a raw-extraction step in decode, and a header/raw
insert split - real code weight for a per-chain-type memory saving.
Traded the memory optimization for simplicity: StoredBlock::Evm goes back to
holding Arc<simple_types::Block> alone, decode_evm_block_field reads
timestamp/hash straight off it via fill_masked exactly like every other
field, and the store keeps every response block (not just ones with extra
fields selected) since the trio now depends on the record being present.
* Add chunked columnar store core
One fetch response = one immutable chunk: sorted keys + one column per
selected field (flat fixed/var byte columns with validity bitmaps, boxed
lists for collection cells). Store lifecycle is append/drop chunks:
prune drops whole chunks below progress (watermark on a bisected one),
rollback truncates tails. Reads resolve per field newest-chunk-first,
which unions rows that arrived with different field subsets and
deduplicates identical rows re-fetched by overlapping partitions.
Adjacent runs of small chunks coalesce (k-way merge preserving
newest-wins) once the stack exceeds 64 chunks.
* Rewrite block store on chunked columns
Each fetch response's blocks become one chunk (sorted number/slot keys +
only the populated fields' columns) instead of per-row Arc<struct>
entries in a BTreeMap. materialize gathers the requested cells under the
lock into scratch columns and decodes off-thread as before; napi surface
and decode output are unchanged. Sources hand the store their owned
response blocks wholesale (insert_evm_blocks/insert_svm_blocks).
Per stored EVM block this drops the fixed 368-byte struct shell (plus
per-field boxes) to roughly the selected fields' payload bytes.
* Rewrite transaction store on chunked columns + token-balance sub-table
Replace the per-row simple_types::Transaction map with the shared
ChunkStore keyed by (block, txIndex). SVM token balances move to a
duplicate-key companion table in the same store, gathered by key range
at materialisation; the old HashMap union in build_svm_store goes away
since transactionIndex is key-derived.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UbaYTDU4XoGRSDGsQyDhsn
---------
Co-authored-by: Claude <noreply@anthropic.com>
3 tasks done
* Make SVM block materialization mirror EVM: always-included slot/time/hash trio SVM's slot/time/hash are now an always-included trio (like EVM's number/timestamp/hash), force-unioned into every event's field selection rather than being user-selectable. The Rust block store is now always populated for SVM (removing the has_extra_block_fields gate), so materialize() always returns a self-sufficient block. This lets SvmHyperSyncSource stop stamping an inline block at item construction (mirroring EVM, which omits `block` until batch prep), and removes enrichBlock/Object.assign entirely: materializeEvmItems and materializeSvmItems collapse into one BlockStore.materializeItems, since both ecosystems now follow the identical group/materialize/assign path. time/hash come out of the selectable SvmBlockField config enum (this feature is unreleased, so no existing configs break); generated instruction block types now render them as always-present nullable fields instead of FieldNotSelected sentinels, matching slot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rkB51uUrb79UrD78sq4Xc * Trim comments to non-obvious invariants only Drop comments that just narrated the refactor or restated what the surrounding code already shows (e.g. "mirroring EVM's X"), keeping only the ones explaining a fact a reader couldn't get from the code itself (why time/hash stay nullable, why grouping relies on item ordering). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rkB51uUrb79UrD78sq4Xc * Regenerate svm.schema.json after SvmBlockField doc comment trim schemars reads the SvmBlockField doc comment into the schema description, so the comment-cleanup commit left the committed schema stale. * Make svmInstructionBlock.hash non-optional, matching EVM's trio Solana's blockhash is a non-optional String at the HyperSync source (unlike block_time/parent_slot/parent_blockhash/block_height, which are genuinely Option<T>), and EVM's equivalent always-included fields (number/timestamp/hash) render as non-optional despite the same theoretical missing-row concern. hash inherited its `| undefined` from when it was still a user-selectable field, before it joined the always-included trio; that reasoning no longer applies now that hash is guaranteed alongside slot. time keeps `| undefined`, since Solana can genuinely omit a block's time even when the block row is present. * Stop marking SVM fields optional-when-selected unless genuinely nullable svm_block_field_specs/svm_transaction_field_specs derived optionality from the materializer's Option<T> columns, but that Option-ness mostly just means "the wire query didn't request this column" — SvmHyperSyncSource unions every event's selection into the query it issues chain-wide, so once a field is selected, the returned column is real. That's the same pattern EVM's own optional block fields already follow (query-gated, rendered non-optional once selected). height/parentSlot/parentHash (block) and feePayer/success/fee/ computeUnitsConsumed/recentBlockhash (transaction) now render as their plain type once selected. err/version keep `| undefined`: those are genuinely null by business logic regardless of selection (err on a successful tx, version on a legacy one), not a selection artifact. svm_block_field_specs_match_runtime now compares names only, matching the already-established svm_transaction_field_specs_match_runtime precedent: Envio.res's svmInstructionBlock is a pre-narrowing fallback that stays optional for any individually-selectable field regardless of this per-selected-field nullability question, so parity with it was never the right check for non-trio fields. --------- Co-authored-by: Claude <noreply@anthropic.com>
* Address block-store review findings: dedup stores, drop Fuel's dead BlockStore, fix SVM optionality - Flip computeUnitsConsumed back to optional in the generated SVM transaction type: Solana's own RPC docs describe it as reported only "when available" (compute-unit metering post-dates mainnet), so it can be absent even when selected. - Hoist the Ecosystem enum, bytes() helper, and field-name-listing boilerplate out of block_store.rs/transaction_store.rs and into the shared field_columns.rs module, removing four near-duplicate definitions. - Extract the "group adjacent items by key, OR-ing masks" algorithm shared by BlockStore.res's groupByBlock and TransactionStore.res's materializeItems into FieldMask.groupAdjacent, so the grouping invariant lives in one place. - Make ChainState's blockStore option<BlockStore.t> (None for Fuel), matching the existing page-level pattern, instead of always allocating a live but permanently-empty Rust store/Mutex for Fuel chains. This also removes the now-redundant ~ecosystem switch threaded through materializeBlockItems/ materializeBatchItems/materializePageItems/materializeBatchTransactions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PdHqppW444ZCCNxm1U8Lpj * Revert BlockStore Option<> change for Fuel, keep it always constructed Fuel chains keep a live BlockStore again instead of None, matching the original PR's design (every chain holds one, mirroring TransactionStore). The ~ecosystem switch in materializeBlockItems/materializeBatchItems/ materializePageItems/materializeBatchTransactions comes back with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PdHqppW444ZCCNxm1U8Lpj * Inline TransactionStore's grouping instead of the generic FieldMask helper Going through FieldMask.groupAdjacent here meant building a (blockNumber, transactionIndex) tuple per item (an allocation the original code avoided by comparing the two ints separately) plus an extra pass to unzip the grouped keys back into blockNumbers/transactionIndices for materialize. Not worth it on this per-batch hot path, so back to the specialized loop. BlockStore.res keeps using FieldMask.groupAdjacent — its key is a bare int, so it doesn't pay the tuple cost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PdHqppW444ZCCNxm1U8Lpj --------- Co-authored-by: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
scenarios/test_codegen/test/Config_test.res (1)
340-341: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDrop the helper-purpose comment.
This restates what
makeAddressFormatConfigJsonalready shows and doesn’t capture a non-obvious constraint.As per coding guidelines, comments in
**/*.resshould not restate what the code already says.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scenarios/test_codegen/test/Config_test.res` around lines 340 - 341, Remove the helper-purpose comment near makeAddressFormatConfigJson in Config_test.res; it only repeats what the function name and implementation already make clear. Delete the comment rather than replacing it, since the code already conveys the minimal single-chain, single-contract config setup and there is no non-obvious constraint to preserve.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/evm_hypersync_source/mod.rs`:
- Around line 343-351: The chunk deduplication in
evm_hypersync_source::mod::transaction processing is incomplete because
transaction_keys is updated but kept still collects duplicates. Update the logic
around the transaction_keys.insert call so kept.push(tx) only happens when the
key was newly inserted, using the existing transaction_keys and kept flow to
ensure transaction_store.insert_evm_txs receives one tx per (block_number,
transaction_index).
---
Nitpick comments:
In `@scenarios/test_codegen/test/Config_test.res`:
- Around line 340-341: Remove the helper-purpose comment near
makeAddressFormatConfigJson in Config_test.res; it only repeats what the
function name and implementation already make clear. Delete the comment rather
than replacing it, since the code already conveys the minimal single-chain,
single-contract config setup and there is no non-obvious constraint to preserve.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a25a2966-426f-43cc-a101-e7e6002940c5
⛔ Files ignored due to path filters (3)
packages/cli/src/hbs_templating/snapshots/envio__hbs_templating__codegen_templates__test__envio_types_dts_generated_for_svm.snapis excluded by!**/*.snappackages/cli/src/hbs_templating/snapshots/envio__hbs_templating__codegen_templates__test__svm_block_ts_type_renders_optionality_and_unselected.snapis excluded by!**/*.snappackages/cli/src/hbs_templating/snapshots/envio__hbs_templating__codegen_templates__test__svm_transaction_ts_type_renders_optionality_and_unselected.snapis excluded by!**/*.snap
📒 Files selected for processing (50)
packages/cli/src/block_store.rspackages/cli/src/chunk_store.rspackages/cli/src/config_parsing/human_config.rspackages/cli/src/config_parsing/public_config.rspackages/cli/src/config_parsing/system_config.rspackages/cli/src/evm_hypersync_source/mod.rspackages/cli/src/field_columns.rspackages/cli/src/hbs_templating/codegen_templates.rspackages/cli/src/lib.rspackages/cli/src/svm_hypersync_source/mod.rspackages/cli/src/svm_hypersync_source/types.rspackages/cli/src/transaction_store.rspackages/envio/index.d.tspackages/envio/src/ChainState.respackages/envio/src/ChainState.resipackages/envio/src/Config.respackages/envio/src/CrossChainState.respackages/envio/src/CrossChainState.resipackages/envio/src/Envio.respackages/envio/src/EventConfigBuilder.respackages/envio/src/IndexerState.respackages/envio/src/Internal.respackages/envio/src/RawEvent.respackages/envio/src/SimulateItems.respackages/envio/src/sources/BlockStore.respackages/envio/src/sources/Evm.respackages/envio/src/sources/FieldMask.respackages/envio/src/sources/Fuel.respackages/envio/src/sources/HyperFuelSource.respackages/envio/src/sources/HyperSyncSource.respackages/envio/src/sources/RpcSource.respackages/envio/src/sources/Source.respackages/envio/src/sources/Svm.respackages/envio/src/sources/SvmHyperSyncSource.respackages/envio/src/sources/TransactionStore.respackages/envio/svm.schema.jsonscenarios/test_codegen/test/BlockStore_test.resscenarios/test_codegen/test/ClientAddressFilter_test.resscenarios/test_codegen/test/Config_test.resscenarios/test_codegen/test/IndexerState_test.resscenarios/test_codegen/test/RpcSource_test.resscenarios/test_codegen/test/SvmHyperSyncSource_test.resscenarios/test_codegen/test/__mocks__/MockEvents.resscenarios/test_codegen/test/helpers/MockIndexer.resscenarios/test_codegen/test/lib_tests/CrossChainState_test.resscenarios/test_codegen/test/lib_tests/FetchState_onBlock_test.resscenarios/test_codegen/test/lib_tests/FetchState_test.resscenarios/test_codegen/test/lib_tests/HyperSyncDecoder_test.resscenarios/test_codegen/test/lib_tests/PgStorage_test.resscenarios/test_codegen/test/rollback/ChainMocking.res
💤 Files with no reviewable changes (7)
- scenarios/test_codegen/test/lib_tests/FetchState_test.res
- scenarios/test_codegen/test/lib_tests/FetchState_onBlock_test.res
- packages/envio/src/sources/HyperFuelSource.res
- scenarios/test_codegen/test/ClientAddressFilter_test.res
- scenarios/test_codegen/test/rollback/ChainMocking.res
- scenarios/test_codegen/test/mocks/MockEvents.res
- scenarios/test_codegen/test/IndexerState_test.res
✅ Files skipped from review due to trivial changes (2)
- packages/envio/src/sources/Source.res
- scenarios/test_codegen/test/RpcSource_test.res
🚧 Files skipped from review as they are similar to previous changes (14)
- packages/cli/src/lib.rs
- packages/envio/src/sources/TransactionStore.res
- packages/envio/svm.schema.json
- packages/cli/src/config_parsing/human_config.rs
- packages/envio/src/sources/Svm.res
- packages/envio/src/Config.res
- packages/cli/src/config_parsing/public_config.rs
- packages/cli/src/svm_hypersync_source/types.rs
- scenarios/test_codegen/test/helpers/MockIndexer.res
- scenarios/test_codegen/test/SvmHyperSyncSource_test.res
- packages/envio/src/sources/SvmHyperSyncSource.res
- packages/envio/src/sources/HyperSyncSource.res
- packages/envio/src/EventConfigBuilder.res
- packages/cli/src/config_parsing/system_config.rs
Comment on lines
343
to
+351
| if let (Some(block_number), Some(transaction_index)) = | ||
| (tx.block_number, tx.transaction_index) | ||
| { | ||
| let block_number = u64::from(block_number); | ||
| let transaction_index = u64::from(transaction_index) as u32; | ||
| transaction_keys.insert((block_number, transaction_index)); | ||
| transaction_store.insert_evm_raw(block_number, transaction_index, Arc::new(tx)); | ||
| transaction_keys | ||
| .insert((u64::from(block_number), u64::from(transaction_index) as u32)); | ||
| kept.push(tx); | ||
| } | ||
| } | ||
| transaction_store.insert_evm_txs(kept); |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Use transaction_keys to deduplicate the stored chunk.
transaction_keys.insert(...) currently only updates the coverage set; duplicates still get pushed into kept. Gate kept.push(tx) on the insert result so the page store receives one transaction per (blockNumber, transactionIndex).
Proposed fix
- {
- transaction_keys
- .insert((u64::from(block_number), u64::from(transaction_index) as u32));
- kept.push(tx);
- }
+ {
+ let key = (u64::from(block_number), u64::from(transaction_index) as u32);
+ if transaction_keys.insert(key) {
+ kept.push(tx);
+ }
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if let (Some(block_number), Some(transaction_index)) = | |
| (tx.block_number, tx.transaction_index) | |
| { | |
| let block_number = u64::from(block_number); | |
| let transaction_index = u64::from(transaction_index) as u32; | |
| transaction_keys.insert((block_number, transaction_index)); | |
| transaction_store.insert_evm_raw(block_number, transaction_index, Arc::new(tx)); | |
| transaction_keys | |
| .insert((u64::from(block_number), u64::from(transaction_index) as u32)); | |
| kept.push(tx); | |
| } | |
| } | |
| transaction_store.insert_evm_txs(kept); | |
| if let (Some(block_number), Some(transaction_index)) = | |
| (tx.block_number, tx.transaction_index) | |
| { | |
| let key = (u64::from(block_number), u64::from(transaction_index) as u32); | |
| if transaction_keys.insert(key) { | |
| kept.push(tx); | |
| } | |
| } | |
| } | |
| transaction_store.insert_evm_txs(kept); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/cli/src/evm_hypersync_source/mod.rs` around lines 343 - 351, The
chunk deduplication in evm_hypersync_source::mod::transaction processing is
incomplete because transaction_keys is updated but kept still collects
duplicates. Update the logic around the transaction_keys.insert call so
kept.push(tx) only happens when the key was newly inserted, using the existing
transaction_keys and kept flow to ensure transaction_store.insert_evm_txs
receives one tx per (block_number, transaction_index).
blockhash/parentBlockhash are base58 strings; storing them via var_from + utf8() round-tripped them through raw bytes and re-validated UTF-8 on every materialize read with String::from_utf8_lossy. Add a dedicated StrCol (like VarCol, but the shared buffer is a String) so the column is text end-to-end: no byte-slice cast on insert, and no UTF-8 re-validation on read since the buffer's validity is an invariant of String. Co-authored-by: Claude <noreply@anthropic.com>
* Replace chunk-columnar field stores with merge-on-insert Table Blocks, transactions, and SVM token balances now dedup on insert instead of accumulating immutable chunks that only get reconciled by coalescing or pruning. Each key owns one slot in a flat columnar Table; a re-fetched key (overlapping partitions, reorgs) overwrites its slot's cells in place, so a large field like EVM `input` is stored once rather than once per response. Var/list cells are individually boxed so an overwrite frees the old blob immediately. Token balances move from a duplicate-key chunk table to a Table keyed by (slot, transactionIndex, account); the SVM query layer now force-adds `account` to the field selection whenever token balances are requested, so the key component is always populated. chunk_store.rs is replaced by field_table.rs; the AnyCol interchange layer (fills/decoders/hex helpers) is unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPirgAMkqHXCoXAPkNDkAs * Address review feedback: trim comments, avoid a clone, add coverage - svm_hypersync_source/query.rs: scope the account force-add comment to our own caller convention instead of asserting general field-selection semantics that conflict with the vendored crate's own doc. - field_table.rs/transaction_store.rs: trim several doc comments that restated what the code already shows, keeping only the non-obvious invariants/preconditions. - transaction_store.rs: avoid cloning `account` per row in insert_svm_token_balances by building the var columns first (borrowing rows) and then consuming rows to build the keys. - Add missing test coverage: a TransactionStore-level merge-dedup test (mirrors the existing BlockStore one), a Table::clear() unit test, and a ReScript-level SvmHyperSyncSource test pinning that token-balance queries always carry `account`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPirgAMkqHXCoXAPkNDkAs --------- Co-authored-by: Claude <noreply@anthropic.com>
HyperSync added chain 4663 (Robinhood) to its API, which the hypersync-health-check asserts is present in the Network enum. Add it as a HypersyncChain; endpoint URL auto-derives from the chain id and reorg depth defaults to unknown (None). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/config_parsing/chain_helpers.rs`:
- Around line 345-347: Robinhood is currently only included under
HypersyncChain, so it is omitted from explorer-backed paths that rely on
NetworkWithExplorer. Update the Robinhood enum entry in chain_helpers to also be
part of NetworkWithExplorer, keeping the existing HypersyncChain membership
intact so the chain is available to both flow types.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 2fed13ef-d686-49dd-9019-1137f8eb73ec
📒 Files selected for processing (1)
packages/cli/src/config_parsing/chain_helpers.rs
…chitecture-7alrtx # Conflicts: # packages/envio/src/EventConfigBuilder.res # packages/envio/src/Internal.res # packages/envio/src/RawEvent.res # packages/envio/src/SimulateItems.res # packages/envio/src/sources/Evm.res # packages/envio/src/sources/HyperFuelSource.res # packages/envio/src/sources/HyperSyncSource.res # packages/envio/src/sources/RpcSource.res # packages/envio/src/sources/SvmHyperSyncSource.res # scenarios/fuel_test/test/HyperFuelSource_test.res # scenarios/test_codegen/test/ClientAddressFilter_test.res # scenarios/test_codegen/test/EventRouter_svm_test.res # scenarios/test_codegen/test/SourceBlockHashes_test.res # scenarios/test_codegen/test/helpers/MockIndexer.res # scenarios/test_codegen/test/lib_tests/CrossChainState_test.res # scenarios/test_codegen/test/lib_tests/FetchState_onBlock_test.res # scenarios/test_codegen/test/lib_tests/FetchState_test.res # scenarios/test_codegen/test/lib_tests/HyperSyncDecoder_test.res # scenarios/test_codegen/test/lib_tests/PgStorage_test.res
DZakh
commented
Jul 8, 2026
| // Materialise store-backed transactions onto payloads before any handler | ||
| // (preload or execute) reads them. | ||
| await materializeBatchTransactions(batch, ~chainStates) | ||
| await materializeBatchTransactions(batch, ~chainStates, ~ecosystem=config.ecosystem.name) |
Member
Author
There was a problem hiding this comment.
This should be renamed into something like materializeBatchEvents - and contain both transaction and block materialization. Also I think we can optimize it by grouping the transaction and block into one function where we'll have a single iteration over items to build both masks.
Member
Author
There was a problem hiding this comment.
Done in 7fadb1f: ChainState.groupBatchItems now walks items once, tracking the transaction-group run (blockNumber, transactionIndex) and the block-group run (blockNumber) independently, feeding both TransactionStore.materialize/BlockStore.materialize. Also renamed materializeBatchTransactions → materializeBatchEvents in EventProcessing.res since it already covered both. Added a regression test (ChainState_materialize_test.res) pinning that a transaction-run break doesn't wrongly interrupt a same-block run in the block dimension.
Generated by Claude Code
- Add NetworkWithExplorer to Robinhood (chain 4663 has a public Blockscout explorer at robinhoodchain.blockscout.com), so contract-import can use it. - Rename EventProcessing.materializeBatchTransactions to materializeBatchEvents: it already materialises both transaction and block fields via ChainState.materializeBatchItems, so the old name was misleading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
`make update-help`: adding NetworkWithExplorer to Robinhood added `robinhood` to the --blockchain option's possible values, which the generated docs snapshot test checks against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
…tems Per review feedback on EventProcessing.res: ChainState.materializeBatchItems and materializePageItems each called TransactionStore.materializeItems and BlockStore.materializeItems, which independently re-walked the same items array to build their own grouping/masks. Replace with groupBatchItems, a single Array.forEach that builds both the (blockNumber, transactionIndex) transaction groups and the blockNumber-only block groups together, tracking each dimension's adjacent-run independently since they can diverge mid-batch. The two materialize calls against the Rust stores still run concurrently. Behavior is unchanged (verified against the full test suite); added a regression test pinning that a transaction-group break doesn't wrongly interrupt a same-block run in the block dimension, and vice versa. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
- FieldMask.groupAdjacent had exactly one caller (BlockStore.groupByBlock); TransactionStore inlines its own grouping and ChainState.groupBatchItems now does too, so the generic version served no other consumer. Inline its logic directly into BlockStore.groupByBlock and remove the abstraction. - RpcSource.res: getBlockTimestamp/getBlockHash externals were only used to stamp timestamp/blockHash onto Internal.item, which no longer carries those fields (they're read off the payload's own block now). getBlockNumber is still used for the item's blockNumber. - svm_hypersync_source/types.rs: TryFrom<simple::Block> for Block was only reachable via QueryResponse::try_from's generic try_map(r.blocks), but that call's only call site always empties r.blocks first (the raw blocks go into the BlockStore, and the header is filled in afterwards from Block::from_raw) so the conversion only ever ran on an empty Vec. Build blocks: Vec::new() directly instead and drop the now-unreachable TryFrom impl. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
…materializeItems Neither had a real caller left: ChainState.materializeBatchItems/ materializePageItems now go through groupBatchItems + applyTransactionGroups/ applyBlockGroups, calling the raw materialize bindings directly. These two per-store functions were kept alive only by their own unit tests. Migrate the coverage they had (dedup by adjacency, mask OR-ing across a shared group, empty-mask still stamps a value, inline items skipped) into ChainState_materialize_test.res against materializePageItems, the path that actually runs in production. BlockStore_test.res/TransactionStore_test.res keep only their field-code contract tests (fieldCodes/orMask/blockFields ordering), which are independent of the removed functions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
DZakh
pushed a commit
that referenced
this pull request
Jul 8, 2026
…get-redesign-yl7t1u Brings in the block-store/ChainState rework (#1369) and per-call metrics aggregation (#1390). Auto-merged cleanly; verified envio + test_codegen compile and the scheduler, rollback, SourceManager, and IndexerState tests pass against the merged tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174BGgup49cCEGmFyBW7Wyx
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a per-chain block store (
BlockStore) that mirrors the existing transaction store pattern. Blocks are kept as raw upstream structs in Rust (keyed by block number/slot), deduplicated across many events that share them, and materialized on-demand into columnar form for bulk decoding off the JS thread.Key Changes
New
BlockStore(Rust): Parallel toTransactionStore, stores EVM and SVM blocks by number/slot with merge/prune/rollback lifecycle. Decodes selected fields in bulk into columnar form viadecode_evm_block_columnsanddecode_svm_block_columns.Extracted
field_columnsmodule: Factored out columnar materialization logic shared by both stores (Columnenum,Columnsstruct,build_columns,fill_masked). MovedSvmTokenBalanceOuthere as a shared type.Block field selection: Added
EvmBlockFieldandSvmBlockFieldenums with ordinal-based field codes (matching ReScript field arrays). EVM supports 27 fields (number, timestamp, hash, etc.); SVM supports 6 (slot, time, hash, height, parentSlot, parentHash).ReScript bindings: New
BlockStore.resmodule withnewEvm/newSvm/newFuelconstructors,merge,materialize,prune,rollbackmethods. AddedFieldMask.resfor shared bitmask helpers (makeMaskFn,orMask).Config & codegen: Extended SVM config to accept
block_fieldsselection. UpdatedEventConfigBuilderand codegen templates to computeblockFieldMaskper event. Generated types now includeblockfield on instruction types with per-field optionality.Integration:
ChainStatenow holds ablockStorealongsidetransactionStore. HyperSync sources return(items, transactionStore, blockStore)tuples. Block materialization happens at batch prep viamaterializeEvmItems/materializeSvmItems(analogous to transaction materialization).Tests: Added
BlockStore_test.resvalidating field-code contract between ReScript and Rust. Updated existing tests to pass block stores through the pipeline.Notable Details
Number/Slotfields resolve from the key regardless of fetch success.enrichBlock.blockomitted from the payload (lives in store); RPC/simulate/Fuel build it inline.https://claude.ai/code/session_01Pk9WdM4e4PyeG6TVXrDyPh
Summary by CodeRabbit
New Features
Bug Fixes