feat(snapshot): export a stele from a live node

[scarmuega]

Plan: plans/dolos-stelae-export.md in the TxPipe domain root (Phase 2 of dolos-stelae-snapshots, the export half). Base is main at 2276f55a — the #1167 streaming-sink merge this plan awaited.

Done criterion: met. All seven items below, with the verification run at the end.

What changed

crates/snapshot/src/export.rs is the whole of it: the driver that walks a live store set and hands records to the protocol in ADR-004's order. It invents no byte shape — every record shape, ordering rule and name was already pinned by a golden in #1166 — and it streams through SteleDir::layer_sink, so no layer is ever materialized.

• Per-epoch layers. blocks from ArchiveStore::get_range, decoding each block's header for the hash the codec takes as an input; logs from iter_logs over NAMESPACES in registry order, which is the (ns, log_key) sortedness the codec requires; indexes from iter_archive_tags then iter_exact_records, in the order the trait already promises.
• State. One walk of the store with sixteen sinks open, each record routed by shard_of and checked by its shard's own OrderCheck. All sixteen shards are always written, empty ones included, so the shard count a reader sees is never data-dependent.
• dolos snapshot publish --output-dir DIR [--epochs RANGE] [--dry-run], a new command group wired into main.rs. The epoch range takes Rust's own spellings (500..520, 500..=520, 500.., ..520, 500) because silently picking one meaning of .. is how an operator publishes an epoch short.

Two facts nothing else in the tree settled are settled here.

position.network.name has no source, so it is now a fixed function of the magic. Network::for_magic is the only constructor and the fields are private: 764824073 → mainnet, 1 → preprod, 2 → preview, anything else → testnet-{magic}. The name rides inside the canonical JSON, so a name read from user configuration would let two publishers on one chain produce two different digests over a spelling.

Epoch geometry. sequence is epoch_of(cursor) + 1 — ADR-004's E, the newly started epoch — and layers cover 0..=epoch_of(cursor). The final window's end_slot is min(epoch_start(e+1) - 1, cursor.slot), so a stele cut mid-epoch is still byte-identical between two publishers standing at the same point. Landing on a true boundary stays the publisher pipeline's job.

ToyDomain gains a ToyStores backend parameter defaulting to the builtin memory pair, so the same harness can be pointed at fjall. That is what makes criterion 3 possible without a sibling harness type; every existing consumer compiles unchanged.

Done criteria

1. Complete stele from a harness-built store set, through the sink rather than a materialized layer — a_harness_domain_exports_a_complete_stele.
2. Read-back equality per kind, including a non-empty logs layer, both index record shapes, and all sixteen state shards — every_layer_reads_back_as_the_store_yields_it. Each expectation is collected from the store and sorted independently rather than by repeating the exporter's registry walk, so an ordering bug shared by both would still show.
3. Determinism. both_backends_publish_the_same_inscription applies the same blocks to a fjall-backed store set and a builtin-memory-backed one and compares layer by layer, then the whole document, then the blobs. It passed on the first run — the residual risk ADR-004 named did not materialize, and no entity encoder needed a fix.
4. A golden inscription — an_empty_store_set_exports_the_pinned_skeleton. Deliberately over an empty store set: the layers carry only their header records, so the pinned document is a function of literals alone and freezes the network table, the sequence/position/parameters shapes, the order layers are listed in, and every header and scope encoding — without being something a ledger change re-pins as a matter of routine.
5. The CLI's directory opens and verifies — tests/snapshot_publish.rs builds a real on-disk node from a dolos.toml, runs the actual binary as a subprocess, and checks SteleDir::open plus a clean blob_index (which decompresses and re-hashes every blob). Also covers --dry-run, --epochs, and that republishing over an existing stele is refused.
6. The gate — below.
7. The cost of iter_exact_records — below.

Criterion 7: the per-epoch index cost

Measured with the harness #1165 built (cargo test --release --test index_roundtrip -- --ignored --nocapture measure_one_epoch_iteration_cost), fjall, Apple M4 / 16 GiB:

store depth	tag records	exact records	one epoch slice
8 epochs	5,184,000	864,000	800.7 ms (729.6 ms tags + 71.0 ms exacts)
32 epochs	20,736,000	3,456,000	2.973 s (2.579 s tags + 393.8 ms exacts)

Four times the depth costs 3.7 times the time to slice one epoch, and the epoch sliced is the same size in both rows. The cost is O(store), not O(range) — exactly as the trait documents.

So a first publish is O(epochs²). Extrapolating to mainnet's ~580 epochs at the fixture's per-epoch shape: ~376M tag records and ~63M exact records, at the measured ~8.0M tags/s and ~8.8M exacts/s, is ~54 s per epoch layer and ~8.7 hours for the indexes layers of one publish. The fixture assumes 3 txs per block, well under mainnet's recent density, so that is a floor.

This is stated, not absorbed. It is filed as plans/dolos-stelae-publish-cost.md (draft) — band the traversal over K epochs per pass, the same move the state pass makes with sixteen shard sinks, with K sized against a memory ceiling rather than by taste. Filed together with progress reporting, which the export slice left out deliberately rather than widening its own API, and which an eight-hour silent command obviously needs.

Two other findings

EntityKey::full_range() excludes its own maximum. Its end bound is [0xff; 32] and iter_entities is half-open, so an entity keyed with thirty-two 0xff bytes is invisible to every caller, export included. A definitional limit of a fixed-width key type rather than a defect — there is no representable exclusive bound above the maximum key — and unreachable in practice since state keys are hash-derived. Noted at the call site rather than silently inherited.

The harness fixture cannot cross an epoch boundary under strict. Preview at protocol 6 raises PParamsNotFound("drep_inactivity_period") on the epoch-0/1 transition; devnet overflows the pots. Same class as the exclusions CI already carries for dolos-minibf/minikupo/trp. Rather than joining that list, the export fixtures stay inside one epoch and multi-epoch geometry is pinned by the three-epoch skeleton golden and Plan's unit tests instead — it is arithmetic over a ChainSummary, not something a ledger fixture proves.

Verification

• cargo test --workspace --all-targets — green.
• cargo test --workspace --all-targets --all-features --exclude dolos-minibf --exclude dolos-minikupo --exclude dolos-trp (CI's command) — green, 28 targets. The two new targets are not excluded.
• cargo clippy --all-targets --all-features -- -D warnings — clean.
• cargo +nightly fmt --all -- --check — clean.
• cargo deny check advisories — ok.
• cargo tree -p stelae -e normal — matches nothing ^dolos(-|$).

No escalation was raised. No store trait changed and crates/stelae is untouched.

Draft until reviewed; this role does not merge its own work.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added snapshot export support for blocks, indexes, logs, state, and optional digest layers.
  • Added the dolos snapshot publish command with epoch selection, dry-run mode, and output validation.
  • Added network identification for mainnet, preprod, and preview environments.
• Bug Fixes
  • Added validation for invalid cursors, empty selections, duplicate outputs, and undecodable blocks.
• Testing
  • Added comprehensive unit and end-to-end coverage for snapshot publishing and deterministic output.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8899aefe-2e39-4dbb-b610-38dc5f09b825

📥 Commits

Reviewing files that changed from the base of the PR and between 5265d65 and dfc4703.

📒 Files selected for processing (1)

• tests/snapshot_publish.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/snapshot_publish.rs

---

📝 Walkthrough

Walkthrough

This PR adds live snapshot planning and export to dolos-snapshot, adds dolos snapshot publish to the CLI, derives network names from network magic, and extends memory and Fjall test coverage.

Changes

Snapshot export and publish flow

Layer / File(s)	Summary
Snapshot API and network foundation Cargo.toml, crates/snapshot/Cargo.toml, crates/snapshot/src/lib.rs, crates/snapshot/tests/common/mod.rs	Adds dependencies, exposes export, extends errors, and derives network identity from magic values.
Export plan and epoch geometry crates/snapshot/src/export.rs	Adds epoch windows and plans with cursor validation, clamping, epoch restriction, and network-aware metadata.
Snapshot layer writing and publication crates/snapshot/src/export.rs	Writes block, log, index, sharded state, and optional digest layers, then creates and validates the inscription.
CLI snapshot publish command src/bin/dolos/main.rs, src/bin/dolos/snapshot/*	Adds the publish command, epoch-range parsing, dry-run handling, store loading, plan restriction, and output reporting.
Backend abstraction and export validation crates/testing/src/toy_domain.rs, crates/snapshot/tests/export.rs, tests/snapshot_publish.rs	Adds generic memory and Fjall backends and validates exported content, determinism, CLI behavior, and epoch selection.

Estimated code review effort: 5 (Critical) | ~100 minutes

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant CLI as dolos snapshot publish
  participant Stores as archive/state/index stores
  participant Export as dolos_snapshot::export
  participant Output as stele directory

  User->>CLI: run publish with output_dir and optional epochs
  CLI->>Stores: open configured stores and genesis data
  CLI->>Export: build Plan and apply epoch restriction
  Export->>Stores: read blocks, logs, indexes, state, and digests
  Export->>Output: write snapshot layers and inscription
  CLI-->>User: report digest, layer count, and size

Loading

Possibly related PRs

• txpipe/dolos#1166: Adds snapshot profile, layer codecs, scopes, and Network APIs used by this PR.
• txpipe/dolos#1150: Adds store iteration and prehashed index APIs consumed by snapshot export.
• txpipe/dolos#1147: Provides Stelae APIs used for stele, layer, inscription, and profile generation.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: exporting a Stele snapshot from a live node.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/stelae-export

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (6)

crates/snapshot/src/export.rs (1)

521-637: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a unit test for the empty digests refusal.

The tests cover the epoch geometry rules well. One documented contract in this file has no test: write_digests rejects an empty record slice with the message "a digests layer with no records names no immutable file; omit the layer instead". A caller reaches it through export with Some(&[]). A test pins that error variant so a later refactor cannot turn the empty case into a layer that names immutable file zero.

Also consider a case for restrict_epochs with bounds outside the covered range, which lines 145-146 document as selecting nothing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/snapshot/src/export.rs` around lines 521 - 637, Add a unit test in the
existing tests module that calls export with Some(&[]) for digests and asserts
write_digests rejects it with the documented error message about omitting an
empty digests layer; also add coverage for restrict_epochs bounds outside the
available epochs selecting no windows if supported by the existing API.

crates/snapshot/tests/export.rs (1)

213-215: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use u64::from instead of an as cast.

src/bin/dolos/snapshot/publish.rs line 99 calls the same API as u64::from(genesis.network_magic()). This line uses as u64. Keep the two call sites the same, and prefer the infallible conversion: if network_magic() ever returns a wider type, as u64 truncates silently while u64::from fails to compile.

♻️ Proposed change

 fn plan_for<B: ToyStores>(domain: &ToyDomain<B>) -> Plan {
-    export::plan(domain.state(), domain.genesis().network_magic() as u64).unwrap()
+    export::plan(domain.state(), u64::from(domain.genesis().network_magic())).unwrap()
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/snapshot/tests/export.rs` around lines 213 - 215, Update plan_for’s
network_magic conversion to use the infallible u64::from form instead of an as
u64 cast, matching the corresponding snapshot publish call site while leaving
the export::plan invocation unchanged.

tests/snapshot_publish.rs (1)

185-188: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert that the refused publish left the first stele intact.

The comment states the refusal prevents a second stele's blobs from layering on top of the first. The assertion checks only the exit status. A command that wrote some blobs and then failed would still pass.

Re-open and re-verify the directory after the refused run to prove the stated property.

♻️ Proposed change

     let second = node.publish(&out, &[]);
     assert!(!second.status.success());
+
+    // The refused run left the first stele exactly as it was.
+    let reopened = SteleDir::open(&out).unwrap();
+    assert_eq!(reopened.read_inscription().unwrap(), inscription);
+    assert_eq!(reopened.blob_index().unwrap().len(), index.len());

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/snapshot_publish.rs` around lines 185 - 188, Extend the refused
republish test around node.publish so it reopens the existing output directory
after the failed second publish and verifies the original stele remains intact,
including its expected contents or identity. Keep the existing
unsuccessful-status assertion and use the test’s established directory-opening
and verification helpers where available.

crates/testing/src/toy_domain.rs (1)

196-200: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Document that _dir must stay the last field.

Struct fields drop in declaration order. state and indexes close before _dir removes the directory, which is the order fjall needs. If a later refactor moves _dir above the stores, TempDir::drop runs while fjall still holds open file handles. On Windows the removal then fails silently and the temporary directory leaks.

The doc comment explains why _dir is an Arc. Add why it is last.

♻️ Proposed change

 pub struct FjallStores {
     state: dolos_fjall::StateStore,
     indexes: dolos_fjall::IndexStore,
+    // Last on purpose: fields drop in declaration order, so both stores close
+    // before the directory is removed.
     _dir: Arc<tempfile::TempDir>,
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/testing/src/toy_domain.rs` around lines 196 - 200, Add a doc comment
to the _dir field in FjallStores documenting that it must remain the last field
so state and indexes are dropped before TempDir removes the directory; preserve
the existing explanation for why _dir is an Arc.

src/bin/dolos/snapshot/publish.rs (1)

108-108: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add .context to the remaining fallible calls.

Lines 99 and 137 attach context to their errors. Lines 108 and 148 do not. If plan.tag() or inscription.digest() fails, the operator sees a bare error with no indication of which step produced it.

♻️ Proposed change

-    let tag = plan.tag().into_diagnostic()?;
+    let tag = plan
+        .tag()
+        .into_diagnostic()
+        .context("computing the stele tag")?;

-    let digest = inscription.digest().into_diagnostic()?;
+    let digest = inscription
+        .digest()
+        .into_diagnostic()
+        .context("computing the inscription digest")?;

Also applies to: 148-148

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bin/dolos/snapshot/publish.rs` at line 108, Update the fallible calls in
the snapshot publish flow at plan.tag() and inscription.digest() to attach
descriptive .context messages before converting their errors with
into_diagnostic(). Match the existing contextual error-handling style used by
the nearby calls so failures identify the operation that produced them.

src/bin/dolos/main.rs (1)

11-11: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Gate the snapshot module with a dedicated Cargo feature.

The published Snapshot command and its dolos-snapshot dependency are enabled unconditionally in Cargo.toml, unlike nearby optional commands such as bootstrap, minibf, and minikupo. Gate mod snapshot, the command variant, and the dependency under a snapshot feature so dolos-snapshot and stelae are not included in a minimal node build unless publishing is intended.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bin/dolos/main.rs` at line 11, Gate the snapshot functionality behind a
dedicated snapshot Cargo feature: conditionally compile the mod snapshot
declaration in main.rs and the corresponding Snapshot command variant, and mark
the dolos-snapshot dependency and stelae dependency optional in Cargo.toml while
wiring them into the feature. Preserve the existing behavior when the snapshot
feature is enabled and exclude these dependencies from minimal node builds.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/snapshot/src/export.rs`:
- Around line 385-397: Update export() to use one consistent IndexStore snapshot
for all block, index, log, state, archive-tag, and exact-record reads, rather
than opening independent point-in-time views per store call. Ensure every
iterator, including iter_archive_tags and iter_exact_records, reads from that
shared snapshot; alternatively, revalidate Plan::cursor after publishing and
fail if it changed.

In `@tests/snapshot_publish.rs`:
- Around line 35-58: Update the TOML template in the snapshot_publish test to
use single-quoted literal strings for the interpolated data and genesis paths,
including storage.path, byron_path, shelley_path, alonzo_path, and conway_path.
Leave non-path string values unchanged so Windows backslashes are treated
literally during toml::from_str parsing.

---

Nitpick comments:
In `@crates/snapshot/src/export.rs`:
- Around line 521-637: Add a unit test in the existing tests module that calls
export with Some(&[]) for digests and asserts write_digests rejects it with the
documented error message about omitting an empty digests layer; also add
coverage for restrict_epochs bounds outside the available epochs selecting no
windows if supported by the existing API.

In `@crates/snapshot/tests/export.rs`:
- Around line 213-215: Update plan_for’s network_magic conversion to use the
infallible u64::from form instead of an as u64 cast, matching the corresponding
snapshot publish call site while leaving the export::plan invocation unchanged.

In `@crates/testing/src/toy_domain.rs`:
- Around line 196-200: Add a doc comment to the _dir field in FjallStores
documenting that it must remain the last field so state and indexes are dropped
before TempDir removes the directory; preserve the existing explanation for why
_dir is an Arc.

In `@src/bin/dolos/main.rs`:
- Line 11: Gate the snapshot functionality behind a dedicated snapshot Cargo
feature: conditionally compile the mod snapshot declaration in main.rs and the
corresponding Snapshot command variant, and mark the dolos-snapshot dependency
and stelae dependency optional in Cargo.toml while wiring them into the feature.
Preserve the existing behavior when the snapshot feature is enabled and exclude
these dependencies from minimal node builds.

In `@src/bin/dolos/snapshot/publish.rs`:
- Line 108: Update the fallible calls in the snapshot publish flow at plan.tag()
and inscription.digest() to attach descriptive .context messages before
converting their errors with into_diagnostic(). Match the existing contextual
error-handling style used by the nearby calls so failures identify the operation
that produced them.

In `@tests/snapshot_publish.rs`:
- Around line 185-188: Extend the refused republish test around node.publish so
it reopens the existing output directory after the failed second publish and
verifies the original stele remains intact, including its expected contents or
identity. Keep the existing unsuccessful-status assertion and use the test’s
established directory-opening and verification helpers where available.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 35d3ae80-921b-40d5-b34c-fda6b2ca4a4b

📥 Commits

Reviewing files that changed from the base of the PR and between 2276f55 and 5265d65.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (11)

• Cargo.toml
• crates/snapshot/Cargo.toml
• crates/snapshot/src/export.rs
• crates/snapshot/src/lib.rs
• crates/snapshot/tests/common/mod.rs
• crates/snapshot/tests/export.rs
• crates/testing/src/toy_domain.rs
• src/bin/dolos/main.rs
• src/bin/dolos/snapshot/mod.rs
• src/bin/dolos/snapshot/publish.rs
• tests/snapshot_publish.rs