feat: introduce reth era export #15909

lean-apple · 2025-04-24T13:12:38Z

Only start introducing the functional logic (not the interface with the cli) :

For a certain number of block set up in the config, it fetches the data (header, block body, receipts) to reconstruct the block history and write them directly into era files.

Still to do :

Add writing logic,
Add tests with import/export rountrip --> started but not finished

crates/era-export/tests/it/roundtrip.rs

mattsse

awesome, this looks pretty good!

I think we should considering merging both import and export into one crate and rename era-import @RomanHodulak

and I believe we're missing tx.commit calls in import @RomanHodulak

we could also handle those externally but maybe a better idea to commit file after file?

Cargo.toml

crates/era-export/tests/it/roundtrip.rs

RomanHodulak · 2025-04-28T12:00:07Z

@mattsse @lean-apple

I think merging and renaming the crates fits.

Commit per ERA file seems best for this use-case, i.e. the file remains imported even if the process is interupted

mattsse

cool, only thing missing really is a rebase and moving a few things around

Cargo.toml

RomanHodulak · 2025-06-17T10:08:40Z

crates/era-utils/src/export.rs

+    /// Last block to export
+    pub last_block_number: BlockNumber,
+    /// Number of blocks per era1 file
+    pub step: u64,


The amount of blocks per ERA1 file is capped at 8192: https://github.com/eth-clients/e2store-format-specs/blob/main/formats/era1.md

The importer assumes that every file contains 8192 blocks except the last one.

I think the step should not be configurable and simply set to that constant.

Suggested change

pub step: u64,

Initially I added because of the geth function that is referred for in the issue :
https://github.com/ethereum/go-ethereum/blob/9b4eab6a29704f55fa7b4b92e296094f0dbcee22/cmd/utils/cmd.go#L403-L405

We can add one better Default otherwise :

impl Default for ExportConfig { fn default() -> Self { Self { dir: PathBuf::new(), first_block_number: 0, last_block_number: MAX_BLOCKS_PER_ERA1, step: MAX_BLOCKS_PER_ERA1, network: String::new(), } } }

and checked in the cli args it never goes up, just wondering how much for the user experience it can be good to keep it

may I have a final word on it @mattsse @RomanHodulak if we keep it ?
is there an advantage to reduce the number of blocks per file and so have a bigger number of files to export the blocks?

can't hurt if we have a setting for this and use the epoch consts as default imo

maybe there's some usecase for dumping them in one big file

mattsse

cool

I have a few nits/suggestions and would appreciate some additional docs, because this is a bit tricky to follow for anyone unfamiliar with era format

mattsse · 2025-06-19T16:36:13Z

crates/era-utils/src/export.rs

+    /// Last block to export
+    pub last_block_number: BlockNumber,
+    /// Number of blocks per era1 file
+    pub step: u64,


can't hurt if we have a setting for this and use the epoch consts as default imo

maybe there's some usecase for dumping them in one big file

crates/era-utils/src/export.rs

crates/era-utils/tests/it/genesis.rs

crates/era-utils/tests/it/main.rs

mattsse · 2025-06-19T16:43:07Z

crates/era-utils/tests/it/history.rs

-            }
-        }
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_roundtrip_export_after_import() {


this is test does some advanced era specific math, I wont be able to debug this, because not obvious what this does.

can we document the purpose of the test and the individual steps involved here

Let me know if it is more clear now.

mattsse · 2025-06-19T16:43:39Z

crates/era-utils/tests/it/history.rs

+    let last_block = EXPORT_FIRST_BLOCK + EXPORT_TOTAL_BLOCKS - 1;
+    let expected_files = EXPORT_TOTAL_BLOCKS.div_ceil(EXPORT_BLOCK_PER_FILE);


this requires era specific knowdlege, non obvious what this checks

Maybe the required knowledge here is that according to spec each ERA1 file can contain up to 8192 blocks?

The last_block could also be a const.

crates/era-utils/tests/it/history.rs

…max_blocks_per_file

RomanHodulak · 2025-06-23T13:54:54Z

crates/era-utils/tests/it/main.rs

-mod history;
-
-const fn main() {}
-


Should the main function be kept to make this work as a single binary? cc @mattsse

we can keep the main here because this is a single binary

Ah yes sorry I will restore it probably some merge conflicts

Upon looking into this further, I don't think the const fn main() {} is actually needed or should even be here.

https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html

Then rustc compiles the library with --cfg test. It also injects a generated fn main(), which invokes all functions annotated with #[test]. The result is an executable file which, when run subsequently by Cargo, executes the tests.

https://doc.rust-lang.org/cargo/guide/project-layout.html

If a binary, example, bench, or integration test consists of multiple source files, place a main.rs file along with the extra modules within a subdirectory of the src/bin, examples, benches, or tests directory. The name of the executable will be the directory name.

mattsse

lgtm, pending @RomanHodulak

mattsse · 2025-06-23T13:57:31Z

crates/era-utils/tests/it/main.rs

-mod history;
-
-const fn main() {}
-


we can keep the main here because this is a single binary

RomanHodulak

Great work, @lean-apple. I appreciate you spending time on this.

lean-apple · 2025-06-24T14:31:34Z

Great work, @lean-apple. I appreciate you spending time on this.

May I continue on the cli?

lean-apple added 3 commits April 23, 2025 06:32

feat: init era export crate

d514fa6

feat: add base structure

3cbc5d8

feat: complete data fetching from db

fba7db2

github-project-automation bot added this to Reth Tracker Apr 24, 2025

github-project-automation bot moved this to Backlog in Reth Tracker Apr 24, 2025

lean-apple added 13 commits April 24, 2025 15:13

chore: fmt

07b0110

fix: remove unused imports

2909a25

Merge branch 'main' into reth-era-export

d114282

chore: specify crate member path

8c8bfd8

fix: enable db-api feature for reth-storage-api

bae286a

feat: v1 with files writing

68544e1

refactor: add dynamic offset tracking

83ec52f

Merge branch 'main' into reth-era-export

76c596c

fix: fix docs

dff1c52

Merge branch 'main' into reth-era-export

395786d

test: init folder structure

269a1a2

test: add first tests

deca5c8

refactor: optimize provider calls and block writing

4103e79

lean-apple commented Apr 27, 2025

View reviewed changes

crates/era-export/tests/it/roundtrip.rs Outdated Show resolved Hide resolved

lean-apple marked this pull request as ready for review April 28, 2025 06:22

lean-apple requested review from mattsse and gakonst as code owners April 28, 2025 06:22

mattsse reviewed Apr 28, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

crates/era-export/tests/it/roundtrip.rs Outdated Show resolved Hide resolved

This was referenced Apr 28, 2025

chore: rename crate era-import to era-utils #15980

Merged

feat(era-utils): add persistence per file during era history data import #15982

Closed

feat: add era import persistence commit #16059

Closed

mattsse requested changes May 6, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

github-project-automation bot moved this from Backlog to In Progress in Reth Tracker May 6, 2025

mattsse mentioned this pull request May 6, 2025

feat(stages): Add ERA pre-merge history import stage #16008

Merged

RomanHodulak reviewed Jun 17, 2025

View reviewed changes

This was referenced Jun 18, 2025

Support for ERA (not ERA1) export #16898

Open

Support for ERA (not ERA1) in the export-era command #16899

Open

Merge branch 'main' into reth-era-export

b8a516a

lean-apple changed the title ~~feat: introdue reth era export~~ feat: introduce reth era export Jun 18, 2025

lean-apple requested review from RomanHodulak and mattsse June 19, 2025 15:31

mattsse requested changes Jun 19, 2025

View reviewed changes

RomanHodulak reviewed Jun 20, 2025

View reviewed changes

crates/era-utils/tests/it/history.rs Outdated Show resolved Hide resolved

lean-apple mentioned this pull request Jun 21, 2025

chore(era): complete doc for ClientWithFakeIndex #16984

Merged

lean-apple added 6 commits June 22, 2025 13:32

fix: remove async for genesis test

c053a52

fix: use reth_fs_util for fs

30f58a3

Merge branch 'main' into reth-era-export

fdffa9a

refactor: add more specific default export config and rename step to …

674c5dc

…max_blocks_per_file

refactor(test): add doc and simplify the test

e61999a

chore: fmt

cac076c

lean-apple requested a review from RomanHodulak June 23, 2025 05:01

test: add simple unit test on export config validation

0c09131

lean-apple requested a review from mattsse June 23, 2025 05:22

RomanHodulak reviewed Jun 23, 2025

View reviewed changes

mattsse approved these changes Jun 24, 2025

View reviewed changes

RomanHodulak approved these changes Jun 24, 2025

View reviewed changes

chore: add back main

a24af8b

mattsse added this pull request to the merge queue Jun 25, 2025

Merged via the queue into paradigmxyz:main with commit 4afe790 Jun 25, 2025
40 checks passed

github-project-automation bot moved this from In Progress to Done in Reth Tracker Jun 25, 2025

auricom mentioned this pull request Jun 25, 2025

[EVM] Backup reth state from the sequencer periodically in some database evstack/ev-node#2391

Closed

lean-apple deleted the reth-era-export branch June 26, 2025 03:25

		let last_block = EXPORT_FIRST_BLOCK + EXPORT_TOTAL_BLOCKS - 1;
		let expected_files = EXPORT_TOTAL_BLOCKS.div_ceil(EXPORT_BLOCK_PER_FILE);

		mod history;

		const fn main() {}

		mod history;

		const fn main() {}

feat: introduce reth era export #15909

feat: introduce reth era export #15909

Uh oh!

Conversation

lean-apple commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

RomanHodulak commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lean-apple Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lean-apple Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RomanHodulak left a comment

Choose a reason for hiding this comment

Uh oh!

lean-apple commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

lean-apple commented Apr 24, 2025 •

edited

Loading

RomanHodulak commented Apr 28, 2025 •

edited

Loading

lean-apple Jun 18, 2025 •

edited

Loading

lean-apple Jun 19, 2025 •

edited

Loading