Stream blocks during import

[Alenar]

Content

This PR changes the way the blocks are retrieved from the BlockScanner. Instead of giving the full list of blocks from the given cardano database it yields now a BlockStreamer trait that allow to poll the blocks at a pace given by it's implementation (the ImmutableBlockStreamer yield them immutables per immutables).

This reduce the memory load needed during the import process.

The BlockStreamer trait has been designed with the transition to the chain_sync protocol using pallas in mind. As such it sadly don't implement the rust Iterator trait since it needs to be async.

Note: Since mithril-common/src/cardano_block_scanner.rs would become quite huge with the new types introduced, it has been promoted to a module directory instead and it's types spread out to dedicated files.

Pre-submit checklist

• Branch
  • Tests are provided (if possible)
  • Crates versions are updated (if relevant)
  • CHANGELOG file is updated (if relevant)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
• PR
  • No clippy warnings in the CI
  • Self-reviewed the diff
  • Useful pull request description
  • Reviewer requested
• Documentation
  • Update README file (if relevant)
  • Update documentation website (if relevant)
  • Add dev blog post (if relevant)

Comments

This PR should be merged after #1650 since its branch is based on that PR code.

Issue(s)

Closes #1646

[github-actions]

Test Results

    3 files  ±0     43 suites  ±0   8m 23s ⏱️ -1s
  976 tests +4    976 ✅ +4  0 💤 ±0  0 ❌ ±0 
1 070 runs  +4  1 070 ✅ +4  0 💤 ±0  0 ❌ ±0 

Results for commit e32d61b. ± Comparison against base commit c0a7b67.

This pull request removes 7 and adds 11 tests. Note that renamed tests count towards both.

mithril-common ‑ cardano_block_scanner::tests::test_instantiate_parser_with_allow_unparsable_block_should_log_warning
mithril-common ‑ cardano_block_scanner::tests::test_instantiate_parser_without_allow_unparsable_block_should_not_log_warning
mithril-common ‑ cardano_block_scanner::tests::test_parse_expected_number_of_transactions
mithril-common ‑ cardano_block_scanner::tests::test_parse_from_lower_bound_until_upper_bound
mithril-common ‑ cardano_block_scanner::tests::test_parse_should_error_with_unparsable_block_format
mithril-common ‑ cardano_block_scanner::tests::test_parse_should_log_error_with_unparsable_block_format
mithril-common ‑ cardano_block_scanner::tests::test_parse_up_to_given_beacon

mithril-common ‑ cardano_block_scanner::block_scanner::tests::test_instantiate_parser_with_allow_unparsable_block_should_log_warning
mithril-common ‑ cardano_block_scanner::block_scanner::tests::test_instantiate_parser_without_allow_unparsable_block_should_not_log_warning
mithril-common ‑ cardano_block_scanner::block_scanner::tests::test_parse_from_lower_bound_until_upper_bound
mithril-common ‑ cardano_block_scanner::block_scanner::tests::test_parse_up_to_given_beacon
mithril-common ‑ cardano_block_scanner::dumb_block_scanner::tests::dumb_scanned_construct_a_streamer_based_on_its_stored_blocks
mithril-common ‑ cardano_block_scanner::dumb_block_scanner::tests::polling_with_multiple_sets_of_blocks_returns_some_once
mithril-common ‑ cardano_block_scanner::dumb_block_scanner::tests::polling_with_one_set_of_block_returns_some_once
mithril-common ‑ cardano_block_scanner::dumb_block_scanner::tests::polling_without_set_of_block_return_none
mithril-common ‑ cardano_block_scanner::immutable_block_streamer::tests::if_allowed_reading_unparsable_block_should_log_an_error
mithril-common ‑ cardano_block_scanner::immutable_block_streamer::tests::if_disallowed_reading_unparsable_block_should_fail
…

♻️ This comment has been updated with latest results.

[jpraynaud]

It looks a bit more complicated to implement that what I expected and it probably means that we will have to handle the back pressure from the incoming stream to control memory consumption 🤔

[dlachaume]

LGTM 👍, just a few suggestions.